2024-12-18T03:30:27.1582884Z Current runner version: '2.321.0'
2024-12-18T03:30:27.1596414Z Runner name: 'gpu6991.jax.cs.cpe.ice.amd.com'
2024-12-18T03:30:27.1598225Z Runner group name: 'linux.rocm.gpu.group'
2024-12-18T03:30:27.1599911Z Machine name: 'gpu6991'
2024-12-18T03:30:27.1605387Z ##[group]GITHUB_TOKEN Permissions
2024-12-18T03:30:27.1609335Z Contents: read
2024-12-18T03:30:27.1610645Z Metadata: read
2024-12-18T03:30:27.1611543Z ##[endgroup]
2024-12-18T03:30:27.1616596Z Secret source: Actions
2024-12-18T03:30:27.1618323Z Prepare workflow directory
2024-12-18T03:30:27.7654046Z Prepare all required actions
2024-12-18T03:30:27.7690477Z Getting action download info
2024-12-18T03:30:28.1675272Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T03:30:33.3893788Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2024-12-18T03:30:33.8656313Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2024-12-18T03:30:34.2890266Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c)
2024-12-18T03:30:35.1475858Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b)
2024-12-18T03:30:35.8110574Z Getting action download info
2024-12-18T03:30:35.9619522Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-12-18T03:30:36.5184668Z Getting action download info
2024-12-18T03:30:36.6759846Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2024-12-18T03:30:37.1298950Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T03:30:37.1300952Z ##[group] Inputs
2024-12-18T03:30:37.1301237Z   build-environment: linux-focal-rocm6.2-py3.10
2024-12-18T03:30:37.1302419Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}
2024-12-18T03:30:37.1303767Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:30:37.1304262Z   sync-tag: 
2024-12-18T03:30:37.1304905Z   timeout-minutes: 300
2024-12-18T03:30:37.1305111Z   tests-to-include: 
2024-12-18T03:30:37.1305586Z   disable-monitor: true
2024-12-18T03:30:37.1305794Z ##[endgroup]
2024-12-18T03:30:37.1306180Z Complete job name: linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:30:37.2905404Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6
2024-12-18T03:30:37.2906047Z with:
2024-12-18T03:30:37.2906223Z   no-sudo: true
2024-12-18T03:30:37.2906422Z   submodules: recursive
2024-12-18T03:30:37.2906603Z   fetch-depth: 0
2024-12-18T03:30:37.2907002Z env:
2024-12-18T03:30:37.2907168Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:30:37.2907367Z ##[endgroup]
2024-12-18T03:30:37.2979171Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T03:30:37.2979914Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T03:30:37.3011505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:30:37.3011824Z env:
2024-12-18T03:30:37.3011985Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:30:37.3012178Z ##[endgroup]
2024-12-18T03:30:37.3347711Z ##[group]Run retry () {
2024-12-18T03:30:37.3348372Z retry () {
2024-12-18T03:30:37.3349863Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-12-18T03:30:37.3350668Z }
2024-12-18T03:30:37.3351112Z echo "${GITHUB_WORKSPACE}"
2024-12-18T03:30:37.3351696Z if [ -z "${NO_SUDO}" ]; then
2024-12-18T03:30:37.3352339Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T03:30:37.3352957Z else
2024-12-18T03:30:37.3353415Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T03:30:37.3353998Z fi
2024-12-18T03:30:37.3354431Z mkdir "${GITHUB_WORKSPACE}"
2024-12-18T03:30:37.3410020Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:30:37.3410647Z env:
2024-12-18T03:30:37.3410988Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:30:37.3411383Z   NO_SUDO: true
2024-12-18T03:30:37.3411741Z ##[endgroup]
2024-12-18T03:30:37.3528364Z /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T03:30:41.4613450Z ##[group]Run malfet/checkout@silent-checkout
2024-12-18T03:30:41.4614124Z with:
2024-12-18T03:30:41.4614586Z   ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:30:41.4615182Z   fetch-depth: 0
2024-12-18T03:30:41.4615597Z   submodules: recursive
2024-12-18T03:30:41.4616052Z   quiet-checkout: true
2024-12-18T03:30:41.4616516Z   repository: pytorch/pytorch
2024-12-18T03:30:41.4617261Z   token: ***
2024-12-18T03:30:41.4617684Z   ssh-strict: true
2024-12-18T03:30:41.4618136Z   persist-credentials: true
2024-12-18T03:30:41.4618613Z   clean: true
2024-12-18T03:30:41.4619063Z   sparse-checkout-cone-mode: true
2024-12-18T03:30:41.4619598Z   lfs: false
2024-12-18T03:30:41.4620007Z   set-safe-directory: true
2024-12-18T03:30:41.4620496Z env:
2024-12-18T03:30:41.4620893Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:30:41.4621344Z ##[endgroup]
2024-12-18T03:30:41.5844885Z Syncing repository: pytorch/pytorch
2024-12-18T03:30:41.5847456Z ##[group]Getting Git version info
2024-12-18T03:30:41.5848462Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2024-12-18T03:30:41.5849761Z [command]/usr/bin/git version
2024-12-18T03:30:41.5850263Z git version 2.34.1
2024-12-18T03:30:41.5852145Z ##[endgroup]
2024-12-18T03:30:41.5859549Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/329b1100-a4e2-497d-9b69-3e536ad2384e' before making global git config changes
2024-12-18T03:30:41.5861250Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T03:30:41.5862652Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T03:30:41.5889620Z Deleting the contents of '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2024-12-18T03:30:41.5894096Z ##[group]Initializing the repository
2024-12-18T03:30:41.5895722Z [command]/usr/bin/git init /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T03:30:41.5960478Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-12-18T03:30:41.5961768Z hint: is subject to change. To configure the initial branch name to use in all
2024-12-18T03:30:41.5962778Z hint: of your new repositories, which will suppress this warning, call:
2024-12-18T03:30:41.5963471Z hint: 
2024-12-18T03:30:41.5964147Z hint: 	git config --global init.defaultBranch <name>
2024-12-18T03:30:41.5964808Z hint: 
2024-12-18T03:30:41.5965435Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-12-18T03:30:41.5966393Z hint: 'development'. The just-created branch can be renamed via this command:
2024-12-18T03:30:41.5967135Z hint: 
2024-12-18T03:30:41.5967523Z hint: 	git branch -m <name>
2024-12-18T03:30:41.5974985Z Initialized empty Git repository in /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/
2024-12-18T03:30:41.5984938Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-12-18T03:30:41.6047584Z ##[endgroup]
2024-12-18T03:30:41.6048414Z ##[group]Disabling automatic garbage collection
2024-12-18T03:30:41.6049718Z [command]/usr/bin/git config --local gc.auto 0
2024-12-18T03:30:41.6106476Z ##[endgroup]
2024-12-18T03:30:41.6107273Z ##[group]Setting up auth
2024-12-18T03:30:41.6115133Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T03:30:41.6167489Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T03:30:41.6809982Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T03:30:41.6869727Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T03:30:41.7487396Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T03:30:41.7567698Z ##[endgroup]
2024-12-18T03:30:41.7568558Z ##[group]Fetching the repository
2024-12-18T03:30:41.7579255Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-12-18T03:30:47.2411547Z remote: Enumerating objects: 1055652        
2024-12-18T03:30:47.2412636Z remote: Enumerating objects: 1056587, done.        
2024-12-18T03:30:47.2413386Z remote: Counting objects:   0% (1/935)        
2024-12-18T03:30:47.2414041Z remote: Counting objects:   1% (10/935)        
2024-12-18T03:30:47.2414676Z remote: Counting objects:   2% (19/935)        
2024-12-18T03:30:47.2415294Z remote: Counting objects:   3% (29/935)        
2024-12-18T03:30:47.2415887Z remote: Counting objects:   4% (38/935)        
2024-12-18T03:30:47.2416463Z remote: Counting objects:   5% (47/935)        
2024-12-18T03:30:47.2417102Z remote: Counting objects:   6% (57/935)        
2024-12-18T03:30:47.2417794Z remote: Counting objects:   7% (66/935)        
2024-12-18T03:30:47.2418492Z remote: Counting objects:   8% (75/935)        
2024-12-18T03:30:47.2419186Z remote: Counting objects:   9% (85/935)        
2024-12-18T03:30:47.2419929Z remote: Counting objects:  10% (94/935)        
2024-12-18T03:30:47.2420643Z remote: Counting objects:  11% (103/935)        
2024-12-18T03:30:47.2421376Z remote: Counting objects:  12% (113/935)        
2024-12-18T03:30:47.2422082Z remote: Counting objects:  13% (122/935)        
2024-12-18T03:30:47.2422781Z remote: Counting objects:  14% (131/935)        
2024-12-18T03:30:47.2423482Z remote: Counting objects:  15% (141/935)        
2024-12-18T03:30:47.2424103Z remote: Counting objects:  16% (150/935)        
2024-12-18T03:30:47.2424679Z remote: Counting objects:  17% (159/935)        
2024-12-18T03:30:47.2425267Z remote: Counting objects:  18% (169/935)        
2024-12-18T03:30:47.2425860Z remote: Counting objects:  19% (178/935)        
2024-12-18T03:30:47.2427476Z remote: Counting objects:  20% (187/935)        
2024-12-18T03:30:47.2428190Z remote: Counting objects:  21% (197/935)        
2024-12-18T03:30:47.2428791Z remote: Counting objects:  22% (206/935)        
2024-12-18T03:30:47.2429367Z remote: Counting objects:  23% (216/935)        
2024-12-18T03:30:47.2429962Z remote: Counting objects:  24% (225/935)        
2024-12-18T03:30:47.2430548Z remote: Counting objects:  25% (234/935)        
2024-12-18T03:30:47.2431129Z remote: Counting objects:  26% (244/935)        
2024-12-18T03:30:47.2431840Z remote: Counting objects:  27% (253/935)        
2024-12-18T03:30:47.2432532Z remote: Counting objects:  28% (262/935)        
2024-12-18T03:30:47.2433120Z remote: Counting objects:  29% (272/935)        
2024-12-18T03:30:47.2433693Z remote: Counting objects:  30% (281/935)        
2024-12-18T03:30:47.2434280Z remote: Counting objects:  31% (290/935)        
2024-12-18T03:30:47.2434868Z remote: Counting objects:  32% (300/935)        
2024-12-18T03:30:47.2435454Z remote: Counting objects:  33% (309/935)        
2024-12-18T03:30:47.2436507Z remote: Counting objects:  34% (318/935)        
2024-12-18T03:30:47.2437229Z remote: Counting objects:  35% (328/935)        
2024-12-18T03:30:47.2437914Z remote: Counting objects:  36% (337/935)        
2024-12-18T03:30:47.2438611Z remote: Counting objects:  37% (346/935)        
2024-12-18T03:30:47.2439309Z remote: Counting objects:  38% (356/935)        
2024-12-18T03:30:47.2439970Z remote: Counting objects:  39% (365/935)        
2024-12-18T03:30:47.2440545Z remote: Counting objects:  40% (374/935)        
2024-12-18T03:30:47.2441159Z remote: Counting objects:  41% (384/935)        
2024-12-18T03:30:47.2441737Z remote: Counting objects:  42% (393/935)        
2024-12-18T03:30:47.2442319Z remote: Counting objects:  43% (403/935)        
2024-12-18T03:30:47.2442881Z remote: Counting objects:  44% (412/935)        
2024-12-18T03:30:47.2443463Z remote: Counting objects:  45% (421/935)        
2024-12-18T03:30:47.2444046Z remote: Counting objects:  46% (431/935)        
2024-12-18T03:30:47.2444634Z remote: Counting objects:  47% (440/935)        
2024-12-18T03:30:47.2445211Z remote: Counting objects:  48% (449/935)        
2024-12-18T03:30:47.2445818Z remote: Counting objects:  49% (459/935)        
2024-12-18T03:30:47.2446396Z remote: Counting objects:  50% (468/935)        
2024-12-18T03:30:47.2446971Z remote: Counting objects:  51% (477/935)        
2024-12-18T03:30:47.2447558Z remote: Counting objects:  52% (487/935)        
2024-12-18T03:30:47.2448123Z remote: Counting objects:  53% (496/935)        
2024-12-18T03:30:47.2448703Z remote: Counting objects:  54% (505/935)        
2024-12-18T03:30:47.2449279Z remote: Counting objects:  55% (515/935)        
2024-12-18T03:30:47.2449842Z remote: Counting objects:  56% (524/935)        
2024-12-18T03:30:47.2450419Z remote: Counting objects:  57% (533/935)        
2024-12-18T03:30:47.2450998Z remote: Counting objects:  58% (543/935)        
2024-12-18T03:30:47.2451587Z remote: Counting objects:  59% (552/935)        
2024-12-18T03:30:47.2452181Z remote: Counting objects:  60% (561/935)        
2024-12-18T03:30:47.2452764Z remote: Counting objects:  61% (571/935)        
2024-12-18T03:30:47.2453331Z remote: Counting objects:  62% (580/935)        
2024-12-18T03:30:47.2453960Z remote: Counting objects:  63% (590/935)        
2024-12-18T03:30:47.2454573Z remote: Counting objects:  64% (599/935)        
2024-12-18T03:30:47.2455164Z remote: Counting objects:  65% (608/935)        
2024-12-18T03:30:47.2455733Z remote: Counting objects:  66% (618/935)        
2024-12-18T03:30:47.2456306Z remote: Counting objects:  67% (627/935)        
2024-12-18T03:30:47.2456888Z remote: Counting objects:  68% (636/935)        
2024-12-18T03:30:47.2457462Z remote: Counting objects:  69% (646/935)        
2024-12-18T03:30:47.2458038Z remote: Counting objects:  70% (655/935)        
2024-12-18T03:30:47.2458611Z remote: Counting objects:  71% (664/935)        
2024-12-18T03:30:47.2459505Z remote: Counting objects:  72% (674/935)        
2024-12-18T03:30:47.2460146Z remote: Counting objects:  73% (683/935)        
2024-12-18T03:30:47.2460765Z remote: Counting objects:  74% (692/935)        
2024-12-18T03:30:47.2461355Z remote: Counting objects:  75% (702/935)        
2024-12-18T03:30:47.2461940Z remote: Counting objects:  76% (711/935)        
2024-12-18T03:30:47.2462518Z remote: Counting objects:  77% (720/935)        
2024-12-18T03:30:47.2463092Z remote: Counting objects:  78% (730/935)        
2024-12-18T03:30:47.2463686Z remote: Counting objects:  79% (739/935)        
2024-12-18T03:30:47.2464268Z remote: Counting objects:  80% (748/935)        
2024-12-18T03:30:47.2464846Z remote: Counting objects:  81% (758/935)        
2024-12-18T03:30:47.2465434Z remote: Counting objects:  82% (767/935)        
2024-12-18T03:30:47.2466016Z remote: Counting objects:  83% (777/935)        
2024-12-18T03:30:47.2466591Z remote: Counting objects:  84% (786/935)        
2024-12-18T03:30:47.2467755Z remote: Counting objects:  85% (795/935)        
2024-12-18T03:30:47.2468419Z remote: Counting objects:  86% (805/935)        
2024-12-18T03:30:47.2469356Z remote: Counting objects:  87% (814/935)        
2024-12-18T03:30:47.2469941Z remote: Counting objects:  88% (823/935)        
2024-12-18T03:30:47.2470528Z remote: Counting objects:  89% (833/935)        
2024-12-18T03:30:47.2471099Z remote: Counting objects:  90% (842/935)        
2024-12-18T03:30:47.2471675Z remote: Counting objects:  91% (851/935)        
2024-12-18T03:30:47.2472248Z remote: Counting objects:  92% (861/935)        
2024-12-18T03:30:47.2472827Z remote: Counting objects:  93% (870/935)        
2024-12-18T03:30:47.2473404Z remote: Counting objects:  94% (879/935)        
2024-12-18T03:30:47.2473989Z remote: Counting objects:  95% (889/935)        
2024-12-18T03:30:47.2474556Z remote: Counting objects:  96% (898/935)        
2024-12-18T03:30:47.2475139Z remote: Counting objects:  97% (907/935)        
2024-12-18T03:30:47.2475734Z remote: Counting objects:  98% (917/935)        
2024-12-18T03:30:47.2476310Z remote: Counting objects:  99% (926/935)        
2024-12-18T03:30:47.2476904Z remote: Counting objects: 100% (935/935)        
2024-12-18T03:30:47.2477537Z remote: Counting objects: 100% (935/935), done.        
2024-12-18T03:30:47.2524924Z remote: Compressing objects:   0% (1/401)        
2024-12-18T03:30:47.2661817Z remote: Compressing objects:   1% (5/401)        
2024-12-18T03:30:47.2877594Z remote: Compressing objects:   2% (9/401)        
2024-12-18T03:30:47.2958812Z remote: Compressing objects:   3% (13/401)        
2024-12-18T03:30:47.3190409Z remote: Compressing objects:   4% (17/401)        
2024-12-18T03:30:47.3602461Z remote: Compressing objects:   5% (21/401)        
2024-12-18T03:30:47.4282975Z remote: Compressing objects:   6% (25/401)        
2024-12-18T03:30:47.4972679Z remote: Compressing objects:   7% (29/401)        
2024-12-18T03:30:47.5387266Z remote: Compressing objects:   8% (33/401)        
2024-12-18T03:30:47.5826099Z remote: Compressing objects:   9% (37/401)        
2024-12-18T03:30:47.6010177Z remote: Compressing objects:  10% (41/401)        
2024-12-18T03:30:47.6074718Z remote: Compressing objects:  11% (45/401)        
2024-12-18T03:30:47.6075509Z remote: Compressing objects:  12% (49/401)        
2024-12-18T03:30:47.6085436Z remote: Compressing objects:  13% (53/401)        
2024-12-18T03:30:47.6092184Z remote: Compressing objects:  14% (57/401)        
2024-12-18T03:30:47.6092980Z remote: Compressing objects:  15% (61/401)        
2024-12-18T03:30:47.6099504Z remote: Compressing objects:  16% (65/401)        
2024-12-18T03:30:47.6102303Z remote: Compressing objects:  17% (69/401)        
2024-12-18T03:30:47.6104131Z remote: Compressing objects:  18% (73/401)        
2024-12-18T03:30:47.6104915Z remote: Compressing objects:  19% (77/401)        
2024-12-18T03:30:47.6111585Z remote: Compressing objects:  20% (81/401)        
2024-12-18T03:30:47.6117407Z remote: Compressing objects:  21% (85/401)        
2024-12-18T03:30:47.6119078Z remote: Compressing objects:  22% (89/401)        
2024-12-18T03:30:47.6126888Z remote: Compressing objects:  23% (93/401)        
2024-12-18T03:30:47.6131731Z remote: Compressing objects:  24% (97/401)        
2024-12-18T03:30:47.6134890Z remote: Compressing objects:  25% (101/401)        
2024-12-18T03:30:47.6137998Z remote: Compressing objects:  26% (105/401)        
2024-12-18T03:30:47.6142927Z remote: Compressing objects:  27% (109/401)        
2024-12-18T03:30:47.6148195Z remote: Compressing objects:  28% (113/401)        
2024-12-18T03:30:47.6155653Z remote: Compressing objects:  29% (117/401)        
2024-12-18T03:30:47.6158258Z remote: Compressing objects:  30% (121/401)        
2024-12-18T03:30:47.6161260Z remote: Compressing objects:  31% (125/401)        
2024-12-18T03:30:47.6165029Z remote: Compressing objects:  32% (129/401)        
2024-12-18T03:30:47.6170481Z remote: Compressing objects:  33% (133/401)        
2024-12-18T03:30:47.6179988Z remote: Compressing objects:  34% (137/401)        
2024-12-18T03:30:47.6180669Z remote: Compressing objects:  35% (141/401)        
2024-12-18T03:30:47.6182837Z remote: Compressing objects:  36% (145/401)        
2024-12-18T03:30:47.6195982Z remote: Compressing objects:  37% (149/401)        
2024-12-18T03:30:47.6204072Z remote: Compressing objects:  38% (153/401)        
2024-12-18T03:30:47.6209048Z remote: Compressing objects:  39% (157/401)        
2024-12-18T03:30:47.6212889Z remote: Compressing objects:  40% (161/401)        
2024-12-18T03:30:47.6213491Z remote: Compressing objects:  41% (165/401)        
2024-12-18T03:30:47.6214090Z remote: Compressing objects:  42% (169/401)        
2024-12-18T03:30:47.6215177Z remote: Compressing objects:  43% (173/401)        
2024-12-18T03:30:47.6216218Z remote: Compressing objects:  44% (177/401)        
2024-12-18T03:30:47.6225155Z remote: Compressing objects:  45% (181/401)        
2024-12-18T03:30:47.6225814Z remote: Compressing objects:  46% (185/401)        
2024-12-18T03:30:47.6226852Z remote: Compressing objects:  47% (189/401)        
2024-12-18T03:30:47.6232042Z remote: Compressing objects:  48% (193/401)        
2024-12-18T03:30:47.6241905Z remote: Compressing objects:  49% (197/401)        
2024-12-18T03:30:47.6254257Z remote: Compressing objects:  50% (201/401)        
2024-12-18T03:30:47.6257721Z remote: Compressing objects:  51% (205/401)        
2024-12-18T03:30:47.6258325Z remote: Compressing objects:  52% (209/401)        
2024-12-18T03:30:47.6260108Z remote: Compressing objects:  53% (213/401)        
2024-12-18T03:30:47.6263748Z remote: Compressing objects:  54% (217/401)        
2024-12-18T03:30:47.6264697Z remote: Compressing objects:  55% (221/401)        
2024-12-18T03:30:47.6267697Z remote: Compressing objects:  56% (225/401)        
2024-12-18T03:30:47.6270177Z remote: Compressing objects:  57% (229/401)        
2024-12-18T03:30:47.6274078Z remote: Compressing objects:  58% (233/401)        
2024-12-18T03:30:47.6275486Z remote: Compressing objects:  59% (237/401)        
2024-12-18T03:30:47.6278457Z remote: Compressing objects:  60% (241/401)        
2024-12-18T03:30:47.6281654Z remote: Compressing objects:  61% (245/401)        
2024-12-18T03:30:47.6285777Z remote: Compressing objects:  62% (249/401)        
2024-12-18T03:30:47.6302404Z remote: Compressing objects:  63% (253/401)        
2024-12-18T03:30:47.6316428Z remote: Compressing objects:  64% (257/401)        
2024-12-18T03:30:47.6317026Z remote: Compressing objects:  65% (261/401)        
2024-12-18T03:30:47.6317629Z remote: Compressing objects:  66% (265/401)        
2024-12-18T03:30:47.6318223Z remote: Compressing objects:  67% (269/401)        
2024-12-18T03:30:47.6318865Z remote: Compressing objects:  68% (273/401)        
2024-12-18T03:30:47.6319474Z remote: Compressing objects:  69% (277/401)        
2024-12-18T03:30:47.6320077Z remote: Compressing objects:  70% (281/401)        
2024-12-18T03:30:47.6320673Z remote: Compressing objects:  71% (285/401)        
2024-12-18T03:30:47.6321273Z remote: Compressing objects:  72% (289/401)        
2024-12-18T03:30:47.6322270Z remote: Compressing objects:  73% (293/401)        
2024-12-18T03:30:47.6322945Z remote: Compressing objects:  74% (297/401)        
2024-12-18T03:30:47.6323559Z remote: Compressing objects:  75% (301/401)        
2024-12-18T03:30:47.6325233Z remote: Compressing objects:  76% (305/401)        
2024-12-18T03:30:47.6326225Z remote: Compressing objects:  77% (309/401)        
2024-12-18T03:30:47.6327368Z remote: Compressing objects:  78% (313/401)        
2024-12-18T03:30:47.6329721Z remote: Compressing objects:  79% (317/401)        
2024-12-18T03:30:47.6330382Z remote: Compressing objects:  80% (321/401)        
2024-12-18T03:30:47.6331231Z remote: Compressing objects:  81% (325/401)        
2024-12-18T03:30:47.6332656Z remote: Compressing objects:  82% (329/401)        
2024-12-18T03:30:47.6333281Z remote: Compressing objects:  83% (333/401)        
2024-12-18T03:30:47.6335514Z remote: Compressing objects:  84% (337/401)        
2024-12-18T03:30:47.6338563Z remote: Compressing objects:  85% (341/401)        
2024-12-18T03:30:47.6339401Z remote: Compressing objects:  86% (345/401)        
2024-12-18T03:30:47.6341822Z remote: Compressing objects:  87% (349/401)        
2024-12-18T03:30:47.6343198Z remote: Compressing objects:  88% (353/401)        
2024-12-18T03:30:47.6344265Z remote: Compressing objects:  89% (357/401)        
2024-12-18T03:30:47.6346028Z remote: Compressing objects:  90% (361/401)        
2024-12-18T03:30:47.6347443Z remote: Compressing objects:  91% (365/401)        
2024-12-18T03:30:47.6348407Z remote: Compressing objects:  92% (369/401)        
2024-12-18T03:30:47.6349478Z remote: Compressing objects:  93% (373/401)        
2024-12-18T03:30:47.6350345Z remote: Compressing objects:  94% (377/401)        
2024-12-18T03:30:47.6351367Z remote: Compressing objects:  95% (381/401)        
2024-12-18T03:30:47.6352366Z remote: Compressing objects:  96% (385/401)        
2024-12-18T03:30:47.6353026Z remote: Compressing objects:  97% (389/401)        
2024-12-18T03:30:47.6354609Z remote: Compressing objects:  98% (393/401)        
2024-12-18T03:30:47.6355279Z remote: Compressing objects:  99% (397/401)        
2024-12-18T03:30:47.6355918Z remote: Compressing objects: 100% (401/401)        
2024-12-18T03:30:47.6356595Z remote: Compressing objects: 100% (401/401), done.        
2024-12-18T03:31:04.3676161Z remote: Total 1056587 (delta 762), reused 543 (delta 534), pack-reused 1055652 (from 5)        
2024-12-18T03:31:15.3630766Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object}
2024-12-18T03:31:15.3687448Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:31:15.3699020Z ##[endgroup]
2024-12-18T03:31:15.3703800Z ##[group]Determining the checkout info
2024-12-18T03:31:15.3704735Z ##[endgroup]
2024-12-18T03:31:15.3705453Z ##[group]Checking out the ref
2024-12-18T03:31:15.3707165Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:31:16.8610956Z ##[endgroup]
2024-12-18T03:31:16.8611911Z ##[group]Setting up auth for fetching submodules
2024-12-18T03:31:16.8616764Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T03:31:16.8699963Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-12-18T03:31:16.8769580Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-12-18T03:31:16.8838873Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-12-18T03:31:16.8908142Z ##[endgroup]
2024-12-18T03:31:16.8909028Z ##[group]Fetching submodules
2024-12-18T03:31:16.8913610Z [command]/usr/bin/git submodule sync --recursive
2024-12-18T03:31:16.9606030Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-12-18T03:31:17.0306549Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-12-18T03:31:17.0316268Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-12-18T03:31:17.0322698Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-12-18T03:31:17.0331322Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-12-18T03:31:17.0340239Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2024-12-18T03:31:17.0348724Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-12-18T03:31:17.0357400Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-12-18T03:31:17.0366429Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-12-18T03:31:17.0375149Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2024-12-18T03:31:17.0383982Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-12-18T03:31:17.0393920Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-12-18T03:31:17.0401583Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-12-18T03:31:17.0410840Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-12-18T03:31:17.0420173Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-12-18T03:31:17.0430142Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-12-18T03:31:17.0445437Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-12-18T03:31:17.0455615Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-12-18T03:31:17.0465886Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:31:17.0477810Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-12-18T03:31:17.0487903Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-12-18T03:31:17.0498071Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-12-18T03:31:17.0509066Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-12-18T03:31:17.0520036Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-12-18T03:31:17.0531133Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-12-18T03:31:17.0542034Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-12-18T03:31:17.0553996Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-12-18T03:31:17.0565535Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-12-18T03:31:17.0577195Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-12-18T03:31:17.0590106Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-12-18T03:31:17.0601609Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-12-18T03:31:17.0614037Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-12-18T03:31:17.0625733Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-12-18T03:31:17.0638312Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-12-18T03:31:17.0651196Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-12-18T03:31:17.0663244Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-12-18T03:31:17.0676399Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-12-18T03:31:17.0860554Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-12-18T03:31:19.9488095Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-12-18T03:31:22.5881472Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-12-18T03:31:24.8064224Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-12-18T03:31:27.0697433Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2024-12-18T03:31:30.0463054Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-12-18T03:31:33.6153006Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-12-18T03:31:42.6872320Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-12-18T03:31:44.6574855Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2024-12-18T03:31:48.4384347Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-12-18T03:31:51.1750792Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-12-18T03:31:54.5096422Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-12-18T03:31:57.8434002Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-12-18T03:32:01.3999978Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-12-18T03:32:05.4092210Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-12-18T03:32:08.3690966Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-12-18T03:32:11.6388082Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-12-18T03:32:15.2008566Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-12-18T03:32:17.9693774Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-12-18T03:32:20.5551891Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-12-18T03:32:23.3579363Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-12-18T03:32:26.0791413Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-12-18T03:32:28.7125261Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-12-18T03:32:32.2664494Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-12-18T03:32:34.6841844Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-12-18T03:32:36.7000234Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-12-18T03:32:43.2037015Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-12-18T03:32:46.6495324Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-12-18T03:32:51.0943831Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-12-18T03:32:53.2262937Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-12-18T03:33:01.4715861Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-12-18T03:33:03.6373290Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-12-18T03:33:05.8369386Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-12-18T03:33:08.7760898Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-12-18T03:33:12.2750190Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-12-18T03:33:15.4975360Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-12-18T03:33:18.3684037Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-12-18T03:33:18.4372032Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-12-18T03:33:18.4963912Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-12-18T03:33:18.5905862Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-12-18T03:33:18.7026033Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2024-12-18T03:33:18.8143008Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-12-18T03:33:19.6940157Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8'
2024-12-18T03:33:19.7914241Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-12-18T03:33:20.1185993Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57'
2024-12-18T03:33:20.2504317Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-12-18T03:33:20.4302173Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99'
2024-12-18T03:33:20.5420434Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45'
2024-12-18T03:33:21.1515219Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-12-18T03:33:21.4857625Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-12-18T03:33:21.6916323Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-12-18T03:33:21.7046509Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:33:21.7056121Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:33:21.7065007Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:33:21.7074473Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:33:21.7085142Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:33:21.7207175Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-12-18T03:33:24.3832234Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-12-18T03:33:27.4407055Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-12-18T03:33:30.9253153Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-12-18T03:33:34.0045526Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-12-18T03:33:36.7022663Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-12-18T03:33:36.8747576Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-12-18T03:33:37.3779448Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-12-18T03:33:37.5306361Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-12-18T03:33:37.5924609Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-12-18T03:33:37.8134091Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-12-18T03:33:37.9350637Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-12-18T03:33:38.0548824Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-12-18T03:33:38.1547160Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-12-18T03:33:38.2716432Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2024-12-18T03:33:38.3398202Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b'
2024-12-18T03:33:38.3520769Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-12-18T03:33:38.3637024Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-12-18T03:33:51.6099290Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af'
2024-12-18T03:33:51.6922528Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-12-18T03:33:51.8578776Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d'
2024-12-18T03:33:51.8711756Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:33:51.8719635Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:33:51.8728971Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:33:51.8850184Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-12-18T03:33:54.7006662Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-12-18T03:33:57.6583946Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-12-18T03:34:00.5966960Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-12-18T03:34:00.6103806Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:34:00.6112622Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:34:00.6121795Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:34:00.6131035Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:34:00.6140017Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:34:00.6149364Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:34:00.6158848Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:34:00.6168639Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:34:00.6290680Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-12-18T03:34:03.5624671Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-12-18T03:34:06.7224701Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-12-18T03:34:10.3060522Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-12-18T03:34:13.0272221Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-12-18T03:34:15.6656215Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-12-18T03:34:18.4259046Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-12-18T03:34:26.1974045Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-12-18T03:34:29.5636709Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-12-18T03:34:29.6424851Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-12-18T03:34:29.7583977Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-12-18T03:34:29.8171138Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-12-18T03:34:29.8295760Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:34:29.8413223Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-12-18T03:34:32.6354217Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-12-18T03:34:32.7150401Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-12-18T03:34:32.8268076Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-12-18T03:34:33.0022931Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-12-18T03:34:33.0737403Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-12-18T03:34:33.1805032Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-12-18T03:34:33.3254430Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-12-18T03:34:33.4431509Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-12-18T03:34:33.5488451Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-12-18T03:34:33.7461628Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-12-18T03:34:34.2569331Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2024-12-18T03:34:34.2730875Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-12-18T03:34:34.2885289Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-12-18T03:34:37.9704031Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-12-18T03:34:38.1174813Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-12-18T03:34:38.1308886Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:34:38.1315795Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:34:38.1324672Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:34:38.1333826Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:34:38.1342613Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:34:38.1350980Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:34:38.1360630Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:34:38.1370257Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:34:38.1475640Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-12-18T03:34:41.6556234Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-12-18T03:34:45.0526954Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-12-18T03:34:47.9918707Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-12-18T03:34:55.0976498Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-12-18T03:34:58.0292646Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-12-18T03:35:01.2148831Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-12-18T03:35:04.1852213Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-12-18T03:35:14.8982000Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-12-18T03:35:15.0072156Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-12-18T03:35:15.0766016Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-12-18T03:35:15.2634439Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-12-18T03:35:15.3294545Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-12-18T03:35:15.3959747Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-12-18T03:35:15.4679569Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-12-18T03:35:15.4801168Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:15.4809478Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:15.4934363Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-12-18T03:35:18.9806363Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-12-18T03:35:22.7361254Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-12-18T03:35:22.8535426Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-12-18T03:35:23.4685816Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-12-18T03:35:23.5381003Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-12-18T03:35:23.9219431Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-12-18T03:35:23.9353946Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:23.9363677Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:23.9479950Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-12-18T03:35:27.1547658Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-12-18T03:35:30.8843259Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-12-18T03:35:31.0312457Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-12-18T03:35:31.0888201Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-12-18T03:35:31.1551477Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-12-18T03:35:31.2697440Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2024-12-18T03:35:31.3738451Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-12-18T03:35:31.4890104Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-12-18T03:35:31.5825149Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-12-18T03:35:31.5945337Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:31.5954449Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:31.5963588Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:31.5972287Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:31.6082644Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-12-18T03:35:34.6239970Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-12-18T03:35:37.4224664Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-12-18T03:35:43.2357530Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-12-18T03:35:47.0382984Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-12-18T03:35:47.1041093Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-12-18T03:35:47.2567211Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-12-18T03:35:47.3558059Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-12-18T03:35:47.3679029Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:47.3799595Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-12-18T03:35:50.4728073Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-12-18T03:35:50.4980320Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-12-18T03:35:50.5650573Z Entering 'android/libs/fbjni'
2024-12-18T03:35:50.5757136Z Entering 'third_party/FP16'
2024-12-18T03:35:50.5873004Z Entering 'third_party/FXdiv'
2024-12-18T03:35:50.5975743Z Entering 'third_party/NNPACK'
2024-12-18T03:35:50.6085565Z Entering 'third_party/NVTX'
2024-12-18T03:35:50.6199303Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T03:35:50.6305213Z Entering 'third_party/XNNPACK'
2024-12-18T03:35:50.6451480Z Entering 'third_party/benchmark'
2024-12-18T03:35:50.6559692Z Entering 'third_party/composable_kernel'
2024-12-18T03:35:50.6687980Z Entering 'third_party/cpp-httplib'
2024-12-18T03:35:50.6800544Z Entering 'third_party/cpuinfo'
2024-12-18T03:35:50.6911309Z Entering 'third_party/cudnn_frontend'
2024-12-18T03:35:50.7024694Z Entering 'third_party/cutlass'
2024-12-18T03:35:50.7153286Z Entering 'third_party/eigen'
2024-12-18T03:35:50.7264726Z Entering 'third_party/fbgemm'
2024-12-18T03:35:50.7371708Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:35:50.7486365Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:35:50.7558237Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:35:50.7680175Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:35:50.7784005Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:35:50.7892868Z Entering 'third_party/flatbuffers'
2024-12-18T03:35:50.8009576Z Entering 'third_party/fmt'
2024-12-18T03:35:50.8119900Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:35:50.8229089Z Entering 'third_party/gloo'
2024-12-18T03:35:50.8339288Z Entering 'third_party/googletest'
2024-12-18T03:35:50.8453673Z Entering 'third_party/ideep'
2024-12-18T03:35:50.8559028Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T03:35:50.8682835Z Entering 'third_party/ittapi'
2024-12-18T03:35:50.8792572Z Entering 'third_party/kineto'
2024-12-18T03:35:50.8897074Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:35:50.8998319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:35:50.9104341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:35:50.9206510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:35:50.9309109Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:35:50.9412193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:35:50.9524506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:35:50.9630947Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:35:50.9730881Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:35:50.9836094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:35:50.9945659Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:35:51.0048842Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:35:51.0160634Z Entering 'third_party/mimalloc'
2024-12-18T03:35:51.0271727Z Entering 'third_party/nccl/nccl'
2024-12-18T03:35:51.0378865Z Entering 'third_party/nlohmann'
2024-12-18T03:35:51.0488233Z Entering 'third_party/onnx'
2024-12-18T03:35:51.0627497Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T03:35:51.0746037Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T03:35:51.0856173Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:35:51.0961109Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:35:51.1064706Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:35:51.1169584Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:35:51.1282550Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:35:51.1396112Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:35:51.1489185Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:35:51.1583709Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:51.1695757Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:51.1807975Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:35:51.1961025Z Entering 'third_party/pocketfft'
2024-12-18T03:35:51.2068683Z Entering 'third_party/protobuf'
2024-12-18T03:35:51.2184293Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:51.2289228Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:51.2399687Z Entering 'third_party/psimd'
2024-12-18T03:35:51.2511299Z Entering 'third_party/pthreadpool'
2024-12-18T03:35:51.2617717Z Entering 'third_party/pybind11'
2024-12-18T03:35:51.2723360Z Entering 'third_party/python-peachpy'
2024-12-18T03:35:51.2832355Z Entering 'third_party/sleef'
2024-12-18T03:35:51.2939616Z Entering 'third_party/tensorpipe'
2024-12-18T03:35:51.3044636Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:51.3155176Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:51.3254989Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:51.3359057Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:51.3461713Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:51.3609808Z ##[endgroup]
2024-12-18T03:35:51.3615850Z ##[group]Persisting credentials for submodules
2024-12-18T03:35:51.3618971Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-12-18T03:35:51.4263605Z Entering 'android/libs/fbjni'
2024-12-18T03:35:51.4393428Z Entering 'third_party/FP16'
2024-12-18T03:35:51.4518225Z Entering 'third_party/FXdiv'
2024-12-18T03:35:51.4638563Z Entering 'third_party/NNPACK'
2024-12-18T03:35:51.4759383Z Entering 'third_party/NVTX'
2024-12-18T03:35:51.4884507Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T03:35:51.5006137Z Entering 'third_party/XNNPACK'
2024-12-18T03:35:51.5164582Z Entering 'third_party/benchmark'
2024-12-18T03:35:51.5286477Z Entering 'third_party/composable_kernel'
2024-12-18T03:35:51.5420448Z Entering 'third_party/cpp-httplib'
2024-12-18T03:35:51.5541246Z Entering 'third_party/cpuinfo'
2024-12-18T03:35:51.5663260Z Entering 'third_party/cudnn_frontend'
2024-12-18T03:35:51.5778880Z Entering 'third_party/cutlass'
2024-12-18T03:35:51.5917134Z Entering 'third_party/eigen'
2024-12-18T03:35:51.6040235Z Entering 'third_party/fbgemm'
2024-12-18T03:35:51.6156533Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:35:51.6275560Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:35:51.6392745Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:35:51.6527003Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:35:51.6645784Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:35:51.6766320Z Entering 'third_party/flatbuffers'
2024-12-18T03:35:51.6892638Z Entering 'third_party/fmt'
2024-12-18T03:35:51.7010050Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:35:51.7129349Z Entering 'third_party/gloo'
2024-12-18T03:35:51.7248043Z Entering 'third_party/googletest'
2024-12-18T03:35:51.7369120Z Entering 'third_party/ideep'
2024-12-18T03:35:51.7483170Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T03:35:51.7615425Z Entering 'third_party/ittapi'
2024-12-18T03:35:51.7728987Z Entering 'third_party/kineto'
2024-12-18T03:35:51.7845590Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:35:51.7958251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:35:51.8077327Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:35:51.8191532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:35:51.8306401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:35:51.8416481Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:35:51.8542664Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:35:51.8659301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:35:51.8775988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:35:51.8893656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:35:51.9016505Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:35:51.9134512Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:35:51.9258396Z Entering 'third_party/mimalloc'
2024-12-18T03:35:51.9378783Z Entering 'third_party/nccl/nccl'
2024-12-18T03:35:51.9497545Z Entering 'third_party/nlohmann'
2024-12-18T03:35:51.9620206Z Entering 'third_party/onnx'
2024-12-18T03:35:51.9769875Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T03:35:51.9900508Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T03:35:52.0020536Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:35:52.0133988Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:35:52.0250859Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:35:52.0366144Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:35:52.0483400Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:35:52.0597331Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:35:52.0714133Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:35:52.0822100Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:52.0947018Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:52.1078569Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:35:52.1243817Z Entering 'third_party/pocketfft'
2024-12-18T03:35:52.1365909Z Entering 'third_party/protobuf'
2024-12-18T03:35:52.1490041Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:52.1608202Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:52.1733185Z Entering 'third_party/psimd'
2024-12-18T03:35:52.1852825Z Entering 'third_party/pthreadpool'
2024-12-18T03:35:52.1974931Z Entering 'third_party/pybind11'
2024-12-18T03:35:52.2098820Z Entering 'third_party/python-peachpy'
2024-12-18T03:35:52.2218869Z Entering 'third_party/sleef'
2024-12-18T03:35:52.2342953Z Entering 'third_party/tensorpipe'
2024-12-18T03:35:52.2460134Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:52.2576146Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:52.2691203Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:52.2808605Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:52.2921211Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:52.3083396Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-12-18T03:35:52.3753597Z Entering 'android/libs/fbjni'
2024-12-18T03:35:52.3854249Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-12-18T03:35:52.3914931Z Entering 'third_party/FP16'
2024-12-18T03:35:52.4016792Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-12-18T03:35:52.4076909Z Entering 'third_party/FXdiv'
2024-12-18T03:35:52.4184620Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-12-18T03:35:52.4242745Z Entering 'third_party/NNPACK'
2024-12-18T03:35:52.4344919Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-12-18T03:35:52.4401917Z Entering 'third_party/NVTX'
2024-12-18T03:35:52.4504270Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2024-12-18T03:35:52.4562777Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T03:35:52.4663868Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-12-18T03:35:52.4721938Z Entering 'third_party/XNNPACK'
2024-12-18T03:35:52.4824711Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-12-18T03:35:52.4918077Z Entering 'third_party/benchmark'
2024-12-18T03:35:52.5022449Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T03:35:52.5081503Z Entering 'third_party/composable_kernel'
2024-12-18T03:35:52.5191823Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2024-12-18T03:35:52.5262567Z Entering 'third_party/cpp-httplib'
2024-12-18T03:35:52.5366927Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-12-18T03:35:52.5423757Z Entering 'third_party/cpuinfo'
2024-12-18T03:35:52.5525264Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T03:35:52.5583886Z Entering 'third_party/cudnn_frontend'
2024-12-18T03:35:52.5687503Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-12-18T03:35:52.5744797Z Entering 'third_party/cutlass'
2024-12-18T03:35:52.5852128Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T03:35:52.5929258Z Entering 'third_party/eigen'
2024-12-18T03:35:52.6031758Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-12-18T03:35:52.6094035Z Entering 'third_party/fbgemm'
2024-12-18T03:35:52.6198039Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-12-18T03:35:52.6250786Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:35:52.6350938Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-12-18T03:35:52.6405183Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:35:52.6511198Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T03:35:52.6565103Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:35:52.6665206Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T03:35:52.6735216Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:35:52.6834542Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:52.6888128Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:35:52.6989825Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-12-18T03:35:52.7049680Z Entering 'third_party/flatbuffers'
2024-12-18T03:35:52.7152534Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-12-18T03:35:52.7214478Z Entering 'third_party/fmt'
2024-12-18T03:35:52.7316347Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-12-18T03:35:52.7372151Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:35:52.7473752Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-12-18T03:35:52.7529541Z Entering 'third_party/gloo'
2024-12-18T03:35:52.7628517Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-12-18T03:35:52.7685500Z Entering 'third_party/googletest'
2024-12-18T03:35:52.7785489Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:52.7841230Z Entering 'third_party/ideep'
2024-12-18T03:35:52.7939952Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-12-18T03:35:52.7990974Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T03:35:52.8091220Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-12-18T03:35:52.8166964Z Entering 'third_party/ittapi'
2024-12-18T03:35:52.8268317Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-12-18T03:35:52.8325786Z Entering 'third_party/kineto'
2024-12-18T03:35:52.8426140Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-12-18T03:35:52.8482853Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:35:52.8583063Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-12-18T03:35:52.8634920Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:35:52.8736321Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-12-18T03:35:52.8796777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:35:52.8894997Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-12-18T03:35:52.8950230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:35:52.9050040Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-12-18T03:35:52.9104165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:35:52.9204898Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-12-18T03:35:52.9254390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:35:52.9356187Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-12-18T03:35:52.9416264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:35:52.9517120Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-12-18T03:35:52.9572839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:35:52.9674220Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:52.9729827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:35:52.9830453Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-12-18T03:35:52.9888092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:35:52.9986209Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-12-18T03:35:53.0049424Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:35:53.0149991Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-12-18T03:35:53.0204314Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:35:53.0301622Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:53.0365387Z Entering 'third_party/mimalloc'
2024-12-18T03:35:53.0465157Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-12-18T03:35:53.0521278Z Entering 'third_party/nccl/nccl'
2024-12-18T03:35:53.0622440Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-12-18T03:35:53.0678695Z Entering 'third_party/nlohmann'
2024-12-18T03:35:53.0781305Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-12-18T03:35:53.0839905Z Entering 'third_party/onnx'
2024-12-18T03:35:53.0940432Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-12-18T03:35:53.1023501Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T03:35:53.1123413Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T03:35:53.1189603Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T03:35:53.1292200Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-12-18T03:35:53.1347904Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:35:53.1447163Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T03:35:53.1499105Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:35:53.1599928Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:53.1653660Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:35:53.1754195Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-12-18T03:35:53.1805977Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:35:53.1904555Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-12-18T03:35:53.1961092Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:35:53.2059715Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-12-18T03:35:53.2111462Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:35:53.2207911Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-12-18T03:35:53.2262833Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:35:53.2362051Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-12-18T03:35:53.2409806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:53.2508570Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-12-18T03:35:53.2567595Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:53.2666188Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-12-18T03:35:53.2726787Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:35:53.2825436Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-12-18T03:35:53.2926871Z Entering 'third_party/pocketfft'
2024-12-18T03:35:53.3027891Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-12-18T03:35:53.3084295Z Entering 'third_party/protobuf'
2024-12-18T03:35:53.3183831Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-12-18T03:35:53.3243247Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:53.3343305Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T03:35:53.3397169Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:53.3495824Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:53.3557626Z Entering 'third_party/psimd'
2024-12-18T03:35:53.3659967Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-12-18T03:35:53.3717704Z Entering 'third_party/pthreadpool'
2024-12-18T03:35:53.3817380Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-12-18T03:35:53.3873647Z Entering 'third_party/pybind11'
2024-12-18T03:35:53.3977520Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T03:35:53.4034897Z Entering 'third_party/python-peachpy'
2024-12-18T03:35:53.4135331Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-12-18T03:35:53.4194724Z Entering 'third_party/sleef'
2024-12-18T03:35:53.4294970Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-12-18T03:35:53.4354353Z Entering 'third_party/tensorpipe'
2024-12-18T03:35:53.4457972Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-12-18T03:35:53.4511833Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:53.4610382Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-12-18T03:35:53.4665775Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:53.4763910Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-12-18T03:35:53.4818166Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:53.4915916Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-12-18T03:35:53.4972437Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:53.5070804Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T03:35:53.5121260Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:53.5220585Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-12-18T03:35:53.5761800Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-12-18T03:35:53.6442270Z Entering 'android/libs/fbjni'
2024-12-18T03:35:53.6556093Z Entering 'third_party/FP16'
2024-12-18T03:35:53.6666343Z Entering 'third_party/FXdiv'
2024-12-18T03:35:53.6773932Z Entering 'third_party/NNPACK'
2024-12-18T03:35:53.6881673Z Entering 'third_party/NVTX'
2024-12-18T03:35:53.6990109Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T03:35:53.7097368Z Entering 'third_party/XNNPACK'
2024-12-18T03:35:53.7236858Z Entering 'third_party/benchmark'
2024-12-18T03:35:53.7342757Z Entering 'third_party/composable_kernel'
2024-12-18T03:35:53.7465050Z Entering 'third_party/cpp-httplib'
2024-12-18T03:35:53.7576297Z Entering 'third_party/cpuinfo'
2024-12-18T03:35:53.7685405Z Entering 'third_party/cudnn_frontend'
2024-12-18T03:35:53.7791420Z Entering 'third_party/cutlass'
2024-12-18T03:35:53.7914464Z Entering 'third_party/eigen'
2024-12-18T03:35:53.8027067Z Entering 'third_party/fbgemm'
2024-12-18T03:35:53.8131137Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:35:53.8230898Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:35:53.8332030Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:35:53.8449124Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:35:53.8549061Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:35:53.8657049Z Entering 'third_party/flatbuffers'
2024-12-18T03:35:53.8778156Z Entering 'third_party/fmt'
2024-12-18T03:35:53.8887445Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:35:53.8998335Z Entering 'third_party/gloo'
2024-12-18T03:35:53.9108177Z Entering 'third_party/googletest'
2024-12-18T03:35:53.9217477Z Entering 'third_party/ideep'
2024-12-18T03:35:53.9323928Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T03:35:53.9447920Z Entering 'third_party/ittapi'
2024-12-18T03:35:53.9556008Z Entering 'third_party/kineto'
2024-12-18T03:35:53.9659759Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:35:53.9758744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:35:53.9865496Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:35:53.9970076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:35:54.0073746Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:35:54.0168921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:35:54.0277859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:35:54.0378694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:35:54.0476939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:35:54.0579105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:35:54.0684852Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:35:54.0784585Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:35:54.0890651Z Entering 'third_party/mimalloc'
2024-12-18T03:35:54.0995658Z Entering 'third_party/nccl/nccl'
2024-12-18T03:35:54.1101819Z Entering 'third_party/nlohmann'
2024-12-18T03:35:54.1210750Z Entering 'third_party/onnx'
2024-12-18T03:35:54.1349603Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T03:35:54.1460741Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T03:35:54.1568770Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:35:54.1672059Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:35:54.1775242Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:35:54.1876624Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:35:54.1987820Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:35:54.2084375Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:35:54.2182824Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:35:54.2278393Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:54.2380687Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:54.2486268Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:35:54.2633264Z Entering 'third_party/pocketfft'
2024-12-18T03:35:54.2742834Z Entering 'third_party/protobuf'
2024-12-18T03:35:54.2851982Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:54.2954216Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:54.3061403Z Entering 'third_party/psimd'
2024-12-18T03:35:54.3169181Z Entering 'third_party/pthreadpool'
2024-12-18T03:35:54.3275572Z Entering 'third_party/pybind11'
2024-12-18T03:35:54.3380901Z Entering 'third_party/python-peachpy'
2024-12-18T03:35:54.3487062Z Entering 'third_party/sleef'
2024-12-18T03:35:54.3591293Z Entering 'third_party/tensorpipe'
2024-12-18T03:35:54.3694633Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:54.3791582Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:54.3889604Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:54.3985709Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:54.4079222Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:54.4224674Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-12-18T03:35:54.4864560Z Entering 'android/libs/fbjni'
2024-12-18T03:35:54.4975628Z Entering 'third_party/FP16'
2024-12-18T03:35:54.5088178Z Entering 'third_party/FXdiv'
2024-12-18T03:35:54.5196206Z Entering 'third_party/NNPACK'
2024-12-18T03:35:54.5303014Z Entering 'third_party/NVTX'
2024-12-18T03:35:54.5409754Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T03:35:54.5520330Z Entering 'third_party/XNNPACK'
2024-12-18T03:35:54.5657464Z Entering 'third_party/benchmark'
2024-12-18T03:35:54.5761758Z Entering 'third_party/composable_kernel'
2024-12-18T03:35:54.5880092Z Entering 'third_party/cpp-httplib'
2024-12-18T03:35:54.5982624Z Entering 'third_party/cpuinfo'
2024-12-18T03:35:54.6086143Z Entering 'third_party/cudnn_frontend'
2024-12-18T03:35:54.6190861Z Entering 'third_party/cutlass'
2024-12-18T03:35:54.6310644Z Entering 'third_party/eigen'
2024-12-18T03:35:54.6417068Z Entering 'third_party/fbgemm'
2024-12-18T03:35:54.6519287Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T03:35:54.6619535Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T03:35:54.6723153Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T03:35:54.6839709Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T03:35:54.6940697Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T03:35:54.7046109Z Entering 'third_party/flatbuffers'
2024-12-18T03:35:54.7157285Z Entering 'third_party/fmt'
2024-12-18T03:35:54.7263520Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T03:35:54.7367878Z Entering 'third_party/gloo'
2024-12-18T03:35:54.7476591Z Entering 'third_party/googletest'
2024-12-18T03:35:54.7582296Z Entering 'third_party/ideep'
2024-12-18T03:35:54.7680734Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T03:35:54.7800475Z Entering 'third_party/ittapi'
2024-12-18T03:35:54.7903498Z Entering 'third_party/kineto'
2024-12-18T03:35:54.8003230Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T03:35:54.8099755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T03:35:54.8205491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T03:35:54.8302035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T03:35:54.8401871Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T03:35:54.8496174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T03:35:54.8600538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T03:35:54.8700539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T03:35:54.8799339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T03:35:54.8899139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T03:35:54.9006026Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T03:35:54.9105082Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T03:35:54.9211495Z Entering 'third_party/mimalloc'
2024-12-18T03:35:54.9316396Z Entering 'third_party/nccl/nccl'
2024-12-18T03:35:54.9421162Z Entering 'third_party/nlohmann'
2024-12-18T03:35:54.9532115Z Entering 'third_party/onnx'
2024-12-18T03:35:54.9663576Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T03:35:54.9779509Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T03:35:54.9882842Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T03:35:54.9981198Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T03:35:55.0078685Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T03:35:55.0174192Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T03:35:55.0274938Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T03:35:55.0370779Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T03:35:55.0469811Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T03:35:55.0561881Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T03:35:55.0665418Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T03:35:55.0769208Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T03:35:55.0912963Z Entering 'third_party/pocketfft'
2024-12-18T03:35:55.1017509Z Entering 'third_party/protobuf'
2024-12-18T03:35:55.1125417Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T03:35:55.1226197Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T03:35:55.1332643Z Entering 'third_party/psimd'
2024-12-18T03:35:55.1435231Z Entering 'third_party/pthreadpool'
2024-12-18T03:35:55.1540212Z Entering 'third_party/pybind11'
2024-12-18T03:35:55.1644767Z Entering 'third_party/python-peachpy'
2024-12-18T03:35:55.1755094Z Entering 'third_party/sleef'
2024-12-18T03:35:55.1857674Z Entering 'third_party/tensorpipe'
2024-12-18T03:35:55.1957691Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T03:35:55.2056660Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T03:35:55.2155085Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T03:35:55.2256228Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T03:35:55.2350527Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T03:35:55.2495726Z ##[endgroup]
2024-12-18T03:35:55.2575038Z [command]/usr/bin/git log -1 --format='%H'
2024-12-18T03:35:55.2634292Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7'
2024-12-18T03:35:55.2961616Z Prepare all required actions
2024-12-18T03:35:55.2962506Z Getting action download info
2024-12-18T03:35:55.3070002Z ##[group]Run ./.github/actions/setup-rocm
2024-12-18T03:35:55.3070482Z env:
2024-12-18T03:35:55.3070807Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.3071195Z ##[endgroup]
2024-12-18T03:35:55.3107833Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}"
2024-12-18T03:35:55.3108803Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}"
2024-12-18T03:35:55.3156980Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.3157573Z env:
2024-12-18T03:35:55.3157902Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.3158283Z ##[endgroup]
2024-12-18T03:35:55.3327316Z ##[group]Run set -ex
2024-12-18T03:35:55.3327788Z set -ex
2024-12-18T03:35:55.3328191Z 
2024-12-18T03:35:55.3328614Z cat ~/.docker/config.json || true
2024-12-18T03:35:55.3330142Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not
2024-12-18T03:35:55.3331761Z rm -f ~/.docker/config.json
2024-12-18T03:35:55.3383643Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.3384330Z env:
2024-12-18T03:35:55.3384718Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.3385293Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.3385927Z ##[endgroup]
2024-12-18T03:35:55.3481901Z + cat /var/home/pytorchci/.docker/config.json
2024-12-18T03:35:55.3505143Z {
2024-12-18T03:35:55.3505498Z 	"auths": {}
2024-12-18T03:35:55.3510392Z + rm -f /var/home/pytorchci/.docker/config.json
2024-12-18T03:35:55.3544132Z }
2024-12-18T03:35:55.3579121Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T03:35:55.3579967Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T03:35:55.3580597Z # shellcheck disable=SC2046
2024-12-18T03:35:55.3581117Z docker stop $(docker ps -q) || true
2024-12-18T03:35:55.3581639Z # Prune all stopped containers.
2024-12-18T03:35:55.3582120Z docker container prune -f
2024-12-18T03:35:55.3626465Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.3627236Z env:
2024-12-18T03:35:55.3627567Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.3628071Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.3628693Z ##[endgroup]
2024-12-18T03:35:55.4368482Z "docker stop" requires at least 1 argument.
2024-12-18T03:35:55.4369121Z See 'docker stop --help'.
2024-12-18T03:35:55.4369432Z 
2024-12-18T03:35:55.4370025Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-12-18T03:35:55.4370735Z 
2024-12-18T03:35:55.4371187Z Stop one or more running containers
2024-12-18T03:35:55.4742567Z Total reclaimed space: 0B
2024-12-18T03:35:55.4833857Z ##[group]Run cat /etc/os-release || true
2024-12-18T03:35:55.4834545Z cat /etc/os-release || true
2024-12-18T03:35:55.4835175Z cat /etc/apt/sources.list.d/rocm.list || true
2024-12-18T03:35:55.4835827Z cat /opt/rocm/.info/version || true
2024-12-18T03:35:55.4836367Z whoami
2024-12-18T03:35:55.4894294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.4894980Z env:
2024-12-18T03:35:55.4895360Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.4895920Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.4896560Z ##[endgroup]
2024-12-18T03:35:55.5013114Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2024-12-18T03:35:55.5013726Z NAME="Ubuntu"
2024-12-18T03:35:55.5014181Z VERSION_ID="22.04"
2024-12-18T03:35:55.5014670Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2024-12-18T03:35:55.5015249Z VERSION_CODENAME=jammy
2024-12-18T03:35:55.5015697Z ID=ubuntu
2024-12-18T03:35:55.5016066Z ID_LIKE=debian
2024-12-18T03:35:55.5016534Z HOME_URL="https://www.ubuntu.com/"
2024-12-18T03:35:55.5017888Z SUPPORT_URL="https://help.ubuntu.com/"
2024-12-18T03:35:55.5018653Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2024-12-18T03:35:55.5019635Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2024-12-18T03:35:55.5020508Z UBUNTU_CODENAME=jammy
2024-12-18T03:35:55.5041798Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.2.1 jammy main
2024-12-18T03:35:55.5069598Z 6.2.1-112
2024-12-18T03:35:55.5114475Z pytorchci
2024-12-18T03:35:55.5167167Z ##[group]Run rocm-smi
2024-12-18T03:35:55.5167692Z rocm-smi
2024-12-18T03:35:55.5225547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.5226254Z env:
2024-12-18T03:35:55.5226995Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.5227607Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.5228629Z ##[endgroup]
2024-12-18T03:35:55.7235240Z 
2024-12-18T03:35:55.7235439Z 
2024-12-18T03:35:55.7236561Z ========================================= ROCm System Management Interface =========================================
2024-12-18T03:35:55.7238150Z =================================================== Concise Info ===================================================
2024-12-18T03:35:55.7242210Z Device  Node  IDs              Temp    Power   Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2024-12-18T03:35:55.7244700Z               (DID,     GUID)  (Edge)  (Avg)   (Mem, Compute, ID)                                                   
2024-12-18T03:35:55.7245705Z ====================================================================================================================
2024-12-18T03:35:55.7248238Z 0       4     0x740c,   57586  43.0°C  94.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2024-12-18T03:35:55.7249744Z 1       5     0x740c,   45873  41.0°C  N/A     N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2024-12-18T03:35:55.7251446Z 2       2     0x740c,   51627  39.0°C  95.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2024-12-18T03:35:55.7252823Z 3       3     0x740c,   64489  38.0°C  N/A     N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2024-12-18T03:35:55.7254250Z 4       8     0x740c,   30939  44.0°C  97.0W   N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2024-12-18T03:35:55.7255891Z 5       9     0x740c,   8466   40.0°C  N/A     N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2024-12-18T03:35:55.7257212Z 6       6     0x740c,   41154  43.0°C  102.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2024-12-18T03:35:55.7258638Z 7       7     0x740c,   63755  37.0°C  N/A     N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2024-12-18T03:35:55.7259984Z ====================================================================================================================
2024-12-18T03:35:55.7260998Z =============================================== End of ROCm SMI Log ================================================
2024-12-18T03:35:55.7413661Z ##[group]Run rocminfo
2024-12-18T03:35:55.7414431Z rocminfo
2024-12-18T03:35:55.7472386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.7473591Z env:
2024-12-18T03:35:55.7474244Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.7474985Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.7476075Z ##[endgroup]
2024-12-18T03:35:55.9007259Z ROCk module version 6.8.5 is loaded
2024-12-18T03:35:55.9008111Z =====================    
2024-12-18T03:35:55.9008638Z HSA System Attributes    
2024-12-18T03:35:55.9009108Z =====================    
2024-12-18T03:35:55.9009552Z Runtime Version:         1.14
2024-12-18T03:35:55.9010041Z Runtime Ext Version:     1.6
2024-12-18T03:35:55.9010558Z System Timestamp Freq.:  1000.000000MHz
2024-12-18T03:35:55.9011436Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2024-12-18T03:35:55.9013273Z Machine Model:           LARGE                              
2024-12-18T03:35:55.9014082Z System Endianness:       LITTLE                             
2024-12-18T03:35:55.9014731Z Mwaitx:                  DISABLED
2024-12-18T03:35:55.9015254Z DMAbuf Support:          YES
2024-12-18T03:35:55.9015595Z 
2024-12-18T03:35:55.9015757Z ==========               
2024-12-18T03:35:55.9016207Z HSA Agents               
2024-12-18T03:35:55.9016632Z ==========               
2024-12-18T03:35:55.9017048Z *******                  
2024-12-18T03:35:55.9017489Z Agent 1                  
2024-12-18T03:35:55.9017903Z *******                  
2024-12-18T03:35:55.9018507Z   Name:                    AMD EPYC 7713 64-Core Processor    
2024-12-18T03:35:55.9019245Z   Uuid:                    CPU-XX                             
2024-12-18T03:35:55.9019946Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:35:55.9020673Z   Vendor Name:             CPU                                
2024-12-18T03:35:55.9021359Z   Feature:                 None specified                     
2024-12-18T03:35:55.9022033Z   Profile:                 FULL_PROFILE                       
2024-12-18T03:35:55.9022789Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9023518Z   Max Queue Number:        0(0x0)                             
2024-12-18T03:35:55.9024199Z   Queue Min Size:          0(0x0)                             
2024-12-18T03:35:55.9024870Z   Queue Max Size:          0(0x0)                             
2024-12-18T03:35:55.9025942Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9026568Z   Node:                    0                                  
2024-12-18T03:35:55.9027373Z   Device Type:             CPU                                
2024-12-18T03:35:55.9027986Z   Cache Info:              
2024-12-18T03:35:55.9028494Z     L1:                      32768(0x8000) KB                   
2024-12-18T03:35:55.9029126Z   Chip ID:                 0(0x0)                             
2024-12-18T03:35:55.9029805Z   ASIC Revision:           0(0x0)                             
2024-12-18T03:35:55.9030498Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9031198Z   Max Clock Freq. (MHz):   2000                               
2024-12-18T03:35:55.9031850Z   BDFID:                   0                                  
2024-12-18T03:35:55.9032495Z   Internal Node ID:        0                                  
2024-12-18T03:35:55.9033179Z   Compute Unit:            64                                 
2024-12-18T03:35:55.9033854Z   SIMDs per CU:            0                                  
2024-12-18T03:35:55.9034537Z   Shader Engines:          0                                  
2024-12-18T03:35:55.9035234Z   Shader Arrs. per Eng.:   0                                  
2024-12-18T03:35:55.9035956Z   WatchPts on Addr. Ranges:1                                  
2024-12-18T03:35:55.9036603Z   Memory Properties:       
2024-12-18T03:35:55.9037080Z   Features:                None
2024-12-18T03:35:55.9037564Z   Pool Info:               
2024-12-18T03:35:55.9038013Z     Pool 1                   
2024-12-18T03:35:55.9038607Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9039309Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:35:55.9039989Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9040689Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9041584Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9042493Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9043338Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9043967Z     Pool 2                   
2024-12-18T03:35:55.9044543Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2024-12-18T03:35:55.9045569Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:35:55.9046251Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9046956Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9047682Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9048430Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9049153Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9049777Z     Pool 3                   
2024-12-18T03:35:55.9050344Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9051022Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:35:55.9051683Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9052393Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9053184Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9053942Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9054672Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9055307Z   ISA Info:                
2024-12-18T03:35:55.9055744Z *******                  
2024-12-18T03:35:55.9056182Z Agent 2                  
2024-12-18T03:35:55.9056609Z *******                  
2024-12-18T03:35:55.9057115Z   Name:                    AMD EPYC 7713 64-Core Processor    
2024-12-18T03:35:55.9058113Z   Uuid:                    CPU-XX                             
2024-12-18T03:35:55.9058810Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:35:55.9059526Z   Vendor Name:             CPU                                
2024-12-18T03:35:55.9060210Z   Feature:                 None specified                     
2024-12-18T03:35:55.9061012Z   Profile:                 FULL_PROFILE                       
2024-12-18T03:35:55.9061858Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9062692Z   Max Queue Number:        0(0x0)                             
2024-12-18T03:35:55.9063504Z   Queue Min Size:          0(0x0)                             
2024-12-18T03:35:55.9064276Z   Queue Max Size:          0(0x0)                             
2024-12-18T03:35:55.9064956Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9065590Z   Node:                    1                                  
2024-12-18T03:35:55.9066250Z   Device Type:             CPU                                
2024-12-18T03:35:55.9067155Z   Cache Info:              
2024-12-18T03:35:55.9067653Z     L1:                      32768(0x8000) KB                   
2024-12-18T03:35:55.9068273Z   Chip ID:                 0(0x0)                             
2024-12-18T03:35:55.9068967Z   ASIC Revision:           0(0x0)                             
2024-12-18T03:35:55.9069797Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9070629Z   Max Clock Freq. (MHz):   2000                               
2024-12-18T03:35:55.9071370Z   BDFID:                   0                                  
2024-12-18T03:35:55.9072024Z   Internal Node ID:        1                                  
2024-12-18T03:35:55.9072720Z   Compute Unit:            64                                 
2024-12-18T03:35:55.9073413Z   SIMDs per CU:            0                                  
2024-12-18T03:35:55.9074091Z   Shader Engines:          0                                  
2024-12-18T03:35:55.9074835Z   Shader Arrs. per Eng.:   0                                  
2024-12-18T03:35:55.9075707Z   WatchPts on Addr. Ranges:1                                  
2024-12-18T03:35:55.9076461Z   Memory Properties:       
2024-12-18T03:35:55.9077025Z   Features:                None
2024-12-18T03:35:55.9077593Z   Pool Info:               
2024-12-18T03:35:55.9078136Z     Pool 1                   
2024-12-18T03:35:55.9079239Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9080050Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:35:55.9080734Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9081448Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9082201Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9082953Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9083677Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9084309Z     Pool 2                   
2024-12-18T03:35:55.9084890Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2024-12-18T03:35:55.9085705Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:35:55.9086499Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9087635Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9088387Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9089154Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9089882Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9090502Z     Pool 3                   
2024-12-18T03:35:55.9091072Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9092096Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:35:55.9092784Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9093640Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9094520Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:35:55.9095413Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9096287Z       Accessible by all:       TRUE                               
2024-12-18T03:35:55.9097042Z   ISA Info:                
2024-12-18T03:35:55.9097559Z *******                  
2024-12-18T03:35:55.9098004Z Agent 3                  
2024-12-18T03:35:55.9098439Z *******                  
2024-12-18T03:35:55.9098915Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9099565Z   Uuid:                    GPU-72eb622bef58f205               
2024-12-18T03:35:55.9100272Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9100994Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9101686Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9102378Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9103072Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9104052Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9104752Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9105424Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9106100Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9106851Z   Node:                    2                                  
2024-12-18T03:35:55.9107494Z   Device Type:             GPU                                
2024-12-18T03:35:55.9108094Z   Cache Info:              
2024-12-18T03:35:55.9108598Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9109194Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9109793Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9110454Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9111153Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9112185Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9112858Z   BDFID:                   12800                              
2024-12-18T03:35:55.9113522Z   Internal Node ID:        2                                  
2024-12-18T03:35:55.9114209Z   Compute Unit:            104                                
2024-12-18T03:35:55.9114895Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9115580Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9116293Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9117026Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9117761Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9118403Z   Memory Properties:       
2024-12-18T03:35:55.9118914Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9119577Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9120296Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9121064Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9121842Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9122488Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9123169Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9123763Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9124783Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9125495Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9126195Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9126806Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9127327Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9127920Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9128499Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9129178Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9137703Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9138582Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9139333Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9139968Z   Pool Info:               
2024-12-18T03:35:55.9140481Z     Pool 1                   
2024-12-18T03:35:55.9141090Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9141874Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9142821Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9143963Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9145038Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9145961Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9147516Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9148529Z     Pool 2                   
2024-12-18T03:35:55.9149370Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9150768Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9151831Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9152777Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9153902Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9154907Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9156006Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9157189Z     Pool 3                   
2024-12-18T03:35:55.9158665Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9159807Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9160836Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9161791Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9162783Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9163864Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9164867Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9165868Z     Pool 4                   
2024-12-18T03:35:55.9167061Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9168130Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9169056Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9170196Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9171094Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9172087Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9173230Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9174064Z   ISA Info:                
2024-12-18T03:35:55.9174729Z     ISA 1                    
2024-12-18T03:35:55.9176134Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9177125Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9178083Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9179430Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9180350Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9181273Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9182447Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9183286Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9184093Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9185156Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9186016Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9186912Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9188079Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9188899Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9189744Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9191155Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9192155Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9192900Z *******                  
2024-12-18T03:35:55.9193814Z Agent 4                  
2024-12-18T03:35:55.9194494Z *******                  
2024-12-18T03:35:55.9195112Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9196428Z   Uuid:                    GPU-2a0632e13f52b1f3               
2024-12-18T03:35:55.9197573Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9198618Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9199956Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9200934Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9201922Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9203064Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9204008Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9205428Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9206522Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9207639Z   Node:                    3                                  
2024-12-18T03:35:55.9209194Z   Device Type:             GPU                                
2024-12-18T03:35:55.9209974Z   Cache Info:              
2024-12-18T03:35:55.9211072Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9211955Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9212693Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9214017Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9215160Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9216141Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9217429Z   BDFID:                   13568                              
2024-12-18T03:35:55.9218349Z   Internal Node ID:        3                                  
2024-12-18T03:35:55.9219234Z   Compute Unit:            104                                
2024-12-18T03:35:55.9220387Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9221325Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9222422Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9223760Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9224826Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9225844Z   Memory Properties:       
2024-12-18T03:35:55.9226823Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9227807Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9229090Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9230194Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9231179Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9232121Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9232932Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9233803Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9234787Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9235821Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9237016Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9238326Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9239231Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9240227Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9241181Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9242192Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9243135Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9244220Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9245275Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9246158Z   Pool Info:               
2024-12-18T03:35:55.9247107Z     Pool 1                   
2024-12-18T03:35:55.9248222Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9249133Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9250292Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9251368Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9252283Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9253788Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9254858Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9255662Z     Pool 2                   
2024-12-18T03:35:55.9256669Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9257695Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9258700Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9259690Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9260760Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9261955Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9262946Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9263890Z     Pool 3                   
2024-12-18T03:35:55.9264764Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9265693Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9266998Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9268084Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9269111Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9270185Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9271724Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9272796Z     Pool 4                   
2024-12-18T03:35:55.9273885Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9275113Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9276371Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9277358Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9278284Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9279277Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9280043Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9280801Z   ISA Info:                
2024-12-18T03:35:55.9281564Z     ISA 1                    
2024-12-18T03:35:55.9282308Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9283367Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9284426Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9285193Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9286174Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9287140Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9287891Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9288680Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9289544Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9290267Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9290890Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9291797Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9292567Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9293207Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9294062Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9294893Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9295633Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9296506Z *******                  
2024-12-18T03:35:55.9297572Z Agent 5                  
2024-12-18T03:35:55.9298057Z *******                  
2024-12-18T03:35:55.9298872Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9299679Z   Uuid:                    GPU-811789ff388472b0               
2024-12-18T03:35:55.9300387Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9301544Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9302556Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9303632Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9304604Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9305458Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9306393Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9307454Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9308412Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9309264Z   Node:                    4                                  
2024-12-18T03:35:55.9310023Z   Device Type:             GPU                                
2024-12-18T03:35:55.9310756Z   Cache Info:              
2024-12-18T03:35:55.9311272Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9311778Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9312606Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9313160Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9313826Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9314524Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9315168Z   BDFID:                   4352                               
2024-12-18T03:35:55.9315854Z   Internal Node ID:        4                                  
2024-12-18T03:35:55.9316543Z   Compute Unit:            104                                
2024-12-18T03:35:55.9317211Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9317894Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9318572Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9319198Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9319826Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9320374Z   Memory Properties:       
2024-12-18T03:35:55.9320801Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9321370Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9321981Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9322577Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9323140Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9323635Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9324205Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9324787Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9325455Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9326165Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9326812Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9327351Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9327786Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9328290Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9328789Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9329631Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9330293Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9330910Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9331511Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9332029Z   Pool Info:               
2024-12-18T03:35:55.9332428Z     Pool 1                   
2024-12-18T03:35:55.9332927Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9333522Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9334128Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9334737Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9335357Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9335997Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9336634Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9337164Z     Pool 2                   
2024-12-18T03:35:55.9337658Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9338256Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9338819Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9339413Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9340336Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9340966Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9341593Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9342141Z     Pool 3                   
2024-12-18T03:35:55.9342612Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9343190Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9343778Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9344371Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9345025Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9345668Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9346276Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9347061Z     Pool 4                   
2024-12-18T03:35:55.9347529Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9348087Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9348758Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9349502Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9350256Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9350905Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9351518Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9352065Z   ISA Info:                
2024-12-18T03:35:55.9352445Z     ISA 1                    
2024-12-18T03:35:55.9352946Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9353606Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9354245Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9354879Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9355576Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9356272Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9356990Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9358027Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9358618Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9359220Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9359805Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9360425Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9360975Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9361438Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9361931Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9362433Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9362993Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9363501Z *******                  
2024-12-18T03:35:55.9363875Z Agent 6                  
2024-12-18T03:35:55.9364254Z *******                  
2024-12-18T03:35:55.9364663Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9365228Z   Uuid:                    GPU-271d47bdd095d3f3               
2024-12-18T03:35:55.9365878Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9366584Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9367273Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9368317Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9368941Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9369541Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9370121Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9370676Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9371256Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9371797Z   Node:                    5                                  
2024-12-18T03:35:55.9372331Z   Device Type:             GPU                                
2024-12-18T03:35:55.9372846Z   Cache Info:              
2024-12-18T03:35:55.9373270Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9373772Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9374299Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9374886Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9375480Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9376057Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9376599Z   BDFID:                   5120                               
2024-12-18T03:35:55.9377155Z   Internal Node ID:        5                                  
2024-12-18T03:35:55.9377730Z   Compute Unit:            104                                
2024-12-18T03:35:55.9378298Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9378861Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9379445Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9380061Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9380673Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9381210Z   Memory Properties:       
2024-12-18T03:35:55.9381723Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9382360Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9383054Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9383753Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9384390Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9385222Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9385795Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9386355Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9387100Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9387787Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9388418Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9388952Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9389391Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9389916Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9390493Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9391150Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9391922Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9392640Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9393230Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9393750Z   Pool Info:               
2024-12-18T03:35:55.9394124Z     Pool 1                   
2024-12-18T03:35:55.9394607Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9395188Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9396061Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9396650Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9397271Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9397898Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9398506Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9399042Z     Pool 2                   
2024-12-18T03:35:55.9399520Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9400083Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9400642Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9401238Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9401844Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9402472Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9403082Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9403923Z     Pool 3                   
2024-12-18T03:35:55.9404554Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9405209Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9405858Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9406544Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9407217Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9407824Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9408427Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9408943Z     Pool 4                   
2024-12-18T03:35:55.9409388Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9409915Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9410466Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9411049Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9411655Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9412607Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9413219Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9413741Z   ISA Info:                
2024-12-18T03:35:55.9414122Z     ISA 1                    
2024-12-18T03:35:55.9414601Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9415242Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9415867Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9416494Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9417124Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9417717Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9418301Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9418854Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9419341Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9419827Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9420313Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9420843Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9421368Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9421811Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9422675Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9423174Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9423726Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9424244Z *******                  
2024-12-18T03:35:55.9424615Z Agent 7                  
2024-12-18T03:35:55.9424981Z *******                  
2024-12-18T03:35:55.9425415Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9425973Z   Uuid:                    GPU-ffdd23a7d4aa1894               
2024-12-18T03:35:55.9426553Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9427275Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9427861Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9428549Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9429255Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9429952Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9430590Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9431161Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9431721Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9432256Z   Node:                    6                                  
2024-12-18T03:35:55.9432805Z   Device Type:             GPU                                
2024-12-18T03:35:55.9433310Z   Cache Info:              
2024-12-18T03:35:55.9433715Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9434218Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9434724Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9435285Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9435970Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9436662Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9437299Z   BDFID:                   44544                              
2024-12-18T03:35:55.9437963Z   Internal Node ID:        6                                  
2024-12-18T03:35:55.9438645Z   Compute Unit:            104                                
2024-12-18T03:35:55.9439666Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9440345Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9440942Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9441553Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9442154Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9442696Z   Memory Properties:       
2024-12-18T03:35:55.9443133Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9443670Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9444269Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9444867Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9445412Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9445877Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9446367Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9446840Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9447363Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9447953Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9448539Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9449150Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9450021Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9450595Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9451175Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9451802Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9452448Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9453082Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9453685Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9454190Z   Pool Info:               
2024-12-18T03:35:55.9454569Z     Pool 1                   
2024-12-18T03:35:55.9455054Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9455631Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9456206Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9456815Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9457429Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9458066Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9458684Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9459222Z     Pool 2                   
2024-12-18T03:35:55.9459764Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9460517Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9461159Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9461846Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9462497Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9463137Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9463752Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9464271Z     Pool 3                   
2024-12-18T03:35:55.9464743Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9465298Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9465857Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9467262Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9467996Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9468664Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9469278Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9469807Z     Pool 4                   
2024-12-18T03:35:55.9470269Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9470820Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9471385Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9472087Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9472817Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9473554Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9474291Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9474906Z   ISA Info:                
2024-12-18T03:35:55.9475357Z     ISA 1                    
2024-12-18T03:35:55.9475948Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9476712Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9477448Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9478466Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9479095Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9479695Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9480291Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9480852Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9481343Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9481845Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9482318Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9482857Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9483402Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9483932Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9484531Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9485124Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9485778Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9486394Z *******                  
2024-12-18T03:35:55.9486832Z Agent 8                  
2024-12-18T03:35:55.9487187Z *******                  
2024-12-18T03:35:55.9487609Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9488163Z   Uuid:                    GPU-13cb7b90220db844               
2024-12-18T03:35:55.9488748Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9489350Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9489930Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9490504Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9491096Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9491696Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9492268Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9492833Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9493401Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9493933Z   Node:                    7                                  
2024-12-18T03:35:55.9494747Z   Device Type:             GPU                                
2024-12-18T03:35:55.9495268Z   Cache Info:              
2024-12-18T03:35:55.9495701Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9496199Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9496716Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9497280Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9497882Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9498495Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9499064Z   BDFID:                   45824                              
2024-12-18T03:35:55.9499617Z   Internal Node ID:        7                                  
2024-12-18T03:35:55.9500200Z   Compute Unit:            104                                
2024-12-18T03:35:55.9500779Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9501350Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9501958Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9502571Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9503183Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9503730Z   Memory Properties:       
2024-12-18T03:35:55.9504167Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9505000Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9505614Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9506222Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9506864Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9507335Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9507841Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9508338Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9508883Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9509489Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9510078Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9510619Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9511047Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9511547Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9512127Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9512799Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9513565Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9514287Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9515025Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9515639Z   Pool Info:               
2024-12-18T03:35:55.9516092Z     Pool 1                   
2024-12-18T03:35:55.9516693Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9517403Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9518080Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9518711Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9519348Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9519977Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9520604Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9521143Z     Pool 2                   
2024-12-18T03:35:55.9521618Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9522540Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9523152Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9523822Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9524600Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9525362Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9526090Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9526727Z     Pool 3                   
2024-12-18T03:35:55.9527219Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9527783Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9528348Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9528950Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9529571Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9530203Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9530817Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9531335Z     Pool 4                   
2024-12-18T03:35:55.9531791Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9532333Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9533209Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9533809Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9534437Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9535075Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9535689Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9536243Z   ISA Info:                
2024-12-18T03:35:55.9536643Z     ISA 1                    
2024-12-18T03:35:55.9537134Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9537794Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9538433Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9539053Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9539702Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9540296Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9540877Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9541450Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9541953Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9542448Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9542942Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9543489Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9544018Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9544475Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9544981Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9545480Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9546039Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9546567Z *******                  
2024-12-18T03:35:55.9547032Z Agent 9                  
2024-12-18T03:35:55.9547393Z *******                  
2024-12-18T03:35:55.9547826Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9548368Z   Uuid:                    GPU-89e041d1b75d6f09               
2024-12-18T03:35:55.9549247Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9549859Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9550428Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9551002Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9551590Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9552184Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9552769Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9553341Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9553901Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9554443Z   Node:                    8                                  
2024-12-18T03:35:55.9554985Z   Device Type:             GPU                                
2024-12-18T03:35:55.9555514Z   Cache Info:              
2024-12-18T03:35:55.9555929Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9556448Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9556978Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9557535Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9558130Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9559015Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9559561Z   BDFID:                   36352                              
2024-12-18T03:35:55.9560128Z   Internal Node ID:        8                                  
2024-12-18T03:35:55.9560713Z   Compute Unit:            104                                
2024-12-18T03:35:55.9561269Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9561854Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9562454Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9563063Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9563688Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9564234Z   Memory Properties:       
2024-12-18T03:35:55.9564656Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9565212Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9565832Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9566422Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9566972Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9567440Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9567913Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9568393Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9568922Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9569511Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9570101Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9570625Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9571044Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9571537Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9572025Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9572586Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9573228Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9573850Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9574694Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9575228Z   Pool Info:               
2024-12-18T03:35:55.9575621Z     Pool 1                   
2024-12-18T03:35:55.9576104Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9576676Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9577247Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9577841Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9578463Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9579098Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9579709Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9580227Z     Pool 2                   
2024-12-18T03:35:55.9580706Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9581287Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9581838Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9582435Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9583053Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9583670Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9584281Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9585052Z     Pool 3                   
2024-12-18T03:35:55.9585506Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9586057Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9586844Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9587439Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9588064Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9588688Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9589283Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9589806Z     Pool 4                   
2024-12-18T03:35:55.9590257Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9590785Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9591356Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9592023Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9592749Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9593491Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9594216Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9594849Z   ISA Info:                
2024-12-18T03:35:55.9595291Z     ISA 1                    
2024-12-18T03:35:55.9595861Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9596619Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9597349Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9598017Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9598375Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9598672Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9598950Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9599213Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9599434Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9599666Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9600037Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9600289Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9600539Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9600745Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9600975Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9601254Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9601515Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9601753Z *******                  
2024-12-18T03:35:55.9601924Z Agent 10                 
2024-12-18T03:35:55.9602095Z *******                  
2024-12-18T03:35:55.9602286Z   Name:                    gfx90a                             
2024-12-18T03:35:55.9602550Z   Uuid:                    GPU-a911e24894f88cb2               
2024-12-18T03:35:55.9602836Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:35:55.9603121Z   Vendor Name:             AMD                                
2024-12-18T03:35:55.9603400Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:35:55.9603685Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:35:55.9603964Z   Float Round Mode:        NEAR                               
2024-12-18T03:35:55.9604246Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:35:55.9604688Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:35:55.9604947Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:35:55.9605221Z   Queue Type:              MULTI                              
2024-12-18T03:35:55.9605477Z   Node:                    9                                  
2024-12-18T03:35:55.9605725Z   Device Type:             GPU                                
2024-12-18T03:35:55.9605974Z   Cache Info:              
2024-12-18T03:35:55.9606181Z     L1:                      16(0x10) KB                        
2024-12-18T03:35:55.9606417Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:35:55.9606664Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:35:55.9606932Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:35:55.9607218Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:35:55.9607490Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:35:55.9607760Z   BDFID:                   37632                              
2024-12-18T03:35:55.9608028Z   Internal Node ID:        9                                  
2024-12-18T03:35:55.9608295Z   Compute Unit:            104                                
2024-12-18T03:35:55.9608564Z   SIMDs per CU:            4                                  
2024-12-18T03:35:55.9608836Z   Shader Engines:          8                                  
2024-12-18T03:35:55.9609114Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:35:55.9609402Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:35:55.9609697Z   Coherent Host Access:    FALSE                              
2024-12-18T03:35:55.9609952Z   Memory Properties:       
2024-12-18T03:35:55.9610156Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:35:55.9610422Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:35:55.9610706Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:35:55.9610996Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9611258Z   Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9611470Z     x                        1024(0x400)                        
2024-12-18T03:35:55.9611696Z     y                        1024(0x400)                        
2024-12-18T03:35:55.9611918Z     z                        1024(0x400)                        
2024-12-18T03:35:55.9614915Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:35:55.9615224Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:35:55.9615511Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9615759Z   Grid Max Size per Dimension:
2024-12-18T03:35:55.9615967Z     x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9616200Z     y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9616434Z     z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9616705Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:35:55.9617010Z   Packet Processor uCode:: 83                                 
2024-12-18T03:35:55.9617294Z   SDMA engine uCode::      8                                  
2024-12-18T03:35:55.9617574Z   IOMMU Support::          None                               
2024-12-18T03:35:55.9617819Z   Pool Info:               
2024-12-18T03:35:55.9618002Z     Pool 1                   
2024-12-18T03:35:55.9618234Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:35:55.9618515Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9618784Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9619062Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9619353Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9619790Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9620076Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9620327Z     Pool 2                   
2024-12-18T03:35:55.9620556Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:35:55.9620825Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9621112Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9621415Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9621701Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9621991Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9622277Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9622523Z     Pool 3                   
2024-12-18T03:35:55.9622753Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:35:55.9623020Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:35:55.9623277Z       Allocatable:             TRUE                               
2024-12-18T03:35:55.9623561Z       Alloc Granule:           4KB                                
2024-12-18T03:35:55.9623850Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:35:55.9624140Z       Alloc Alignment:         4KB                                
2024-12-18T03:35:55.9624431Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9624688Z     Pool 4                   
2024-12-18T03:35:55.9624898Z       Segment:                 GROUP                              
2024-12-18T03:35:55.9625156Z       Size:                    64(0x40) KB                        
2024-12-18T03:35:55.9625419Z       Allocatable:             FALSE                              
2024-12-18T03:35:55.9625700Z       Alloc Granule:           0KB                                
2024-12-18T03:35:55.9625992Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:35:55.9626285Z       Alloc Alignment:         0KB                                
2024-12-18T03:35:55.9626567Z       Accessible by all:       FALSE                              
2024-12-18T03:35:55.9626869Z   ISA Info:                
2024-12-18T03:35:55.9627043Z     ISA 1                    
2024-12-18T03:35:55.9627413Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:35:55.9627723Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:35:55.9628013Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:35:55.9628313Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9628607Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:35:55.9628883Z       Fast f16:                TRUE                               
2024-12-18T03:35:55.9629165Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:35:55.9629422Z       Workgroup Max Size per Dimension:
2024-12-18T03:35:55.9629653Z         x                        1024(0x400)                        
2024-12-18T03:35:55.9629892Z         y                        1024(0x400)                        
2024-12-18T03:35:55.9630110Z         z                        1024(0x400)                        
2024-12-18T03:35:55.9630365Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:35:55.9630619Z       Grid Max Size per Dimension:
2024-12-18T03:35:55.9630826Z         x                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9631063Z         y                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9631292Z         z                        4294967295(0xffffffff)             
2024-12-18T03:35:55.9631544Z       FBarrier Max Size:       32                                 
2024-12-18T03:35:55.9631934Z *** Done ***             
2024-12-18T03:35:55.9649879Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2024-12-18T03:35:55.9650212Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2024-12-18T03:35:55.9650731Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2024-12-18T03:35:55.9651242Z if [[ $ngpu -eq 0 ]]; then
2024-12-18T03:35:55.9651515Z     echo "Error: Failed to detect any GPUs on the runner"
2024-12-18T03:35:55.9651777Z     echo "$msg"
2024-12-18T03:35:55.9651963Z     exit 1
2024-12-18T03:35:55.9652128Z fi
2024-12-18T03:35:55.9652283Z if [[ $ngpu -eq 1 ]]; then
2024-12-18T03:35:55.9652605Z     echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs"
2024-12-18T03:35:55.9652930Z     echo "$msg"
2024-12-18T03:35:55.9653109Z     exit 1
2024-12-18T03:35:55.9653272Z fi
2024-12-18T03:35:55.9678293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:55.9678577Z env:
2024-12-18T03:35:55.9678734Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:55.9678964Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:55.9679212Z ##[endgroup]
2024-12-18T03:35:56.1423951Z Prepare all required actions
2024-12-18T03:35:56.1483905Z ##[group]Run ./.github/actions/diskspace-cleanup
2024-12-18T03:35:56.1484532Z with:
2024-12-18T03:35:56.1484940Z   diskspace-cutoff: 70
2024-12-18T03:35:56.1485377Z env:
2024-12-18T03:35:56.1485829Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.1486505Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.1487242Z ##[endgroup]
2024-12-18T03:35:56.1559798Z ##[group]Run set -ex
2024-12-18T03:35:56.1560407Z set -ex
2024-12-18T03:35:56.1560864Z diskspace_cutoff=70
2024-12-18T03:35:56.1561562Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2024-12-18T03:35:56.1562681Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2024-12-18T03:35:56.1564310Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2024-12-18T03:35:56.1565768Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2024-12-18T03:35:56.1566596Z     docker system prune -af
2024-12-18T03:35:56.1568165Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2024-12-18T03:35:56.1569293Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2024-12-18T03:35:56.1570403Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2024-12-18T03:35:56.1571356Z         echo "$msg"
2024-12-18T03:35:56.1571847Z         exit 1
2024-12-18T03:35:56.1572301Z     else
2024-12-18T03:35:56.1572808Z         difference=$((diskspace - diskspace_new))
2024-12-18T03:35:56.1573552Z         echo "Diskspace saved: $difference percent"
2024-12-18T03:35:56.1574171Z     fi
2024-12-18T03:35:56.1574560Z fi
2024-12-18T03:35:56.1624906Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:56.1625485Z env:
2024-12-18T03:35:56.1625801Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.1626322Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.1626972Z ##[endgroup]
2024-12-18T03:35:56.1712866Z + diskspace_cutoff=70
2024-12-18T03:35:56.1722537Z ++ docker info -f '{{.DockerRootDir}}'
2024-12-18T03:35:56.2714346Z + docker_root_dir=/media/4TB/docker-rootless
2024-12-18T03:35:56.2731418Z ++ df -H --output=pcent /media/4TB/docker-rootless
2024-12-18T03:35:56.2732483Z ++ sed -n 2p
2024-12-18T03:35:56.2735442Z ++ sed s/%//
2024-12-18T03:35:56.2735839Z ++ sed 's/ //'
2024-12-18T03:35:56.2780257Z + diskspace=31
2024-12-18T03:35:56.2781483Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2024-12-18T03:35:56.2782697Z + [[ 31 -ge 70 ]]
2024-12-18T03:35:56.2843303Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T03:35:56.2844233Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T03:35:56.2845058Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T03:35:56.2898962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:56.2899666Z env:
2024-12-18T03:35:56.2900059Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.2900646Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.2901287Z ##[endgroup]
2024-12-18T03:35:56.3131388Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2024-12-18T03:35:56.3132675Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2024-12-18T03:35:56.3134159Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}"
2024-12-18T03:35:56.3188901Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:56.3189722Z env:
2024-12-18T03:35:56.3190191Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.3190809Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.3191461Z ##[endgroup]
2024-12-18T03:35:56.3403954Z ##[group]Run aws-actions/configure-aws-credentials@v4
2024-12-18T03:35:56.3404620Z with:
2024-12-18T03:35:56.3405268Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2024-12-18T03:35:56.3406052Z   aws-region: us-east-1
2024-12-18T03:35:56.3406514Z   role-duration-seconds: 18000
2024-12-18T03:35:56.3407037Z   audience: sts.amazonaws.com
2024-12-18T03:35:56.3407505Z env:
2024-12-18T03:35:56.3407897Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.3408444Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.3409455Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:56.3410321Z ##[endgroup]
2024-12-18T03:35:56.6643460Z Assuming role with OIDC
2024-12-18T03:35:56.8370974Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2024-12-18T03:35:56.9084271Z ##[group]Run aws-actions/amazon-ecr-login@v2
2024-12-18T03:35:56.9084972Z with:
2024-12-18T03:35:56.9085414Z   mask-password: true
2024-12-18T03:35:56.9085946Z   registry-type: private
2024-12-18T03:35:56.9086469Z   skip-logout: false
2024-12-18T03:35:56.9086921Z env:
2024-12-18T03:35:56.9087337Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:56.9087959Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:56.9089059Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:56.9090089Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:35:56.9090632Z   AWS_REGION: us-east-1
2024-12-18T03:35:56.9091332Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:35:56.9092058Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:35:56.9103593Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:35:56.9104097Z ##[endgroup]
2024-12-18T03:35:57.2465215Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.6462395Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6
2024-12-18T03:35:57.6463239Z with:
2024-12-18T03:35:57.6464262Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6465434Z   docker-build-dir: .ci/docker
2024-12-18T03:35:57.6465910Z   working-directory: .
2024-12-18T03:35:57.6466473Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.6467286Z   force-push: false
2024-12-18T03:35:57.6467669Z env:
2024-12-18T03:35:57.6468512Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:57.6469046Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:57.6469954Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:57.6470776Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:35:57.6471241Z   AWS_REGION: us-east-1
2024-12-18T03:35:57.6471941Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:35:57.6472557Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:35:57.6482186Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:35:57.6482599Z ##[endgroup]
2024-12-18T03:35:57.6517488Z ##[group]Run set -ex
2024-12-18T03:35:57.6518000Z set -ex
2024-12-18T03:35:57.6518368Z 
2024-12-18T03:35:57.6519006Z # If the docker build directory or the build script doesn't exist, the action will
2024-12-18T03:35:57.6520024Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-12-18T03:35:57.6520900Z # job could then download the pre-built image as usual
2024-12-18T03:35:57.6521721Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-12-18T03:35:57.6522457Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6523164Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6523807Z 
2024-12-18T03:35:57.6524383Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-12-18T03:35:57.6525083Z   exit 0
2024-12-18T03:35:57.6525470Z else
2024-12-18T03:35:57.6525914Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6526436Z fi
2024-12-18T03:35:57.6526792Z 
2024-12-18T03:35:57.6527337Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-12-18T03:35:57.6528257Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-12-18T03:35:57.6529108Z   # use it as it is, but first let's extract the tag
2024-12-18T03:35:57.6529893Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-12-18T03:35:57.6530695Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6531461Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6532107Z else
2024-12-18T03:35:57.6532622Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-12-18T03:35:57.6533339Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6534311Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6535165Z fi
2024-12-18T03:35:57.6586267Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:57.6586989Z env:
2024-12-18T03:35:57.6587354Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:57.6587907Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:57.6588809Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:57.6589633Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:35:57.6590100Z   AWS_REGION: us-east-1
2024-12-18T03:35:57.6590656Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:35:57.6591701Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:35:57.6601394Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:35:57.6601847Z   REPO_NAME: pytorch
2024-12-18T03:35:57.6602915Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6604047Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T03:35:57.6604671Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.6605293Z ##[endgroup]
2024-12-18T03:35:57.6693792Z + [[ ! -d .ci/docker ]]
2024-12-18T03:35:57.6694436Z + [[ ! -f .ci/docker/build.sh ]]
2024-12-18T03:35:57.6695704Z + echo skip=false
2024-12-18T03:35:57.6697426Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-12-18T03:35:57.6712919Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6714347Z ++ awk -F '[:,]' '{print $2}'
2024-12-18T03:35:57.6773643Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6774788Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6781471Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6855471Z ##[group]Run set +e
2024-12-18T03:35:57.6856108Z set +e
2024-12-18T03:35:57.6856561Z set -x
2024-12-18T03:35:57.6857023Z 
2024-12-18T03:35:57.6857441Z login() {
2024-12-18T03:35:57.6858361Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T03:35:57.6859345Z }
2024-12-18T03:35:57.6859753Z 
2024-12-18T03:35:57.6860158Z retry () {
2024-12-18T03:35:57.6860700Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T03:35:57.6861301Z }
2024-12-18T03:35:57.6861702Z 
2024-12-18T03:35:57.6862149Z retry login "${DOCKER_REGISTRY}"
2024-12-18T03:35:57.6862719Z 
2024-12-18T03:35:57.6863142Z START_TIME=$(date +%s)
2024-12-18T03:35:57.6863707Z # Wait up to 90 minutes
2024-12-18T03:35:57.6864407Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do
2024-12-18T03:35:57.6865327Z   # Check if image already exists, if it does then skip building it
2024-12-18T03:35:57.6866230Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-12-18T03:35:57.6867333Z     exit 0
2024-12-18T03:35:57.6867814Z   fi
2024-12-18T03:35:57.6868235Z 
2024-12-18T03:35:57.6868951Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2024-12-18T03:35:57.6870169Z   # use this to differentiate between the Docker build and regular build jobs. For the
2024-12-18T03:35:57.6871352Z   # latter, it will wait for the Docker images to become available before continuing
2024-12-18T03:35:57.6872314Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2024-12-18T03:35:57.6873074Z     # It's a Docker build job, let's build the image
2024-12-18T03:35:57.6873775Z     break
2024-12-18T03:35:57.6874301Z   else
2024-12-18T03:35:57.6875060Z     # It's a regular build job, wait for the image to become available
2024-12-18T03:35:57.6875969Z     sleep 300
2024-12-18T03:35:57.6876529Z   fi
2024-12-18T03:35:57.6877042Z done
2024-12-18T03:35:57.6877528Z 
2024-12-18T03:35:57.6878312Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-12-18T03:35:57.6879550Z # be empty.  The default action would be to continue rebuild the image
2024-12-18T03:35:57.6881106Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-12-18T03:35:57.6881998Z   # if we're on the base branch then use the parent commit
2024-12-18T03:35:57.6882749Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-12-18T03:35:57.6883340Z else
2024-12-18T03:35:57.6883950Z   # otherwise we're on a PR, so use the most recent base commit
2024-12-18T03:35:57.6884822Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-12-18T03:35:57.6885497Z fi
2024-12-18T03:35:57.6885964Z 
2024-12-18T03:35:57.6886501Z if [[ -z "${MERGE_BASE}" ]]; then
2024-12-18T03:35:57.6887288Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6888489Z 
2024-12-18T03:35:57.6889432Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-12-18T03:35:57.6890422Z   exit 0
2024-12-18T03:35:57.6890845Z fi
2024-12-18T03:35:57.6891246Z 
2024-12-18T03:35:57.6891836Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-12-18T03:35:57.6893061Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-12-18T03:35:57.6894098Z   exit 1
2024-12-18T03:35:57.6894516Z fi
2024-12-18T03:35:57.6894919Z 
2024-12-18T03:35:57.6895591Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-12-18T03:35:57.6896773Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-12-18T03:35:57.6897857Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-12-18T03:35:57.6899077Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-12-18T03:35:57.6900462Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-12-18T03:35:57.6901299Z fi
2024-12-18T03:35:57.6901699Z 
2024-12-18T03:35:57.6902196Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T03:35:57.6963422Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:57.6964151Z env:
2024-12-18T03:35:57.6964585Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:57.6965204Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:57.6966429Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:57.6967594Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:35:57.6968241Z   AWS_REGION: us-east-1
2024-12-18T03:35:57.6969080Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:35:57.6969855Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:35:57.6981316Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:35:57.6981844Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T03:35:57.6982503Z   BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:35:57.6983980Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6985349Z   DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:57.6986167Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.6987254Z   DOCKER_PUSH: 
2024-12-18T03:35:57.6987726Z ##[endgroup]
2024-12-18T03:35:57.7076066Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.7076797Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:57.7084687Z + aws ecr get-login-password --region us-east-1
2024-12-18T03:35:57.7089197Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:35:58.8153580Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2024-12-18T03:35:58.8154746Z Configure a credential helper to remove this warning. See
2024-12-18T03:35:58.8156346Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2024-12-18T03:35:58.8157072Z 
2024-12-18T03:35:58.8157577Z Login Succeeded
2024-12-18T03:35:58.8219067Z ++ date +%s
2024-12-18T03:35:58.8247481Z + START_TIME=1734492958
2024-12-18T03:35:58.8257282Z ++ date +%s
2024-12-18T03:35:58.8286111Z + [[ 1734487558 -lt 1734492958 ]]
2024-12-18T03:35:58.8287684Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:35:59.5110110Z {
2024-12-18T03:35:59.5110588Z 	"schemaVersion": 2,
2024-12-18T03:35:59.5111428Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-12-18T03:35:59.5113073Z 	"config": {
2024-12-18T03:35:59.5113815Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-12-18T03:35:59.5114531Z 		"size": 24144,
2024-12-18T03:35:59.5115282Z 		"digest": "sha256:07b91116eb1063321da7a63f31f1ef61d34f7dc8b7d56ec47c24f12af9f38edb"
2024-12-18T03:35:59.5116136Z 	},
2024-12-18T03:35:59.5116498Z 	"layers": [
2024-12-18T03:35:59.5116897Z 		{
2024-12-18T03:35:59.5117501Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5118262Z 			"size": 28583948,
2024-12-18T03:35:59.5119008Z 			"digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1"
2024-12-18T03:35:59.5119816Z 		},
2024-12-18T03:35:59.5120166Z 		{
2024-12-18T03:35:59.5120749Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5121620Z 			"size": 1825,
2024-12-18T03:35:59.5122381Z 			"digest": "sha256:7e5cdcc8d39cbfc3cbf5fe048da71a36c2c80d4a1e184cccc8c9f02e7e076f64"
2024-12-18T03:35:59.5123242Z 		},
2024-12-18T03:35:59.5123599Z 		{
2024-12-18T03:35:59.5124195Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5124931Z 			"size": 312642362,
2024-12-18T03:35:59.5125686Z 			"digest": "sha256:231ab5ed5e979cc01fda602a87a43fd305466b7cc01b98a76690ffa0b716f312"
2024-12-18T03:35:59.5126506Z 		},
2024-12-18T03:35:59.5126851Z 		{
2024-12-18T03:35:59.5127431Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5128160Z 			"size": 863,
2024-12-18T03:35:59.5129034Z 			"digest": "sha256:fcbefe9ad79ece7f701532eae83e9de139fe1c0949a7813e55dd3a53003af9cd"
2024-12-18T03:35:59.5129991Z 		},
2024-12-18T03:35:59.5130333Z 		{
2024-12-18T03:35:59.5130896Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5131608Z 			"size": 106,
2024-12-18T03:35:59.5132339Z 			"digest": "sha256:af67ad6d287ebd8cbde55a749a5bbc06170aea40e4902c93cd34922e10c8c408"
2024-12-18T03:35:59.5133162Z 		},
2024-12-18T03:35:59.5133521Z 		{
2024-12-18T03:35:59.5134083Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5134795Z 			"size": 704,
2024-12-18T03:35:59.5135518Z 			"digest": "sha256:a87dd7b9e020ba8d0d077d74c105bbb7fe8cdd8f1e6ae0c98b84c3e2c06ace60"
2024-12-18T03:35:59.5136339Z 		},
2024-12-18T03:35:59.5136681Z 		{
2024-12-18T03:35:59.5137245Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5137961Z 			"size": 1257,
2024-12-18T03:35:59.5138695Z 			"digest": "sha256:aa67f5368a1ddc10ff5b57e69f4ebcf2f83e0f32dfe4648e2a21aee041f107f8"
2024-12-18T03:35:59.5139513Z 		},
2024-12-18T03:35:59.5139856Z 		{
2024-12-18T03:35:59.5140411Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5141122Z 			"size": 3708,
2024-12-18T03:35:59.5141837Z 			"digest": "sha256:e71aede6d46007bf2a44f7cd14800cf5486c895c0d90a0b18ea9c30e6fff4571"
2024-12-18T03:35:59.5142641Z 		},
2024-12-18T03:35:59.5142980Z 		{
2024-12-18T03:35:59.5143548Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5144251Z 			"size": 1860,
2024-12-18T03:35:59.5144967Z 			"digest": "sha256:ee65ff0efe5e4f68873b0389e60a87f13f1a1bb0df1178fd70e5e83e59753fab"
2024-12-18T03:35:59.5145773Z 		},
2024-12-18T03:35:59.5146115Z 		{
2024-12-18T03:35:59.5147216Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5147992Z 			"size": 701,
2024-12-18T03:35:59.5148711Z 			"digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676"
2024-12-18T03:35:59.5149513Z 		},
2024-12-18T03:35:59.5149854Z 		{
2024-12-18T03:35:59.5150421Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5151188Z 			"size": 2717169179,
2024-12-18T03:35:59.5152046Z 			"digest": "sha256:8362d69f07765b32e976c14dd6625372f9d81f263db1dc4ef06f46d99571f992"
2024-12-18T03:35:59.5152982Z 		},
2024-12-18T03:35:59.5153351Z 		{
2024-12-18T03:35:59.5153908Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5154939Z 			"size": 380,
2024-12-18T03:35:59.5155627Z 			"digest": "sha256:0d29ca60b09574f6c0756a7025b88000a59b1009992fa1478637f5bdc5784ad2"
2024-12-18T03:35:59.5156408Z 		},
2024-12-18T03:35:59.5156745Z 		{
2024-12-18T03:35:59.5157306Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5158031Z 			"size": 12135,
2024-12-18T03:35:59.5158726Z 			"digest": "sha256:6697412e0463180656f0051e70d45098fb97bbdb91515623a3ca0d0785ed26c3"
2024-12-18T03:35:59.5159501Z 		},
2024-12-18T03:35:59.5159852Z 		{
2024-12-18T03:35:59.5160409Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5161119Z 			"size": 504,
2024-12-18T03:35:59.5161825Z 			"digest": "sha256:3fe85e92871bbecf1a51782c8349d745adf952dabe41e4023f16496e13c89d6e"
2024-12-18T03:35:59.5162628Z 		},
2024-12-18T03:35:59.5162974Z 		{
2024-12-18T03:35:59.5163532Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5164255Z 			"size": 121477504,
2024-12-18T03:35:59.5164970Z 			"digest": "sha256:db8079b9a247c260d493898268f96390bcef45198f5799ea28a82894417d3409"
2024-12-18T03:35:59.5165751Z 		},
2024-12-18T03:35:59.5166096Z 		{
2024-12-18T03:35:59.5166653Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5167357Z 			"size": 109,
2024-12-18T03:35:59.5168105Z 			"digest": "sha256:1e1cc236bfa694641ad0d2d7fb1a76caa46f0b5364084a30d576a593cdf09838"
2024-12-18T03:35:59.5169027Z 		},
2024-12-18T03:35:59.5169442Z 		{
2024-12-18T03:35:59.5170086Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5170794Z 			"size": 489,
2024-12-18T03:35:59.5171503Z 			"digest": "sha256:5594c816d49f271fafd93fedc956a82586a1ddb33f6a81c6c47324b2bc675e6f"
2024-12-18T03:35:59.5172310Z 		},
2024-12-18T03:35:59.5172651Z 		{
2024-12-18T03:35:59.5173205Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5173913Z 			"size": 296,
2024-12-18T03:35:59.5174608Z 			"digest": "sha256:d0419af25312d587b2c3532b144859e2155c14d2ca73864f20041dd565b0c9c7"
2024-12-18T03:35:59.5175389Z 		},
2024-12-18T03:35:59.5175731Z 		{
2024-12-18T03:35:59.5176288Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5176999Z 			"size": 103,
2024-12-18T03:35:59.5177702Z 			"digest": "sha256:4a7e3c6b1985b14cd275d8480a86dabccb21244b38cca3771881b2f87a880a7c"
2024-12-18T03:35:59.5178509Z 		},
2024-12-18T03:35:59.5178851Z 		{
2024-12-18T03:35:59.5179406Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5180111Z 			"size": 1474,
2024-12-18T03:35:59.5180819Z 			"digest": "sha256:e7e49899a1507f1355d8dd6a16d5beb03f1ced22428dc3b796a8c453fba8b41d"
2024-12-18T03:35:59.5181627Z 		},
2024-12-18T03:35:59.5181969Z 		{
2024-12-18T03:35:59.5182523Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5183236Z 			"size": 430331313,
2024-12-18T03:35:59.5183961Z 			"digest": "sha256:907b5d385c1450d33d646dcce152dec74855431d5aa6d427f39d74f87ab7a183"
2024-12-18T03:35:59.5184785Z 		},
2024-12-18T03:35:59.5185129Z 		{
2024-12-18T03:35:59.5185691Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5186404Z 			"size": 163,
2024-12-18T03:35:59.5187541Z 			"digest": "sha256:e09946f27e30eab5749280587f84a64accfe85ccfaabf422c1fae010dd492cbd"
2024-12-18T03:35:59.5188428Z 		},
2024-12-18T03:35:59.5188838Z 		{
2024-12-18T03:35:59.5189503Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5190788Z 			"size": 1640,
2024-12-18T03:35:59.5191506Z 			"digest": "sha256:7294a9f24454fce9dc046e77ab28a73149429dde57793009bb3223eb424713de"
2024-12-18T03:35:59.5192307Z 		},
2024-12-18T03:35:59.5192675Z 		{
2024-12-18T03:35:59.5193241Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5193966Z 			"size": 7880578701,
2024-12-18T03:35:59.5194716Z 			"digest": "sha256:5cbdd1d7b36d7b6e2d6428e2857ec4f8614e3becbdb0ec6e3b64f3cf63584e59"
2024-12-18T03:35:59.5195874Z 		},
2024-12-18T03:35:59.5196218Z 		{
2024-12-18T03:35:59.5196777Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5197493Z 			"size": 105,
2024-12-18T03:35:59.5198178Z 			"digest": "sha256:f528471f2fcaa24ae54a666344201877b06d22561708fd21987ac10c0873d6a0"
2024-12-18T03:35:59.5198983Z 		},
2024-12-18T03:35:59.5199330Z 		{
2024-12-18T03:35:59.5199888Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5200596Z 			"size": 1116,
2024-12-18T03:35:59.5201282Z 			"digest": "sha256:e1276511038b9b5b700d3a69400a0692e260be884871347905717fbec3009ced"
2024-12-18T03:35:59.5202058Z 		},
2024-12-18T03:35:59.5202401Z 		{
2024-12-18T03:35:59.5202954Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5203670Z 			"size": 318568913,
2024-12-18T03:35:59.5204391Z 			"digest": "sha256:f3d817e0be62e96a38e3a6596705a35dfa864a2658a4d4e92890f192eaa1103e"
2024-12-18T03:35:59.5205201Z 		},
2024-12-18T03:35:59.5205541Z 		{
2024-12-18T03:35:59.5206100Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5206809Z 			"size": 111,
2024-12-18T03:35:59.5207522Z 			"digest": "sha256:babfeddb429861eef35685a83ff84f1b120f8eda1b6a1c552453065dc0bd6cb3"
2024-12-18T03:35:59.5208337Z 		},
2024-12-18T03:35:59.5208735Z 		{
2024-12-18T03:35:59.5209399Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5210202Z 			"size": 1555,
2024-12-18T03:35:59.5210915Z 			"digest": "sha256:e00cbec1ecdd51cc9b1b2471a6bf61f3e93cc03540114c13375e35cb48169f90"
2024-12-18T03:35:59.5211731Z 		},
2024-12-18T03:35:59.5212066Z 		{
2024-12-18T03:35:59.5212617Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5213323Z 			"size": 107,
2024-12-18T03:35:59.5214013Z 			"digest": "sha256:60175e320c6277d6be778429e32b3dd271c508af81bb52bdfec8c5c6edb55a66"
2024-12-18T03:35:59.5214810Z 		},
2024-12-18T03:35:59.5215165Z 		{
2024-12-18T03:35:59.5215721Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5216430Z 			"size": 166,
2024-12-18T03:35:59.5217140Z 			"digest": "sha256:e4fd8cd3183ce9d5e635650cd867e61f68b6edc31b37e605f9c038a2064e42c0"
2024-12-18T03:35:59.5217946Z 		},
2024-12-18T03:35:59.5218285Z 		{
2024-12-18T03:35:59.5218848Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5219558Z 			"size": 2842293,
2024-12-18T03:35:59.5220282Z 			"digest": "sha256:f79cba221c4638d554fcf21bde3a813a5ddd7d4900f5928e43c92c9307e9d58d"
2024-12-18T03:35:59.5221084Z 		},
2024-12-18T03:35:59.5221424Z 		{
2024-12-18T03:35:59.5221980Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5222692Z 			"size": 107,
2024-12-18T03:35:59.5223397Z 			"digest": "sha256:e425c457ab3e0a6637feed1c3ef95776f4c43fc59336a9da4e0a7f19858529c2"
2024-12-18T03:35:59.5224204Z 		},
2024-12-18T03:35:59.5224544Z 		{
2024-12-18T03:35:59.5225115Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5225823Z 			"size": 566,
2024-12-18T03:35:59.5226530Z 			"digest": "sha256:6d2cbef9a0f53dff8893f40509e211ffe284a5d8fb4e03923b9c9646b06ad2e7"
2024-12-18T03:35:59.5227658Z 		},
2024-12-18T03:35:59.5228001Z 		{
2024-12-18T03:35:59.5229080Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5229979Z 			"size": 43195564,
2024-12-18T03:35:59.5230783Z 			"digest": "sha256:2c4472ad27f9258eee43c37d8fdd0ec0db693fc1d33d03902c90d85850a2fa74"
2024-12-18T03:35:59.5231593Z 		},
2024-12-18T03:35:59.5231927Z 		{
2024-12-18T03:35:59.5232477Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5233185Z 			"size": 106,
2024-12-18T03:35:59.5233884Z 			"digest": "sha256:48ffbd6d25729ef5a8cbfc2950686efcd5cc5854803572b079a40e7d59b81481"
2024-12-18T03:35:59.5234692Z 		},
2024-12-18T03:35:59.5235032Z 		{
2024-12-18T03:35:59.5235590Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5236619Z 			"size": 295,
2024-12-18T03:35:59.5237322Z 			"digest": "sha256:cf56c8fabbc0842e7f589f295894c2b6eb980e7f5ec8f2232b8c105f503a46e5"
2024-12-18T03:35:59.5238135Z 		},
2024-12-18T03:35:59.5238475Z 		{
2024-12-18T03:35:59.5239027Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5239748Z 			"size": 88293,
2024-12-18T03:35:59.5240457Z 			"digest": "sha256:b1c0b18891861b71d1c07dfb5e1063225b3ee635b5cf8d088bc62755ae2a3140"
2024-12-18T03:35:59.5241254Z 		},
2024-12-18T03:35:59.5241593Z 		{
2024-12-18T03:35:59.5242147Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5242854Z 			"size": 106,
2024-12-18T03:35:59.5243566Z 			"digest": "sha256:3021fd4a208aeeab27bda896abedaba113f2a3845b3b037c57f855a429bcd61a"
2024-12-18T03:35:59.5244385Z 		},
2024-12-18T03:35:59.5244729Z 		{
2024-12-18T03:35:59.5245284Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5246008Z 			"size": 1402,
2024-12-18T03:35:59.5246737Z 			"digest": "sha256:a6f5eec4a2df539ca39d87dfd6ba37ef1c60304eda99ed48b657823b8b5a83aa"
2024-12-18T03:35:59.5247567Z 		},
2024-12-18T03:35:59.5247908Z 		{
2024-12-18T03:35:59.5248467Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5249308Z 			"size": 701,
2024-12-18T03:35:59.5250140Z 			"digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676"
2024-12-18T03:35:59.5250943Z 		},
2024-12-18T03:35:59.5251286Z 		{
2024-12-18T03:35:59.5251844Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5252563Z 			"size": 137,
2024-12-18T03:35:59.5253278Z 			"digest": "sha256:89e8bdba21ffada6ab13d8b625b5349b0cc0398741ffd1c5155fdea44633255a"
2024-12-18T03:35:59.5254097Z 		},
2024-12-18T03:35:59.5254436Z 		{
2024-12-18T03:35:59.5254991Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5255696Z 			"size": 120,
2024-12-18T03:35:59.5256410Z 			"digest": "sha256:7239a56eb3344ab4d10ba3a41fc1e7af5559c747ee876bb8660f0237ed85a2f0"
2024-12-18T03:35:59.5257209Z 		},
2024-12-18T03:35:59.5257554Z 		{
2024-12-18T03:35:59.5258105Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5258817Z 			"size": 5014581981,
2024-12-18T03:35:59.5259552Z 			"digest": "sha256:22093b6a474afa19ed0f46a0491aa69688d1f0b3954fb7614a8f8e2538f90a43"
2024-12-18T03:35:59.5260353Z 		},
2024-12-18T03:35:59.5260687Z 		{
2024-12-18T03:35:59.5261243Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5261946Z 			"size": 174,
2024-12-18T03:35:59.5262644Z 			"digest": "sha256:48aa755f3a17db6703cc7bff22b93fe216ee7f1112fb2e8330c2e1f7737822c2"
2024-12-18T03:35:59.5263447Z 		},
2024-12-18T03:35:59.5263784Z 		{
2024-12-18T03:35:59.5264338Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5265039Z 			"size": 209,
2024-12-18T03:35:59.5265719Z 			"digest": "sha256:789ba2fe5a5187199259980262efd9e420b90ff489372e2cd60c98470a3db40d"
2024-12-18T03:35:59.5266514Z 		},
2024-12-18T03:35:59.5267000Z 		{
2024-12-18T03:35:59.5267554Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5268319Z 			"size": 701,
2024-12-18T03:35:59.5269136Z 			"digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676"
2024-12-18T03:35:59.5270507Z 		},
2024-12-18T03:35:59.5270859Z 		{
2024-12-18T03:35:59.5271413Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5272119Z 			"size": 633,
2024-12-18T03:35:59.5272794Z 			"digest": "sha256:e2e30e00626e69012828cfe72a0d31832f6ff31805e36d44356537d5f7fcb9ba"
2024-12-18T03:35:59.5273579Z 		},
2024-12-18T03:35:59.5273920Z 		{
2024-12-18T03:35:59.5274470Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5275185Z 			"size": 296742339,
2024-12-18T03:35:59.5275903Z 			"digest": "sha256:c631a2525143e662e72b9b97c2e87b09a43a1038995e76b403dad6ba88966e41"
2024-12-18T03:35:59.5277008Z 		},
2024-12-18T03:35:59.5277481Z 		{
2024-12-18T03:35:59.5278053Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5278765Z 			"size": 1932,
2024-12-18T03:35:59.5279485Z 			"digest": "sha256:08d3deeeb0fbaac4992048b95790661de095ddd8925926cfa2f2ec217bf4e0f7"
2024-12-18T03:35:59.5280296Z 		},
2024-12-18T03:35:59.5280644Z 		{
2024-12-18T03:35:59.5281200Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5281911Z 			"size": 15983403,
2024-12-18T03:35:59.5282645Z 			"digest": "sha256:f098b0de0dbe9076c39d2aa97b15a08de76d7c33be71cc1069255184e8716618"
2024-12-18T03:35:59.5283452Z 		},
2024-12-18T03:35:59.5283793Z 		{
2024-12-18T03:35:59.5284343Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T03:35:59.5285048Z 			"size": 54145664,
2024-12-18T03:35:59.5285753Z 			"digest": "sha256:4e73d9ed9f9510302a97c5ce44701037bd8453e91d3dbe54738434120cf00300"
2024-12-18T03:35:59.5286552Z 		}
2024-12-18T03:35:59.5286888Z 	]
2024-12-18T03:35:59.5287235Z }
2024-12-18T03:35:59.5287594Z + exit 0
2024-12-18T03:35:59.5337979Z ##[group]Run set -eux
2024-12-18T03:35:59.5338433Z set -eux
2024-12-18T03:35:59.5339789Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2024-12-18T03:35:59.5387776Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:35:59.5388415Z env:
2024-12-18T03:35:59.5388837Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:35:59.5389456Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:35:59.5390527Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:35:59.5391350Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:35:59.5391805Z   AWS_REGION: us-east-1
2024-12-18T03:35:59.5392353Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:35:59.5393012Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:35:59.5402680Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:35:59.5403106Z ##[endgroup]
2024-12-18T03:35:59.5494403Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2024-12-18T03:35:59.5501037Z + jq --raw-output .SecretString
2024-12-18T03:35:59.5501820Z + jq -r .docker_hub_readonly_token
2024-12-18T03:35:59.5505368Z + docker login --username pytorchbot --password-stdin
2024-12-18T03:36:00.3245837Z 
2024-12-18T03:36:00.3248646Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2024-12-18T03:36:00.4345211Z Error: Cannot perform an interactive login from a non TTY device
2024-12-18T03:36:00.4422822Z ##[error]Process completed with exit code 1.
2024-12-18T03:36:00.4592236Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6
2024-12-18T03:36:00.4593159Z with:
2024-12-18T03:36:00.4594351Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:36:00.4595862Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:36:00.4596609Z env:
2024-12-18T03:36:00.4597041Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:00.4597681Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:00.4598764Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:00.4599771Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:00.4600321Z   AWS_REGION: us-east-1
2024-12-18T03:36:00.4601001Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:00.4601746Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:00.4614229Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:00.4615496Z ##[endgroup]
2024-12-18T03:36:00.4648405Z ##[group]Run set -x
2024-12-18T03:36:00.4648936Z set -x
2024-12-18T03:36:00.4649375Z set +e
2024-12-18T03:36:00.4649799Z 
2024-12-18T03:36:00.4650217Z login() {
2024-12-18T03:36:00.4651132Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T03:36:00.4652123Z }
2024-12-18T03:36:00.4652546Z 
2024-12-18T03:36:00.4652954Z retry () {
2024-12-18T03:36:00.4653482Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T03:36:00.4654079Z }
2024-12-18T03:36:00.4654478Z 
2024-12-18T03:36:00.4654941Z retry login "${DOCKER_REGISTRY}"
2024-12-18T03:36:00.4655523Z 
2024-12-18T03:36:00.4655923Z set -e
2024-12-18T03:36:00.4656571Z # ignore output since only exit code is used for conditional
2024-12-18T03:36:00.4657499Z # only pull docker image if it's not available locally
2024-12-18T03:36:00.4658519Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-12-18T03:36:00.4659456Z   retry docker pull "${DOCKER_IMAGE}"
2024-12-18T03:36:00.4660057Z fi
2024-12-18T03:36:00.4711757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:00.4712400Z env:
2024-12-18T03:36:00.4712773Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:00.4713311Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:00.4714232Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:00.4715057Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:00.4715520Z   AWS_REGION: us-east-1
2024-12-18T03:36:00.4716052Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:00.4716691Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:00.4726379Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:00.4727458Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:36:00.4728697Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:36:00.4729321Z ##[endgroup]
2024-12-18T03:36:00.4817443Z + set +e
2024-12-18T03:36:00.4818797Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:36:00.4819695Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:36:00.4828616Z + aws ecr get-login-password --region us-east-1
2024-12-18T03:36:00.4830261Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T03:36:01.5728891Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2024-12-18T03:36:01.5730032Z Configure a credential helper to remove this warning. See
2024-12-18T03:36:01.5731050Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2024-12-18T03:36:01.5731723Z 
2024-12-18T03:36:01.5733876Z Login Succeeded
2024-12-18T03:36:01.5791504Z + set -e
2024-12-18T03:36:01.5793142Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:36:01.6324452Z Prepare all required actions
2024-12-18T03:36:01.6325260Z Getting action download info
2024-12-18T03:36:01.8025623Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-12-18T03:36:02.4137129Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16)
2024-12-18T03:36:02.9474613Z ##[group]Run ./.github/actions/download-build-artifacts
2024-12-18T03:36:02.9475220Z with:
2024-12-18T03:36:02.9475624Z   name: linux-focal-rocm6.2-py3.10
2024-12-18T03:36:02.9476163Z   s3-bucket: gha-artifacts
2024-12-18T03:36:02.9476670Z env:
2024-12-18T03:36:02.9477088Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:02.9477690Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:02.9479225Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:02.9480207Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:02.9480757Z   AWS_REGION: us-east-1
2024-12-18T03:36:02.9481207Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:02.9481558Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:02.9486933Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:02.9487222Z ##[endgroup]
2024-12-18T03:36:02.9512782Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T03:36:02.9513081Z with:
2024-12-18T03:36:02.9513298Z   name: linux-focal-rocm6.2-py3.10
2024-12-18T03:36:02.9513574Z   s3-bucket: gha-artifacts
2024-12-18T03:36:02.9513818Z   region: us-east-1
2024-12-18T03:36:02.9514030Z env:
2024-12-18T03:36:02.9514224Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:02.9514502Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:02.9514996Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:02.9515473Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:02.9515733Z   AWS_REGION: us-east-1
2024-12-18T03:36:02.9516033Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:02.9516375Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:02.9521731Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:02.9521970Z ##[endgroup]
2024-12-18T03:36:03.4020721Z (node:3129362) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T03:36:03.4021684Z 
2024-12-18T03:36:03.4022096Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T03:36:03.4023135Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T03:36:03.4024178Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T03:36:03.5526359Z Found 1 objects with prefix pytorch/pytorch/12383255654/linux-focal-rocm6.2-py3.10/
2024-12-18T03:36:03.5527821Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T03:36:30.0688064Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T03:36:30.0704541Z Artifact download has finished successfully
2024-12-18T03:36:30.1310485Z ##[group]Run unzip -o artifacts.zip
2024-12-18T03:36:30.1311141Z unzip -o artifacts.zip
2024-12-18T03:36:30.1368360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:30.1369067Z env:
2024-12-18T03:36:30.1369484Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:30.1370099Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:30.1371129Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:30.1372107Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:30.1372733Z   AWS_REGION: us-east-1
2024-12-18T03:36:30.1373437Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:30.1374247Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:30.1385234Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:30.1385737Z ##[endgroup]
2024-12-18T03:36:30.1517684Z Archive:  artifacts.zip
2024-12-18T03:36:30.1519841Z    creating: dist/
2024-12-18T03:36:32.8339052Z   inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl  
2024-12-18T03:36:32.8463418Z   inflating: dist/.ninja_log         
2024-12-18T03:36:32.8469072Z    creating: build/custom_test_artifacts/
2024-12-18T03:36:32.8469982Z    creating: build/custom_test_artifacts/custom-op-build/
2024-12-18T03:36:32.8470904Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-12-18T03:36:32.8471944Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-12-18T03:36:32.8473167Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T03:36:32.8474322Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-12-18T03:36:32.8475460Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T03:36:32.8477594Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T03:36:32.8478814Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T03:36:32.8480175Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T03:36:32.8481546Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T03:36:32.8482764Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T03:36:32.8483977Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T03:36:32.8485407Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T03:36:32.8486850Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T03:36:32.8488259Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T03:36:32.8489687Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T03:36:32.8491145Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T03:36:32.8492591Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T03:36:32.8493808Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-12-18T03:36:32.8494861Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-12-18T03:36:32.8495959Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-12-18T03:36:32.8497104Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-12-18T03:36:32.8498400Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-12-18T03:36:32.8499865Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-12-18T03:36:32.8501243Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-12-18T03:36:32.8502528Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-12-18T03:36:32.8503845Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-12-18T03:36:32.8505171Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-12-18T03:36:32.8506496Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-12-18T03:36:32.8507951Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-12-18T03:36:32.8509250Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-12-18T03:36:32.8510560Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-12-18T03:36:32.8643214Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-12-18T03:36:32.8644643Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-12-18T03:36:32.8646071Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-12-18T03:36:32.8647626Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-12-18T03:36:32.8649069Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-12-18T03:36:32.8650421Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-12-18T03:36:32.8651836Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-12-18T03:36:32.8653667Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-12-18T03:36:32.8655088Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-12-18T03:36:32.8656497Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-12-18T03:36:32.8657878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-12-18T03:36:32.8667667Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-12-18T03:36:32.8744655Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-12-18T03:36:32.8746302Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T03:36:32.8747871Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T03:36:32.8749131Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-12-18T03:36:32.8750259Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-12-18T03:36:32.8751375Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-12-18T03:36:32.8752441Z   inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc  
2024-12-18T03:36:32.8753434Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-12-18T03:36:32.8754385Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-12-18T03:36:32.8755362Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-12-18T03:36:32.8860856Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-12-18T03:36:32.8917942Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-12-18T03:36:32.8918958Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-12-18T03:36:32.8919867Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-12-18T03:36:32.8920898Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-12-18T03:36:32.8922719Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T03:36:32.8924022Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-12-18T03:36:32.8925197Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T03:36:32.8926442Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T03:36:32.8927656Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T03:36:32.8929057Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T03:36:32.8930488Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T03:36:32.8932301Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T03:36:32.8933540Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T03:36:32.8934974Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T03:36:32.8936400Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T03:36:32.8937786Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T03:36:32.8939171Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T03:36:32.8940582Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T03:36:32.8942434Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T03:36:32.8943685Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-12-18T03:36:32.8944741Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-12-18T03:36:32.8945830Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-12-18T03:36:32.8947366Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-12-18T03:36:32.8948753Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-12-18T03:36:32.8950245Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-12-18T03:36:32.8951659Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-12-18T03:36:32.8952993Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-12-18T03:36:32.8954357Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-12-18T03:36:32.8955720Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-12-18T03:36:32.8957089Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-12-18T03:36:32.8958442Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-12-18T03:36:32.8959812Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-12-18T03:36:32.8961515Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-12-18T03:36:32.9021063Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-12-18T03:36:32.9022771Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T03:36:32.9024175Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T03:36:32.9025442Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-12-18T03:36:32.9026588Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-12-18T03:36:32.9027846Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-12-18T03:36:32.9028917Z   inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc  
2024-12-18T03:36:32.9029914Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-12-18T03:36:32.9030853Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-12-18T03:36:32.9031821Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-12-18T03:36:32.9072086Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-12-18T03:36:32.9073632Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-12-18T03:36:32.9074626Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-12-18T03:36:32.9075752Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-12-18T03:36:32.9077089Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T03:36:32.9078361Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-12-18T03:36:32.9079634Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T03:36:32.9080990Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T03:36:32.9082351Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T03:36:32.9084179Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T03:36:32.9085687Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T03:36:32.9087016Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T03:36:32.9088342Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T03:36:32.9089885Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T03:36:32.9091451Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T03:36:32.9092950Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T03:36:32.9094468Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T03:36:32.9096013Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T03:36:32.9097549Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T03:36:32.9098854Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-12-18T03:36:32.9100009Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-12-18T03:36:32.9101213Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-12-18T03:36:32.9102464Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-12-18T03:36:32.9103880Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-12-18T03:36:32.9105469Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-12-18T03:36:32.9107178Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-12-18T03:36:32.9108605Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-12-18T03:36:32.9110097Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-12-18T03:36:32.9111605Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-12-18T03:36:32.9113094Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-12-18T03:36:32.9114571Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-12-18T03:36:32.9116019Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-12-18T03:36:32.9117597Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-12-18T03:36:32.9214655Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-12-18T03:36:32.9216306Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-12-18T03:36:32.9217831Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-12-18T03:36:32.9219539Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-12-18T03:36:32.9221145Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-12-18T03:36:32.9222665Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-12-18T03:36:32.9224565Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-12-18T03:36:32.9226135Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-12-18T03:36:32.9227833Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-12-18T03:36:32.9229377Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-12-18T03:36:32.9230926Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-12-18T03:36:32.9238441Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-12-18T03:36:32.9289539Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-12-18T03:36:32.9291209Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T03:36:32.9292660Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T03:36:32.9293969Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-12-18T03:36:32.9295165Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-12-18T03:36:32.9296366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-12-18T03:36:32.9297537Z   inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc  
2024-12-18T03:36:32.9298620Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-12-18T03:36:32.9299632Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-12-18T03:36:32.9300668Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-12-18T03:36:32.9387332Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-12-18T03:36:32.9425999Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-12-18T03:36:32.9426990Z    creating: build/lib/
2024-12-18T03:36:32.9507101Z   inflating: build/lib/libprotobuf-lite.a  
2024-12-18T03:36:32.9515635Z   inflating: build/lib/libpthreadpool.a  
2024-12-18T03:36:32.9522937Z   inflating: build/lib/libcpuinfo.a  
2024-12-18T03:36:32.9529989Z   inflating: build/lib/libcpuinfo_internals.a  
2024-12-18T03:36:32.9944634Z   inflating: build/lib/libprotobuf.a  
2024-12-18T03:36:33.0402007Z   inflating: build/lib/libprotoc.a   
2024-12-18T03:36:33.0402723Z   inflating: build/lib/libclog.a     
2024-12-18T03:36:33.0420029Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-12-18T03:36:33.0421579Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-12-18T03:36:33.0438314Z   inflating: build/lib/libnnpack.a   
2024-12-18T03:36:33.0607265Z   inflating: build/lib/libmicrokernels-prod.a  
2024-12-18T03:36:33.1390157Z   inflating: build/lib/libmicrokernels-all.a  
2024-12-18T03:36:33.1452172Z   inflating: build/lib/libgtest.a    
2024-12-18T03:36:33.1469183Z   inflating: build/lib/libgmock.a    
2024-12-18T03:36:33.1469983Z   inflating: build/lib/libgtest_main.a  
2024-12-18T03:36:33.1470633Z   inflating: build/lib/libgmock_main.a  
2024-12-18T03:36:33.1536985Z   inflating: build/lib/libbenchmark.a  
2024-12-18T03:36:33.1537725Z   inflating: build/lib/libbenchmark_main.a  
2024-12-18T03:36:33.1618007Z   inflating: build/lib/libXNNPACK.a  
2024-12-18T03:36:33.1624716Z   inflating: build/lib/libittnotify.a  
2024-12-18T03:36:33.1681741Z   inflating: build/lib/libasmjit.a   
2024-12-18T03:36:33.2842289Z   inflating: build/lib/libfbgemm.a   
2024-12-18T03:36:33.2867233Z   inflating: build/lib/libtensorpipe_uv.a  
2024-12-18T03:36:33.3393727Z   inflating: build/lib/libtensorpipe.a  
2024-12-18T03:36:33.3432129Z   inflating: build/lib/libonnx_proto.a  
2024-12-18T03:36:33.3535615Z   inflating: build/lib/libgloo.a     
2024-12-18T03:36:33.3895279Z   inflating: build/lib/libgloo_hip.a  
2024-12-18T03:36:33.4563848Z   inflating: build/lib/libonnx.a     
2024-12-18T03:36:33.4582284Z   inflating: build/lib/libfmt.a      
2024-12-18T03:36:33.4877457Z   inflating: build/lib/libkineto.a   
2024-12-18T03:36:34.4352788Z   inflating: build/lib/libdnnl.a     
2024-12-18T03:36:34.4441664Z   inflating: build/lib/libc10.so     
2024-12-18T03:36:34.4442433Z   inflating: build/lib/libtorch_global_deps.so  
2024-12-18T03:36:34.4485606Z   inflating: build/lib/libc10_hip.so  
2024-12-18T03:36:34.4486338Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-12-18T03:36:36.7227666Z   inflating: build/lib/libtorch_cpu.so  
2024-12-18T03:36:36.7231783Z   inflating: build/lib/libunbox_lib.a  
2024-12-18T03:36:36.7235479Z   inflating: build/lib/libshm.so     
2024-12-18T03:36:37.9190346Z   inflating: build/lib/libtorch_hip.so  
2024-12-18T03:36:37.9191075Z   inflating: build/lib/libtorch.so   
2024-12-18T03:36:37.9209274Z   inflating: build/lib/libjitbackend_test.so  
2024-12-18T03:36:37.9272436Z   inflating: build/lib/libtorchbind_test.so  
2024-12-18T03:36:37.9294164Z   inflating: build/lib/libbackend_with_compiler.so  
2024-12-18T03:36:37.9316593Z   inflating: build/lib/libaoti_custom_ops.so  
2024-12-18T03:36:38.1177035Z   inflating: build/lib/libtorch_python.so  
2024-12-18T03:36:38.1207507Z   inflating: build/lib/libnnapi_backend.so  
2024-12-18T03:36:38.1208219Z    creating: build/bin/
2024-12-18T03:36:38.1208729Z    creating: build/bin/CMakeFiles/
2024-12-18T03:36:38.1209334Z   inflating: build/bin/cmake_install.cmake  
2024-12-18T03:36:38.1210025Z   inflating: build/bin/CTestTestfile.cmake  
2024-12-18T03:36:38.1608160Z   inflating: build/bin/protoc-3.13.0.0  
2024-12-18T03:36:38.2006268Z   inflating: build/bin/protoc        
2024-12-18T03:36:38.2053972Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-12-18T03:36:38.2102867Z   inflating: build/bin/c10_DeviceGuard_test  
2024-12-18T03:36:38.2151958Z   inflating: build/bin/c10_Device_test  
2024-12-18T03:36:38.2207760Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-12-18T03:36:38.2258252Z   inflating: build/bin/c10_Scalar_test  
2024-12-18T03:36:38.2304554Z   inflating: build/bin/c10_StreamGuard_test  
2024-12-18T03:36:38.2356327Z   inflating: build/bin/c10_SymInt_test  
2024-12-18T03:36:38.2407703Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-12-18T03:36:38.2460246Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-12-18T03:36:38.2513769Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-12-18T03:36:38.2579596Z   inflating: build/bin/c10_cow_test  
2024-12-18T03:36:38.2625960Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-12-18T03:36:38.2675997Z   inflating: build/bin/c10_Bitset_test  
2024-12-18T03:36:38.2723287Z   inflating: build/bin/c10_ArrayRef_test  
2024-12-18T03:36:38.2776443Z   inflating: build/bin/c10_LeftRight_test  
2024-12-18T03:36:38.2824634Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-12-18T03:36:38.2872577Z   inflating: build/bin/c10_Half_test  
2024-12-18T03:36:38.2924995Z   inflating: build/bin/c10_Metaprogramming_test  
2024-12-18T03:36:38.2975687Z   inflating: build/bin/c10_NetworkFlow_test  
2024-12-18T03:36:38.3022510Z   inflating: build/bin/c10_Synchronized_test  
2024-12-18T03:36:38.3071400Z   inflating: build/bin/c10_TypeIndex_test  
2024-12-18T03:36:38.3124168Z   inflating: build/bin/c10_ThreadLocal_test  
2024-12-18T03:36:38.3172663Z   inflating: build/bin/c10_TypeList_test  
2024-12-18T03:36:38.3219263Z   inflating: build/bin/c10_TypeTraits_test  
2024-12-18T03:36:38.3268046Z   inflating: build/bin/c10_accumulate_test  
2024-12-18T03:36:38.3321145Z   inflating: build/bin/c10_bfloat16_test  
2024-12-18T03:36:38.3368810Z   inflating: build/bin/c10_bit_cast_test  
2024-12-18T03:36:38.3420986Z   inflating: build/bin/c10_complex_test  
2024-12-18T03:36:38.3474474Z   inflating: build/bin/c10_complex_math_test  
2024-12-18T03:36:38.3521712Z   inflating: build/bin/c10_error_test  
2024-12-18T03:36:38.3571464Z   inflating: build/bin/c10_exception_test  
2024-12-18T03:36:38.3619535Z   inflating: build/bin/c10_flags_test  
2024-12-18T03:36:38.3667277Z   inflating: build/bin/c10_generic_math_test  
2024-12-18T03:36:38.3717824Z   inflating: build/bin/c10_lazy_test  
2024-12-18T03:36:38.3765970Z   inflating: build/bin/c10_irange_test  
2024-12-18T03:36:38.3921091Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-12-18T03:36:38.3974987Z   inflating: build/bin/c10_logging_test  
2024-12-18T03:36:38.4046844Z   inflating: build/bin/c10_optional_test  
2024-12-18T03:36:38.4105753Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-12-18T03:36:38.4156514Z   inflating: build/bin/c10_registry_test  
2024-12-18T03:36:38.4301184Z   inflating: build/bin/c10_small_vector_test  
2024-12-18T03:36:38.4350314Z   inflating: build/bin/c10_string_util_test  
2024-12-18T03:36:38.4399603Z   inflating: build/bin/c10_ssize_test  
2024-12-18T03:36:38.4456065Z   inflating: build/bin/c10_string_view_test  
2024-12-18T03:36:38.4504112Z   inflating: build/bin/c10_tempfile_test  
2024-12-18T03:36:38.4547216Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-12-18T03:36:38.4600378Z   inflating: build/bin/c10_typeid_test  
2024-12-18T03:36:38.4646735Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2024-12-18T03:36:38.4693265Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2024-12-18T03:36:38.4739575Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2024-12-18T03:36:38.4786004Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2024-12-18T03:36:38.4832441Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-12-18T03:36:38.4879047Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2024-12-18T03:36:38.4925108Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2024-12-18T03:36:38.4971565Z   inflating: build/bin/c10_hip_HIPTest  
2024-12-18T03:36:38.5334998Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-12-18T03:36:38.5707248Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-12-18T03:36:38.6092377Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-12-18T03:36:38.6142942Z   inflating: build/bin/test_edge_op_registration  
2024-12-18T03:36:38.6193352Z   inflating: build/bin/BackoffTest   
2024-12-18T03:36:38.6243491Z   inflating: build/bin/FileStoreTest  
2024-12-18T03:36:38.6296003Z   inflating: build/bin/TCPStoreTest  
2024-12-18T03:36:38.6346115Z   inflating: build/bin/HashStoreTest  
2024-12-18T03:36:38.6408177Z   inflating: build/bin/ProcessGroupGlooTest  
2024-12-18T03:36:38.6409841Z   inflating: build/bin/example_allreduce  
2024-12-18T03:36:38.6463446Z   inflating: build/bin/static_runtime_bench  
2024-12-18T03:36:38.6699873Z   inflating: build/bin/static_runtime_test  
2024-12-18T03:36:38.6769437Z   inflating: build/bin/Dict_test     
2024-12-18T03:36:38.6816229Z   inflating: build/bin/hip_half_test  
2024-12-18T03:36:38.6865349Z   inflating: build/bin/Dimname_test  
2024-12-18T03:36:38.6926978Z   inflating: build/bin/MaybeOwned_test  
2024-12-18T03:36:38.6981898Z   inflating: build/bin/NamedTensor_test  
2024-12-18T03:36:38.7037446Z   inflating: build/bin/apply_utils_test  
2024-12-18T03:36:38.7093307Z   inflating: build/bin/atest         
2024-12-18T03:36:38.7153062Z   inflating: build/bin/basic         
2024-12-18T03:36:38.7205068Z   inflating: build/bin/broadcast_test  
2024-12-18T03:36:38.7253565Z   inflating: build/bin/cpu_allocator_test  
2024-12-18T03:36:38.7304206Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-12-18T03:36:38.7360651Z   inflating: build/bin/cpu_generator_test  
2024-12-18T03:36:38.7447221Z   inflating: build/bin/cpu_rng_test  
2024-12-18T03:36:38.7494781Z   inflating: build/bin/dispatch_key_set_test  
2024-12-18T03:36:38.7543081Z   inflating: build/bin/dlconvertor_test  
2024-12-18T03:36:38.7598061Z   inflating: build/bin/extension_backend_test  
2024-12-18T03:36:38.7650481Z   inflating: build/bin/half_test     
2024-12-18T03:36:38.7739736Z   inflating: build/bin/ivalue_test   
2024-12-18T03:36:38.7787085Z   inflating: build/bin/lazy_tensor_test  
2024-12-18T03:36:38.7838321Z   inflating: build/bin/math_kernel_test  
2024-12-18T03:36:38.7886463Z   inflating: build/bin/operator_name_test  
2024-12-18T03:36:38.7937816Z   inflating: build/bin/memory_format_test  
2024-12-18T03:36:38.7988033Z   inflating: build/bin/memory_overlapping_test  
2024-12-18T03:36:38.8041476Z   inflating: build/bin/native_test   
2024-12-18T03:36:38.8091725Z   inflating: build/bin/mobile_memory_cleanup  
2024-12-18T03:36:38.8140626Z   inflating: build/bin/operators_test  
2024-12-18T03:36:38.8190109Z   inflating: build/bin/packedtensoraccessor_test  
2024-12-18T03:36:38.8253310Z   inflating: build/bin/pow_test      
2024-12-18T03:36:38.8300738Z   inflating: build/bin/reduce_ops_test  
2024-12-18T03:36:38.8355415Z   inflating: build/bin/quantized_test  
2024-12-18T03:36:38.8403891Z   inflating: build/bin/reportMemoryUsage_test  
2024-12-18T03:36:38.8458735Z   inflating: build/bin/scalar_tensor_test  
2024-12-18T03:36:38.8507113Z   inflating: build/bin/StorageUtils_test  
2024-12-18T03:36:38.8562611Z   inflating: build/bin/scalar_test   
2024-12-18T03:36:38.8612888Z   inflating: build/bin/stride_properties_test  
2024-12-18T03:36:38.8688024Z   inflating: build/bin/tensor_iterator_test  
2024-12-18T03:36:38.8739554Z   inflating: build/bin/test_parallel  
2024-12-18T03:36:38.8792460Z   inflating: build/bin/type_ptr_test  
2024-12-18T03:36:38.8793954Z   inflating: build/bin/thread_init_test  
2024-12-18T03:36:38.8795446Z   inflating: build/bin/verify_api_visibility  
2024-12-18T03:36:38.8846207Z   inflating: build/bin/undefined_tensor_test  
2024-12-18T03:36:38.8902639Z   inflating: build/bin/type_test     
2024-12-18T03:36:38.8968395Z   inflating: build/bin/legacy_vmap_test  
2024-12-18T03:36:38.9017391Z   inflating: build/bin/weakref_test  
2024-12-18T03:36:38.9117956Z   inflating: build/bin/List_test     
2024-12-18T03:36:38.9166738Z   inflating: build/bin/wrapdim_test  
2024-12-18T03:36:38.9223451Z   inflating: build/bin/IListRef_test  
2024-12-18T03:36:38.9272674Z   inflating: build/bin/xla_tensor_test  
2024-12-18T03:36:38.9385601Z   inflating: build/bin/kernel_function_legacy_test  
2024-12-18T03:36:38.9475891Z   inflating: build/bin/kernel_function_test  
2024-12-18T03:36:38.9595198Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-12-18T03:36:38.9691967Z   inflating: build/bin/kernel_lambda_test  
2024-12-18T03:36:38.9755392Z   inflating: build/bin/KernelFunction_test  
2024-12-18T03:36:38.9813414Z   inflating: build/bin/kernel_stackbased_test  
2024-12-18T03:36:38.9862270Z   inflating: build/bin/CppSignature_test  
2024-12-18T03:36:38.9952386Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-12-18T03:36:38.9998620Z   inflating: build/bin/op_allowlist_test  
2024-12-18T03:36:39.0059042Z   inflating: build/bin/inline_container_test  
2024-12-18T03:36:39.0338758Z   inflating: build/bin/op_registration_test  
2024-12-18T03:36:39.0391410Z   inflating: build/bin/backend_fallback_test  
2024-12-18T03:36:39.0438601Z   inflating: build/bin/hip_complex_math_test  
2024-12-18T03:36:39.0488326Z   inflating: build/bin/hip_apply_test  
2024-12-18T03:36:39.0535020Z   inflating: build/bin/hip_complex_test  
2024-12-18T03:36:39.0581242Z   inflating: build/bin/hip_distributions_test  
2024-12-18T03:36:39.0627944Z   inflating: build/bin/hip_generator_test  
2024-12-18T03:36:39.0676299Z   inflating: build/bin/hip_dlconvertor_test  
2024-12-18T03:36:39.0723071Z   inflating: build/bin/hip_integer_divider_test  
2024-12-18T03:36:39.0769914Z   inflating: build/bin/hip_optional_test  
2024-12-18T03:36:39.0816654Z   inflating: build/bin/hip_packedtensoraccessor_test  
2024-12-18T03:36:39.0862968Z   inflating: build/bin/hip_vectorized_test  
2024-12-18T03:36:39.1388831Z   inflating: build/bin/test_jit      
2024-12-18T03:36:39.2113080Z   inflating: build/bin/test_tensorexpr  
2024-12-18T03:36:39.2125879Z   inflating: build/bin/tutorial_tensorexpr  
2024-12-18T03:36:39.2177948Z   inflating: build/bin/test_dist_autograd  
2024-12-18T03:36:39.2242946Z   inflating: build/bin/test_cpp_rpc  
2024-12-18T03:36:39.2244376Z   inflating: build/bin/parallel_benchmark  
2024-12-18T03:36:39.3312405Z   inflating: build/bin/test_api      
2024-12-18T03:36:39.3320271Z   inflating: build/bin/aot_model_compiler_test  
2024-12-18T03:36:39.3383169Z   inflating: build/bin/test_mobile_nnc  
2024-12-18T03:36:39.3704824Z   inflating: build/bin/test_lazy     
2024-12-18T03:36:39.3708045Z   inflating: build/bin/torch_shm_manager  
2024-12-18T03:36:39.3708691Z    creating: .additional_ci_files/
2024-12-18T03:36:39.3790517Z   inflating: .additional_ci_files/test-times.json  
2024-12-18T03:36:39.4110424Z   inflating: .additional_ci_files/test-class-times.json  
2024-12-18T03:36:39.4164432Z ##[group]Run rm artifacts.zip
2024-12-18T03:36:39.4165003Z rm artifacts.zip
2024-12-18T03:36:39.4220022Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:39.4220725Z env:
2024-12-18T03:36:39.4221127Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:39.4221726Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:39.4222736Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:39.4223654Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:39.4224167Z   AWS_REGION: us-east-1
2024-12-18T03:36:39.4224748Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:39.4225428Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:39.4236810Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:39.4237372Z ##[endgroup]
2024-12-18T03:36:39.6633027Z ##[group]Run df -H
2024-12-18T03:36:39.6633478Z df -H
2024-12-18T03:36:39.6685193Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:39.6686030Z env:
2024-12-18T03:36:39.6686525Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:39.6687211Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:39.6688246Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:39.6689186Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:39.6689706Z   AWS_REGION: us-east-1
2024-12-18T03:36:39.6690271Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:39.6690943Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:39.6702384Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:39.6702853Z ##[endgroup]
2024-12-18T03:36:39.6862931Z Filesystem                                          Size  Used Avail Use% Mounted on
2024-12-18T03:36:39.6866287Z tmpfs                                               109G   33M  109G   1% /run
2024-12-18T03:36:39.6867327Z /dev/nvme0n1p2                                      944G   65G  832G   8% /
2024-12-18T03:36:39.6868178Z tmpfs                                               542G   33k  542G   1% /dev/shm
2024-12-18T03:36:39.6868924Z tmpfs                                               5.3M     0  5.3M   0% /run/lock
2024-12-18T03:36:39.6869882Z /dev/nvme0n1p1                                      536M  6.4M  530M   2% /boot/efi
2024-12-18T03:36:39.6871352Z /dev/nvme1n1p1                                      3.8T  1.1T  2.5T  31% /media/4TB
2024-12-18T03:36:39.6872360Z tmpfs                                               109G   33k  109G   1% /run/user/1307800118
2024-12-18T03:36:39.6873378Z 172.18.148.8:/export/amd2                           5.5T  260G  5.3T   5% /mnt
2024-12-18T03:36:39.6874666Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch     108G  468M  107G   1% /home/amd-pytorch
2024-12-18T03:36:39.6876086Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage          110T   72T   39T  65% /groups
2024-12-18T03:36:39.6877181Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch  5.5T  375G  5.2T   7% /scratch
2024-12-18T03:36:39.6878312Z pure1.jax.cs.cpe.ice.amd.com:/homes/okakarpa        108G  137M  108G   1% /home/okakarpa
2024-12-18T03:36:39.6879444Z pure1.jax.cs.cpe.ice.amd.com:/homes/mkunredd        108G  1.1M  108G   1% /home/mkunredd
2024-12-18T03:36:39.6943780Z Prepare all required actions
2024-12-18T03:36:39.6944555Z Getting action download info
2024-12-18T03:36:39.8883051Z ##[group]Run ./.github/actions/download-td-artifacts
2024-12-18T03:36:39.8883723Z with:
2024-12-18T03:36:39.8884127Z env:
2024-12-18T03:36:39.8884547Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:39.8885169Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:39.8886243Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:39.8887228Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:39.8887776Z   AWS_REGION: us-east-1
2024-12-18T03:36:39.8888375Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:39.8889097Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:39.8901163Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:39.8901668Z ##[endgroup]
2024-12-18T03:36:39.8959892Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T03:36:39.8960527Z with:
2024-12-18T03:36:39.8960957Z   name: td_results
2024-12-18T03:36:39.8961477Z   s3-bucket: gha-artifacts
2024-12-18T03:36:39.8961989Z   region: us-east-1
2024-12-18T03:36:39.8962417Z env:
2024-12-18T03:36:39.8962830Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:39.8963415Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:39.8964501Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:39.8965484Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:39.8966018Z   AWS_REGION: us-east-1
2024-12-18T03:36:39.8966603Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:39.8967303Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:39.8978824Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:39.8979313Z ##[endgroup]
2024-12-18T03:36:40.3543914Z (node:3129413) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T03:36:40.3544844Z 
2024-12-18T03:36:40.3545229Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T03:36:40.3546259Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T03:36:40.3547457Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T03:36:40.4955078Z Found 1 objects with prefix pytorch/pytorch/12383255654/td_results/
2024-12-18T03:36:40.4956322Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2024-12-18T03:36:40.6980175Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2024-12-18T03:36:40.6990686Z Artifact download has finished successfully
2024-12-18T03:36:40.7474708Z ##[group]Run mkdir -p .additional_ci_files
2024-12-18T03:36:40.7475394Z mkdir -p .additional_ci_files
2024-12-18T03:36:40.7476299Z mv td_results.json .additional_ci_files/td_results.json || true
2024-12-18T03:36:40.7525276Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:40.7526076Z env:
2024-12-18T03:36:40.7526591Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:40.7527237Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:40.7528287Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:40.7529235Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:40.7529767Z   AWS_REGION: us-east-1
2024-12-18T03:36:40.7530370Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:40.7531048Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:40.7541939Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:40.7542419Z ##[endgroup]
2024-12-18T03:36:40.7762544Z ##[group]Run .github/scripts/parse_ref.py
2024-12-18T03:36:40.7763437Z .github/scripts/parse_ref.py
2024-12-18T03:36:40.7815927Z shell: /usr/bin/bash -e {0}
2024-12-18T03:36:40.7816453Z env:
2024-12-18T03:36:40.7816858Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:40.7817463Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:40.7818505Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:40.7819898Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:40.7820412Z   AWS_REGION: us-east-1
2024-12-18T03:36:40.7820993Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:40.7821671Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:40.7833105Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:40.7833583Z ##[endgroup]
2024-12-18T03:36:40.8368734Z Prepare all required actions
2024-12-18T03:36:40.8429487Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-12-18T03:36:40.8430174Z with:
2024-12-18T03:36:40.8430974Z   github-token: ***
2024-12-18T03:36:40.8431449Z env:
2024-12-18T03:36:40.8431864Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:40.8432487Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:40.8433567Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:40.8434556Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:40.8435105Z   AWS_REGION: us-east-1
2024-12-18T03:36:40.8435696Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:40.8436428Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:40.8447930Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:40.8448428Z ##[endgroup]
2024-12-18T03:36:40.8479455Z ##[group]Run set -eux
2024-12-18T03:36:40.8479968Z set -eux
2024-12-18T03:36:40.8480847Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T03:36:40.8537509Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:40.8538127Z env:
2024-12-18T03:36:40.8538486Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:40.8539008Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:40.8539932Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:40.8540762Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:40.8541222Z   AWS_REGION: us-east-1
2024-12-18T03:36:40.8541725Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:40.8542335Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:40.8552168Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:40.8552817Z   GITHUB_TOKEN: ***
2024-12-18T03:36:40.8553208Z ##[endgroup]
2024-12-18T03:36:40.8641603Z + python3 .github/scripts/get_workflow_job_id.py 12383255654 gpu6991.jax.cs.cpe.ice.amd.com
2024-12-18T03:36:42.3469479Z setting job-id=34566687343
2024-12-18T03:36:42.3470733Z setting job-name=linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:36:42.3848363Z Prepare all required actions
2024-12-18T03:36:42.3849191Z Getting action download info
2024-12-18T03:36:42.5691624Z ##[group]Run ./.github/actions/filter-test-configs
2024-12-18T03:36:42.5692285Z with:
2024-12-18T03:36:42.5693050Z   github-token: ***
2024-12-18T03:36:42.5695279Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}
2024-12-18T03:36:42.5698035Z   job-name: linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:36:42.5699003Z env:
2024-12-18T03:36:42.5699413Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:42.5700055Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:42.5701006Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:42.5701870Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:42.5702382Z   AWS_REGION: us-east-1
2024-12-18T03:36:42.5702928Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:42.5703568Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:42.5713345Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:42.5713814Z ##[endgroup]
2024-12-18T03:36:42.5778909Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T03:36:42.5779859Z with:
2024-12-18T03:36:42.5780247Z   shell: bash
2024-12-18T03:36:42.5780671Z   timeout_minutes: 10
2024-12-18T03:36:42.5781147Z   max_attempts: 5
2024-12-18T03:36:42.5781588Z   retry_wait_seconds: 30
2024-12-18T03:36:42.5782848Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-12-18T03:36:42.5784169Z   polling_interval_seconds: 1
2024-12-18T03:36:42.5784678Z   warning_on_retry: true
2024-12-18T03:36:42.5785158Z   continue_on_error: false
2024-12-18T03:36:42.5785611Z env:
2024-12-18T03:36:42.5786013Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:42.5786580Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:42.5787712Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:42.5788586Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:42.5789121Z   AWS_REGION: us-east-1
2024-12-18T03:36:42.5789718Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:42.5790362Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:42.5800788Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:42.5801570Z   GITHUB_TOKEN: ***
2024-12-18T03:36:42.5802093Z ##[endgroup]
2024-12-18T03:36:42.6753667Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-12-18T03:36:42.9307147Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T03:36:43.0926404Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2024-12-18T03:36:43.0929325Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1)
2024-12-18T03:36:43.1017237Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2024-12-18T03:36:43.1020576Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2024-12-18T03:36:43.1024075Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2024-12-18T03:36:43.1034703Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2024-12-18T03:36:43.6750714Z Command completed after 1 attempt(s).
2024-12-18T03:36:43.6897609Z ##[group]Run set -x
2024-12-18T03:36:43.6898688Z set -x
2024-12-18T03:36:43.6899156Z 
2024-12-18T03:36:43.6899911Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T03:36:43.6900804Z # in runner workspace
2024-12-18T03:36:43.6901523Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-12-18T03:36:43.6957222Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:43.6957977Z env:
2024-12-18T03:36:43.6958428Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:43.6959100Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:43.6960181Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:43.6961144Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:43.6961700Z   AWS_REGION: us-east-1
2024-12-18T03:36:43.6962344Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:43.6963059Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:43.6974258Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:43.6974761Z ##[endgroup]
2024-12-18T03:36:43.7072511Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-12-18T03:36:43.7540771Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T03:36:43.7541606Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T03:36:43.7542281Z echo "Job name: ${JOB_NAME}"
2024-12-18T03:36:43.7542885Z 
2024-12-18T03:36:43.7543668Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T03:36:43.7545106Z # in runner workspace
2024-12-18T03:36:43.7545969Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-12-18T03:36:43.7547068Z   --workflow "${GITHUB_WORKFLOW}" \
2024-12-18T03:36:43.7547767Z   --job-name "${JOB_NAME}" \
2024-12-18T03:36:43.7550595Z   --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}" \
2024-12-18T03:36:43.7553399Z   --selected-test-configs "" \
2024-12-18T03:36:43.7554052Z   --pr-number "${PR_NUMBER}" \
2024-12-18T03:36:43.7554685Z   --tag "${TAG}" \
2024-12-18T03:36:43.7555295Z   --event-name "${EVENT_NAME}" \
2024-12-18T03:36:43.7555950Z   --schedule "${SCHEDULE}" \
2024-12-18T03:36:43.7556571Z   --branch "${HEAD_BRANCH}"
2024-12-18T03:36:43.7613648Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:43.7614424Z env:
2024-12-18T03:36:43.7614884Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:43.7615569Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:43.7616711Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:43.7617824Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:43.7618402Z   AWS_REGION: us-east-1
2024-12-18T03:36:43.7619033Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:43.7619793Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:43.7631469Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:43.7632261Z   GITHUB_TOKEN: ***
2024-12-18T03:36:43.7633257Z   JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:36:43.7634356Z   PR_NUMBER: 
2024-12-18T03:36:43.7634822Z   TAG: 
2024-12-18T03:36:43.7635261Z   EVENT_NAME: push
2024-12-18T03:36:43.7635767Z   SCHEDULE: 
2024-12-18T03:36:43.7636224Z   HEAD_BRANCH: 
2024-12-18T03:36:43.7636713Z ##[endgroup]
2024-12-18T03:36:43.7731146Z Workflow: periodic
2024-12-18T03:36:43.7732861Z Job name: linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:36:44.2257175Z ##[group]Run echo "Filtered matrix:"
2024-12-18T03:36:44.2257867Z echo "Filtered matrix:"
2024-12-18T03:36:44.2260397Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}"
2024-12-18T03:36:44.2262944Z 
2024-12-18T03:36:44.2263373Z echo
2024-12-18T03:36:44.2263898Z echo "Is the current job unstable? False"
2024-12-18T03:36:44.2264754Z 
2024-12-18T03:36:44.2265175Z echo
2024-12-18T03:36:44.2265671Z echo "Is keep-going label set? False"
2024-12-18T03:36:44.2266268Z 
2024-12-18T03:36:44.2266800Z echo
2024-12-18T03:36:44.2267272Z echo "Renabled issues? "
2024-12-18T03:36:44.2321839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:44.2322720Z env:
2024-12-18T03:36:44.2323252Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:44.2323976Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:44.2325059Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:44.2326044Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:44.2327059Z   AWS_REGION: us-east-1
2024-12-18T03:36:44.2327761Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:44.2328475Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:44.2340014Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:44.2340527Z ##[endgroup]
2024-12-18T03:36:44.2437794Z Filtered matrix:
2024-12-18T03:36:44.2440320Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}]}
2024-12-18T03:36:44.2442558Z 
2024-12-18T03:36:44.2442787Z Is the current job unstable? False
2024-12-18T03:36:44.2443300Z 
2024-12-18T03:36:44.2443755Z Is keep-going label set? False
2024-12-18T03:36:44.2444204Z 
2024-12-18T03:36:44.2444411Z Renabled issues? 
2024-12-18T03:36:44.2528255Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T03:36:44.2529503Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T03:36:44.2587024Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T03:36:44.2587815Z env:
2024-12-18T03:36:44.2588274Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:44.2588964Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:44.2590057Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:44.2591038Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:44.2591592Z   AWS_REGION: us-east-1
2024-12-18T03:36:44.2592215Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:44.2592946Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:44.2604462Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:44.2604980Z   JOB_TIMEOUT: 300
2024-12-18T03:36:44.2605446Z ##[endgroup]
2024-12-18T03:36:44.2788597Z ##[group]Run set -x
2024-12-18T03:36:44.2789229Z set -x
2024-12-18T03:36:44.2789686Z 
2024-12-18T03:36:44.2790169Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-12-18T03:36:44.2790884Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-12-18T03:36:44.2791601Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-12-18T03:36:44.2792254Z   TEST_COMMAND=.ci/caffe2/test.sh
2024-12-18T03:36:44.2792805Z else
2024-12-18T03:36:44.2793265Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T03:36:44.2793806Z fi
2024-12-18T03:36:44.2794180Z 
2024-12-18T03:36:44.2794786Z # detached container should get cleaned up by teardown_ec2_linux
2024-12-18T03:36:44.2795741Z # TODO: Stop building test binaries as part of the build phase
2024-12-18T03:36:44.2796610Z # Used for GPU_FLAG since that doesn't play nice
2024-12-18T03:36:44.2797384Z # shellcheck disable=SC2086,SC2090
2024-12-18T03:36:44.2797991Z container_name=$(docker run \
2024-12-18T03:36:44.2798569Z   ${GPU_FLAG:-} \
2024-12-18T03:36:44.2799077Z   -e BUILD_ENVIRONMENT \
2024-12-18T03:36:44.2799616Z   -e PR_NUMBER \
2024-12-18T03:36:44.2800106Z   -e GITHUB_ACTIONS \
2024-12-18T03:36:44.2800626Z   -e GITHUB_REPOSITORY \
2024-12-18T03:36:44.2801175Z   -e GITHUB_WORKFLOW \
2024-12-18T03:36:44.2801707Z   -e GITHUB_JOB \
2024-12-18T03:36:44.2802206Z   -e GITHUB_RUN_ID \
2024-12-18T03:36:44.2802706Z   -e GITHUB_RUN_NUMBER \
2024-12-18T03:36:44.2803252Z   -e GITHUB_RUN_ATTEMPT \
2024-12-18T03:36:44.2803771Z   -e JOB_ID \
2024-12-18T03:36:44.2804230Z   -e JOB_NAME \
2024-12-18T03:36:44.2804691Z   -e BRANCH \
2024-12-18T03:36:44.2805129Z   -e SHA1 \
2024-12-18T03:36:44.2805584Z   -e AWS_DEFAULT_REGION \
2024-12-18T03:36:44.2806121Z   -e IN_WHEEL_TEST \
2024-12-18T03:36:44.2806617Z   -e SHARD_NUMBER \
2024-12-18T03:36:44.2807609Z   -e TEST_CONFIG \
2024-12-18T03:36:44.2808102Z   -e NUM_TEST_SHARDS \
2024-12-18T03:36:44.2808609Z   -e REENABLED_ISSUES \
2024-12-18T03:36:44.2809153Z   -e CONTINUE_THROUGH_ERROR \
2024-12-18T03:36:44.2809714Z   -e VERBOSE_TEST_LOGS \
2024-12-18T03:36:44.2810238Z   -e TEST_SHOWLOCALS \
2024-12-18T03:36:44.2810739Z   -e NO_TEST_TIMEOUT \
2024-12-18T03:36:44.2811232Z   -e NO_TD \
2024-12-18T03:36:44.2811749Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-12-18T03:36:44.2812341Z   -e SCCACHE_BUCKET \
2024-12-18T03:36:44.2812894Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-12-18T03:36:44.2813522Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-12-18T03:36:44.2814174Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-12-18T03:36:44.2814796Z   -e TESTS_TO_INCLUDE \
2024-12-18T03:36:44.2815423Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-12-18T03:36:44.2816112Z   --ulimit stack=10485760:83886080 \
2024-12-18T03:36:44.2816676Z   --ulimit core=0 \
2024-12-18T03:36:44.2817235Z   --security-opt seccomp=unconfined \
2024-12-18T03:36:44.2817835Z   --cap-add=SYS_PTRACE \
2024-12-18T03:36:44.2818365Z   --shm-size="8g" \
2024-12-18T03:36:44.2818843Z   --tty \
2024-12-18T03:36:44.2819269Z   --detach \
2024-12-18T03:36:44.2819755Z   --name="${container_name}" \
2024-12-18T03:36:44.2820314Z   --user jenkins \
2024-12-18T03:36:44.2820936Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-12-18T03:36:44.2821642Z   -w /var/lib/jenkins/workspace \
2024-12-18T03:36:44.2822206Z   "${DOCKER_IMAGE}"
2024-12-18T03:36:44.2822665Z )
2024-12-18T03:36:44.2823108Z # save container name for later step
2024-12-18T03:36:44.2824200Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2024-12-18T03:36:44.2825499Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2024-12-18T03:36:44.2827250Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2024-12-18T03:36:44.2883513Z shell: /usr/bin/bash -e {0}
2024-12-18T03:36:44.2884045Z env:
2024-12-18T03:36:44.2884449Z   GIT_DEFAULT_BRANCH: main
2024-12-18T03:36:44.2885044Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T03:36:44.2886075Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T03:36:44.2886995Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T03:36:44.2887508Z   AWS_REGION: us-east-1
2024-12-18T03:36:44.2888084Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T03:36:44.2888775Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T03:36:44.2899597Z   AWS_SESSION_TOKEN: ***
2024-12-18T03:36:44.2900158Z   BUILD_ENVIRONMENT: linux-focal-rocm6.2-py3.10
2024-12-18T03:36:44.2900787Z   PR_NUMBER: 
2024-12-18T03:36:44.2901244Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-12-18T03:36:44.2901823Z   GITHUB_WORKFLOW: periodic
2024-12-18T03:36:44.2902311Z   GITHUB_JOB: test
2024-12-18T03:36:44.2902744Z   GITHUB_RUN_ID: 12383255654
2024-12-18T03:36:44.2903251Z   GITHUB_RUN_NUMBER: 15427
2024-12-18T03:36:44.2903734Z   GITHUB_RUN_ATTEMPT: 1
2024-12-18T03:36:44.2904179Z   JOB_ID: 34566687343
2024-12-18T03:36:44.2905085Z   JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:36:44.2906118Z   BRANCH: release/2.6
2024-12-18T03:36:44.2906950Z   SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:36:44.2907583Z   CONTINUE_THROUGH_ERROR: False
2024-12-18T03:36:44.2908099Z   VERBOSE_TEST_LOGS: False
2024-12-18T03:36:44.2908582Z   TEST_SHOWLOCALS: False
2024-12-18T03:36:44.2909057Z   NO_TEST_TIMEOUT: False
2024-12-18T03:36:44.2909504Z   NO_TD: False
2024-12-18T03:36:44.2909925Z   TEST_CONFIG: distributed
2024-12-18T03:36:44.2910803Z   SHARD_NUMBER: 3
2024-12-18T03:36:44.2911227Z   NUM_TEST_SHARDS: 3
2024-12-18T03:36:44.2911672Z   REENABLED_ISSUES: 
2024-12-18T03:36:44.2912217Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-12-18T03:36:44.2913553Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:36:44.2915064Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-12-18T03:36:44.2915887Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-12-18T03:36:44.2916466Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-12-18T03:36:44.2917014Z   TESTS_TO_INCLUDE: 
2024-12-18T03:36:44.2917451Z ##[endgroup]
2024-12-18T03:36:44.3013743Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2024-12-18T03:36:44.3014354Z + [[ linux-focal-rocm6.2-py3.10 == *onnx* ]]
2024-12-18T03:36:44.3014945Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T03:36:44.3035761Z +++ nproc --ignore=2
2024-12-18T03:36:44.3072182Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_12383255654 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T03:36:44.7015492Z + container_name=ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T03:36:44.7016805Z + echo CONTAINER_NAME=ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T03:36:44.7025074Z + docker exec -t ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2024-12-18T03:36:57.4825811Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl
2024-12-18T03:36:57.8958313Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.16.1)
2024-12-18T03:36:57.8960508Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (4.12.2)
2024-12-18T03:36:57.8962509Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2.8.8)
2024-12-18T03:36:57.8964777Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.1.4)
2024-12-18T03:36:57.8966938Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2024.10.0)
2024-12-18T03:36:57.8970926Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (1.13.1)
2024-12-18T03:36:57.8984946Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1) (1.3.0)
2024-12-18T03:36:57.9286854Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1) (3.0.2)
2024-12-18T03:36:58.2357111Z Installing collected packages: torch
2024-12-18T03:37:06.8515633Z Successfully installed torch-2.6.0a0+git0cdf8b1
2024-12-18T03:37:06.9104864Z + export TERM=vt100
2024-12-18T03:37:06.9105388Z + TERM=vt100
2024-12-18T03:37:06.9115060Z ++ dirname .ci/pytorch/test.sh
2024-12-18T03:37:06.9147023Z + source .ci/pytorch/common.sh
2024-12-18T03:37:06.9159408Z +++ dirname .ci/pytorch/common.sh
2024-12-18T03:37:06.9189914Z ++ source .ci/pytorch/common_utils.sh
2024-12-18T03:37:06.9191795Z +++ declare -f -t trap_add
2024-12-18T03:37:06.9203436Z ++ set -ex
2024-12-18T03:37:06.9204038Z ++ [[ linux-focal-rocm6.2-py3.10 == *rocm* ]]
2024-12-18T03:37:06.9204688Z ++ unset HIP_PLATFORM
2024-12-18T03:37:06.9205199Z ++ export PYTORCH_TEST_WITH_ROCM=1
2024-12-18T03:37:06.9205756Z ++ PYTORCH_TEST_WITH_ROCM=1
2024-12-18T03:37:06.9206276Z ++ export HSAKMT_DEBUG_LEVEL=4
2024-12-18T03:37:06.9206846Z ++ HSAKMT_DEBUG_LEVEL=4
2024-12-18T03:37:06.9207343Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1
2024-12-18T03:37:06.9207915Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1
2024-12-18T03:37:06.9208458Z ++ BUILD_TEST_LIBTORCH=0
2024-12-18T03:37:06.9209019Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]]
2024-12-18T03:37:06.9209632Z + echo 'Environment variables:'
2024-12-18T03:37:06.9210155Z Environment variables:
2024-12-18T03:37:06.9210601Z + env
2024-12-18T03:37:06.9240316Z INSTALLED_DB=yes
2024-12-18T03:37:06.9241268Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T03:37:06.9242205Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton
2024-12-18T03:37:06.9242832Z CONTINUE_THROUGH_ERROR=False
2024-12-18T03:37:06.9243418Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10
2024-12-18T03:37:06.9244027Z HOSTNAME=ce9b65b64934
2024-12-18T03:37:06.9245187Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9247125Z GITHUB_ACTION=__self
2024-12-18T03:37:06.9247669Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T03:37:06.9248326Z GITHUB_RUN_NUMBER=15427
2024-12-18T03:37:06.9248923Z TEST_CONFIG=distributed
2024-12-18T03:37:06.9249505Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T03:37:06.9250169Z AWS_DEFAULT_REGION=us-east-1
2024-12-18T03:37:06.9250697Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T03:37:06.9251222Z GITHUB_REF_TYPE=branch
2024-12-18T03:37:06.9256610Z ***
2024-12-18T03:37:06.9257085Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T03:37:06.9257642Z GITHUB_ACTIONS=true
2024-12-18T03:37:06.9258181Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9258873Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9259873Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/release/2.6
2024-12-18T03:37:06.9260787Z VERBOSE_TEST_LOGS=False
2024-12-18T03:37:06.9261282Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T03:37:06.9261807Z SHARD_NUMBER=3
2024-12-18T03:37:06.9262270Z GITHUB_REF_PROTECTED=true
2024-12-18T03:37:06.9262751Z HOME=/var/lib/jenkins
2024-12-18T03:37:06.9263282Z GITHUB_API_URL=https://api.github.com
2024-12-18T03:37:06.9263913Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T03:37:06.9264455Z LANG=C.UTF-8
2024-12-18T03:37:06.9264892Z PYTORCH_TEST_WITH_ROCM=1
2024-12-18T03:37:06.9265379Z NUM_TEST_SHARDS=3
2024-12-18T03:37:06.9266471Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9268443Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:37:06.9269610Z MAGMA_HOME=/opt/rocm/magma
2024-12-18T03:37:06.9270709Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9271934Z HSAKMT_DEBUG_LEVEL=4
2024-12-18T03:37:06.9272933Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T03:37:06.9273979Z GITHUB_EVENT_NAME=push
2024-12-18T03:37:06.9274455Z GITHUB_RUN_ID=12383255654
2024-12-18T03:37:06.9276117Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9277340Z GITHUB_ACTOR=malfet
2024-12-18T03:37:06.9277777Z PR_NUMBER=
2024-12-18T03:37:06.9278189Z GITHUB_RUN_ATTEMPT=1
2024-12-18T03:37:06.9278663Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T03:37:06.9279276Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T03:37:06.9279901Z TERM=vt100
2024-12-18T03:37:06.9280300Z INSTALLED_VISION=yes
2024-12-18T03:37:06.9280750Z BRANCH=release/2.6
2024-12-18T03:37:06.9281182Z TESTS_TO_INCLUDE=
2024-12-18T03:37:06.9282107Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2024-12-18T03:37:06.9283166Z GITHUB_SERVER_URL=https://github.com
2024-12-18T03:37:06.9283739Z PYTORCH_ROCM_ARCH=gfx90a
2024-12-18T03:37:06.9284220Z REENABLED_ISSUES=
2024-12-18T03:37:06.9284656Z SHLVL=1
2024-12-18T03:37:06.9285067Z MAX_JOBS=126
2024-12-18T03:37:06.9285497Z GITHUB_ACTOR_ID=2453524
2024-12-18T03:37:06.9286116Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9286827Z GITHUB_REF_NAME=release/2.6
2024-12-18T03:37:06.9287323Z ROCM_PATH=/opt/rocm
2024-12-18T03:37:06.9288062Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T03:37:06.9288976Z GITHUB_JOB=test
2024-12-18T03:37:06.9289480Z NO_TEST_TIMEOUT=False
2024-12-18T03:37:06.9289980Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T03:37:06.9290508Z LC_ALL=C.UTF-8
2024-12-18T03:37:06.9290939Z GITHUB_RETENTION_DAYS=90
2024-12-18T03:37:06.9291434Z GITHUB_ACTION_REPOSITORY=
2024-12-18T03:37:06.9293111Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T03:37:06.9295180Z GITHUB_BASE_REF=
2024-12-18T03:37:06.9295613Z CI=true
2024-12-18T03:37:06.9296019Z HSA_FORCE_FINE_GRAIN_PCIE=1
2024-12-18T03:37:06.9296561Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T03:37:06.9297073Z JOB_ID=34566687343
2024-12-18T03:37:06.9297515Z INSTALLED_PROTOBUF=yes
2024-12-18T03:37:06.9297974Z GITHUB_HEAD_REF=
2024-12-18T03:37:06.9298400Z GITHUB_ACTION_REF=
2024-12-18T03:37:06.9298934Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T03:37:06.9299577Z TEST_SHOWLOCALS=False
2024-12-18T03:37:06.9300054Z GITHUB_WORKFLOW=periodic
2024-12-18T03:37:06.9300556Z DEBIAN_FRONTEND=noninteractive
2024-12-18T03:37:06.9301713Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9302872Z NO_TD=False
2024-12-18T03:37:06.9303280Z OLDPWD=/var/lib/jenkins
2024-12-18T03:37:06.9303728Z _=/usr/bin/env
2024-12-18T03:37:06.9304324Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-12-18T03:37:06.9535841Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-12-18T03:37:06.9537054Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-12-18T03:37:06.9538101Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-12-18T03:37:06.9539118Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-12-18T03:37:06.9539907Z + BUILD_DIR=build
2024-12-18T03:37:06.9540387Z + BUILD_RENAMED_DIR=build_renamed
2024-12-18T03:37:06.9540945Z + BUILD_BIN_DIR=build/bin
2024-12-18T03:37:06.9541416Z + SHARD_NUMBER=3
2024-12-18T03:37:06.9541844Z + NUM_TEST_SHARDS=3
2024-12-18T03:37:06.9542300Z + export VALGRIND=ON
2024-12-18T03:37:06.9542752Z + VALGRIND=ON
2024-12-18T03:37:06.9543239Z + [[ linux-focal-rocm6.2-py3.10 == *clang9* ]]
2024-12-18T03:37:06.9543881Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]]
2024-12-18T03:37:06.9544451Z + [[ 0 == \1 ]]
2024-12-18T03:37:06.9544874Z + [[ False == \1 ]]
2024-12-18T03:37:06.9545374Z + [[ linux-focal-rocm6.2-py3.10 != *bazel* ]]
2024-12-18T03:37:06.9546057Z ++ realpath build/custom_test_artifacts
2024-12-18T03:37:06.9578001Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2024-12-18T03:37:06.9578951Z + [[ -n '' ]]
2024-12-18T03:37:06.9579414Z + echo 'Environment variables'
2024-12-18T03:37:06.9579952Z Environment variables
2024-12-18T03:37:06.9580399Z + env
2024-12-18T03:37:06.9605826Z INSTALLED_DB=yes
2024-12-18T03:37:06.9606755Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T03:37:06.9607668Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton
2024-12-18T03:37:06.9608303Z CONTINUE_THROUGH_ERROR=False
2024-12-18T03:37:06.9608895Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10
2024-12-18T03:37:06.9609511Z HOSTNAME=ce9b65b64934
2024-12-18T03:37:06.9610638Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9611827Z GITHUB_ACTION=__self
2024-12-18T03:37:06.9612335Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T03:37:06.9612891Z GITHUB_RUN_NUMBER=15427
2024-12-18T03:37:06.9613379Z TEST_CONFIG=distributed
2024-12-18T03:37:06.9613863Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T03:37:06.9614424Z AWS_DEFAULT_REGION=us-east-1
2024-12-18T03:37:06.9614951Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T03:37:06.9615461Z GITHUB_REF_TYPE=branch
2024-12-18T03:37:06.9616009Z ***
2024-12-18T03:37:06.9616439Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T03:37:06.9616965Z GITHUB_ACTIONS=true
2024-12-18T03:37:06.9617480Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9618161Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9619230Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/release/2.6
2024-12-18T03:37:06.9620141Z VERBOSE_TEST_LOGS=False
2024-12-18T03:37:06.9620628Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T03:37:06.9621141Z SHARD_NUMBER=3
2024-12-18T03:37:06.9622051Z GITHUB_REF_PROTECTED=true
2024-12-18T03:37:06.9622556Z HOME=/var/lib/jenkins
2024-12-18T03:37:06.9623066Z GITHUB_API_URL=https://api.github.com
2024-12-18T03:37:06.9623681Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T03:37:06.9624208Z LANG=C.UTF-8
2024-12-18T03:37:06.9624637Z PYTORCH_TEST_WITH_ROCM=1
2024-12-18T03:37:06.9625132Z NUM_TEST_SHARDS=3
2024-12-18T03:37:06.9626224Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9628060Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 3, 3, linux.rocm.gpu, module:rocm, oncall:distributed)
2024-12-18T03:37:06.9629076Z MAGMA_HOME=/opt/rocm/magma
2024-12-18T03:37:06.9630173Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9631294Z HSAKMT_DEBUG_LEVEL=4
2024-12-18T03:37:06.9632136Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T03:37:06.9633059Z GITHUB_EVENT_NAME=push
2024-12-18T03:37:06.9633531Z GITHUB_RUN_ID=12383255654
2024-12-18T03:37:06.9634713Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9635928Z GITHUB_ACTOR=malfet
2024-12-18T03:37:06.9636357Z PR_NUMBER=
2024-12-18T03:37:06.9636766Z GITHUB_RUN_ATTEMPT=1
2024-12-18T03:37:06.9637208Z VALGRIND=ON
2024-12-18T03:37:06.9637630Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T03:37:06.9638241Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T03:37:06.9638877Z TERM=vt100
2024-12-18T03:37:06.9639275Z INSTALLED_VISION=yes
2024-12-18T03:37:06.9639720Z BRANCH=release/2.6
2024-12-18T03:37:06.9640146Z TESTS_TO_INCLUDE=
2024-12-18T03:37:06.9641056Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2024-12-18T03:37:06.9642102Z GITHUB_SERVER_URL=https://github.com
2024-12-18T03:37:06.9642685Z PYTORCH_ROCM_ARCH=gfx90a
2024-12-18T03:37:06.9643156Z REENABLED_ISSUES=
2024-12-18T03:37:06.9643574Z SHLVL=1
2024-12-18T03:37:06.9644384Z MAX_JOBS=126
2024-12-18T03:37:06.9644806Z GITHUB_ACTOR_ID=2453524
2024-12-18T03:37:06.9645421Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T03:37:06.9646124Z GITHUB_REF_NAME=release/2.6
2024-12-18T03:37:06.9646610Z ROCM_PATH=/opt/rocm
2024-12-18T03:37:06.9647291Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T03:37:06.9648059Z GITHUB_JOB=test
2024-12-18T03:37:06.9648497Z NO_TEST_TIMEOUT=False
2024-12-18T03:37:06.9649094Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T03:37:06.9649736Z LC_ALL=C.UTF-8
2024-12-18T03:37:06.9650232Z GITHUB_RETENTION_DAYS=90
2024-12-18T03:37:06.9650726Z GITHUB_ACTION_REPOSITORY=
2024-12-18T03:37:06.9652405Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T03:37:06.9654091Z GITHUB_BASE_REF=
2024-12-18T03:37:06.9654513Z CI=true
2024-12-18T03:37:06.9654928Z HSA_FORCE_FINE_GRAIN_PCIE=1
2024-12-18T03:37:06.9655452Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T03:37:06.9655957Z JOB_ID=34566687343
2024-12-18T03:37:06.9656387Z INSTALLED_PROTOBUF=yes
2024-12-18T03:37:06.9656849Z GITHUB_HEAD_REF=
2024-12-18T03:37:06.9657270Z GITHUB_ACTION_REF=
2024-12-18T03:37:06.9657802Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T03:37:06.9658444Z TEST_SHOWLOCALS=False
2024-12-18T03:37:06.9658911Z GITHUB_WORKFLOW=periodic
2024-12-18T03:37:06.9659414Z DEBIAN_FRONTEND=noninteractive
2024-12-18T03:37:06.9660571Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_e641380b-bbf8-42e8-b803-cbd8e089f56e
2024-12-18T03:37:06.9661728Z NO_TD=False
2024-12-18T03:37:06.9662138Z OLDPWD=/var/lib/jenkins
2024-12-18T03:37:06.9662590Z _=/usr/bin/env
2024-12-18T03:37:06.9663016Z + echo 'Testing pytorch'
2024-12-18T03:37:06.9663815Z Testing pytorch
2024-12-18T03:37:06.9664261Z + export LANG=C.UTF-8
2024-12-18T03:37:06.9664704Z + LANG=C.UTF-8
2024-12-18T03:37:06.9665121Z + PR_NUMBER=
2024-12-18T03:37:06.9665555Z + [[ distributed == \d\e\f\a\u\l\t ]]
2024-12-18T03:37:06.9666134Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-12-18T03:37:06.9667111Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]]
2024-12-18T03:37:06.9667742Z + export HIP_VISIBLE_DEVICES=0,1
2024-12-18T03:37:06.9668308Z + HIP_VISIBLE_DEVICES=0,1
2024-12-18T03:37:06.9668889Z + [[ distributed == \s\l\o\w ]]
2024-12-18T03:37:06.9669597Z + [[ linux-focal-rocm6.2-py3.10 == *slow-gradcheck* ]]
2024-12-18T03:37:06.9670405Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]]
2024-12-18T03:37:06.9671012Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]]
2024-12-18T03:37:06.9671634Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T03:37:06.9672246Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T03:37:06.9672809Z + [[ distributed == *crossref* ]]
2024-12-18T03:37:06.9673375Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]]
2024-12-18T03:37:06.9673941Z + export VALGRIND=OFF
2024-12-18T03:37:06.9674394Z + VALGRIND=OFF
2024-12-18T03:37:06.9674794Z + rocminfo
2024-12-18T03:37:06.9874726Z ROCk module version 6.8.5 is loaded
2024-12-18T03:37:07.1229997Z =====================    
2024-12-18T03:37:07.1230673Z HSA System Attributes    
2024-12-18T03:37:07.1231238Z =====================    
2024-12-18T03:37:07.1231783Z Runtime Version:         1.14
2024-12-18T03:37:07.1232319Z Runtime Ext Version:     1.6
2024-12-18T03:37:07.1232860Z System Timestamp Freq.:  1000.000000MHz
2024-12-18T03:37:07.1233729Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2024-12-18T03:37:07.1234828Z Machine Model:           LARGE                              
2024-12-18T03:37:07.1235801Z System Endianness:       LITTLE                             
2024-12-18T03:37:07.1236565Z Mwaitx:                  DISABLED
2024-12-18T03:37:07.1237133Z DMAbuf Support:          YES
2024-12-18T03:37:07.1237450Z 
2024-12-18T03:37:07.1237637Z ==========               
2024-12-18T03:37:07.1238651Z HSA Agents               
2024-12-18T03:37:07.1239122Z ==========               
2024-12-18T03:37:07.1239559Z *******                  
2024-12-18T03:37:07.1240000Z Agent 1                  
2024-12-18T03:37:07.1240435Z *******                  
2024-12-18T03:37:07.1240983Z   Name:                    AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.1241697Z   Uuid:                    CPU-XX                             
2024-12-18T03:37:07.1242458Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.1243185Z   Vendor Name:             CPU                                
2024-12-18T03:37:07.1243889Z   Feature:                 None specified                     
2024-12-18T03:37:07.1244581Z   Profile:                 FULL_PROFILE                       
2024-12-18T03:37:07.1245289Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1246004Z   Max Queue Number:        0(0x0)                             
2024-12-18T03:37:07.1246705Z   Queue Min Size:          0(0x0)                             
2024-12-18T03:37:07.1247392Z   Queue Max Size:          0(0x0)                             
2024-12-18T03:37:07.1248083Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1248716Z   Node:                    0                                  
2024-12-18T03:37:07.1249399Z   Device Type:             CPU                                
2024-12-18T03:37:07.1250024Z   Cache Info:              
2024-12-18T03:37:07.1250557Z     L1:                      32768(0x8000) KB                   
2024-12-18T03:37:07.1251484Z   Chip ID:                 0(0x0)                             
2024-12-18T03:37:07.1252175Z   ASIC Revision:           0(0x0)                             
2024-12-18T03:37:07.1252876Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1253946Z   Max Clock Freq. (MHz):   2000                               
2024-12-18T03:37:07.1254638Z   BDFID:                   0                                  
2024-12-18T03:37:07.1255310Z   Internal Node ID:        0                                  
2024-12-18T03:37:07.1256023Z   Compute Unit:            64                                 
2024-12-18T03:37:07.1256701Z   SIMDs per CU:            0                                  
2024-12-18T03:37:07.1257397Z   Shader Engines:          0                                  
2024-12-18T03:37:07.1258112Z   Shader Arrs. per Eng.:   0                                  
2024-12-18T03:37:07.1258841Z   WatchPts on Addr. Ranges:1                                  
2024-12-18T03:37:07.1259496Z   Memory Properties:       
2024-12-18T03:37:07.1260001Z   Features:                None
2024-12-18T03:37:07.1260500Z   Pool Info:               
2024-12-18T03:37:07.1260976Z     Pool 1                   
2024-12-18T03:37:07.1261577Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1262302Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:37:07.1262996Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1263740Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1264512Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1265270Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1266001Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1266822Z     Pool 2                   
2024-12-18T03:37:07.1267426Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2024-12-18T03:37:07.1268131Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:37:07.1268800Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1269506Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1270253Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1271001Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1272093Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1272722Z     Pool 3                   
2024-12-18T03:37:07.1273296Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1273974Z       Size:                    528249788(0x1f7c73bc) KB           
2024-12-18T03:37:07.1274642Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1275339Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1276072Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1276805Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1277523Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1278162Z   ISA Info:                
2024-12-18T03:37:07.1278628Z *******                  
2024-12-18T03:37:07.1279083Z Agent 2                  
2024-12-18T03:37:07.1279520Z *******                  
2024-12-18T03:37:07.1280047Z   Name:                    AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.1280708Z   Uuid:                    CPU-XX                             
2024-12-18T03:37:07.1281408Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.1282240Z   Vendor Name:             CPU                                
2024-12-18T03:37:07.1283046Z   Feature:                 None specified                     
2024-12-18T03:37:07.1283750Z   Profile:                 FULL_PROFILE                       
2024-12-18T03:37:07.1284454Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1285161Z   Max Queue Number:        0(0x0)                             
2024-12-18T03:37:07.1285852Z   Queue Min Size:          0(0x0)                             
2024-12-18T03:37:07.1286874Z   Queue Max Size:          0(0x0)                             
2024-12-18T03:37:07.1287570Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1288212Z   Node:                    1                                  
2024-12-18T03:37:07.1288852Z   Device Type:             CPU                                
2024-12-18T03:37:07.1289461Z   Cache Info:              
2024-12-18T03:37:07.1289982Z     L1:                      32768(0x8000) KB                   
2024-12-18T03:37:07.1290600Z   Chip ID:                 0(0x0)                             
2024-12-18T03:37:07.1291259Z   ASIC Revision:           0(0x0)                             
2024-12-18T03:37:07.1291959Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1292673Z   Max Clock Freq. (MHz):   2000                               
2024-12-18T03:37:07.1293338Z   BDFID:                   0                                  
2024-12-18T03:37:07.1294012Z   Internal Node ID:        1                                  
2024-12-18T03:37:07.1294713Z   Compute Unit:            64                                 
2024-12-18T03:37:07.1295400Z   SIMDs per CU:            0                                  
2024-12-18T03:37:07.1296091Z   Shader Engines:          0                                  
2024-12-18T03:37:07.1296806Z   Shader Arrs. per Eng.:   0                                  
2024-12-18T03:37:07.1297542Z   WatchPts on Addr. Ranges:1                                  
2024-12-18T03:37:07.1298187Z   Memory Properties:       
2024-12-18T03:37:07.1298678Z   Features:                None
2024-12-18T03:37:07.1299170Z   Pool Info:               
2024-12-18T03:37:07.1299635Z     Pool 1                   
2024-12-18T03:37:07.1300217Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1300904Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:37:07.1301592Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1302298Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1303324Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1304067Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1304788Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1305416Z     Pool 2                   
2024-12-18T03:37:07.1305998Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2024-12-18T03:37:07.1306996Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:37:07.1307680Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1308438Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1309299Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1310187Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1310918Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1311555Z     Pool 3                   
2024-12-18T03:37:07.1312125Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1312794Z       Size:                    528402432(0x1f7ec800) KB           
2024-12-18T03:37:07.1313472Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1314172Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1314908Z       Alloc Recommended Granule:4KB                                
2024-12-18T03:37:07.1315657Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1316386Z       Accessible by all:       TRUE                               
2024-12-18T03:37:07.1317014Z   ISA Info:                
2024-12-18T03:37:07.1317478Z *******                  
2024-12-18T03:37:07.1317928Z Agent 3                  
2024-12-18T03:37:07.1318713Z *******                  
2024-12-18T03:37:07.1319233Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1319899Z   Uuid:                    GPU-72eb622bef58f205               
2024-12-18T03:37:07.1320600Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1321324Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1322081Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1322929Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1323791Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1324583Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1325281Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1325976Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1326670Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1327326Z   Node:                    2                                  
2024-12-18T03:37:07.1327990Z   Device Type:             GPU                                
2024-12-18T03:37:07.1328594Z   Cache Info:              
2024-12-18T03:37:07.1329102Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1329707Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1330352Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1331020Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1331727Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1332436Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1333106Z   BDFID:                   12800                              
2024-12-18T03:37:07.1333789Z   Internal Node ID:        2                                  
2024-12-18T03:37:07.1334492Z   Compute Unit:            104                                
2024-12-18T03:37:07.1335172Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1336202Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1336915Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1337643Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1338380Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1339033Z   Memory Properties:       
2024-12-18T03:37:07.1339562Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1340230Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1340950Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1341663Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1342325Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1342908Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1343496Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1344073Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1344713Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1345430Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1346139Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1346888Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1347426Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1348026Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1348707Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1349515Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1350854Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1351630Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1352363Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1352992Z   Pool Info:               
2024-12-18T03:37:07.1353482Z     Pool 1                   
2024-12-18T03:37:07.1354079Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1354784Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1355472Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1356187Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1356936Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1357697Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1358432Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1359074Z     Pool 2                   
2024-12-18T03:37:07.1359661Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1360371Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1361100Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1361855Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1362752Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1363614Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1364345Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1364980Z     Pool 3                   
2024-12-18T03:37:07.1365573Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1366251Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1366941Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1367651Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1368743Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1369497Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1370225Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1370853Z     Pool 4                   
2024-12-18T03:37:07.1371419Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1372082Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1372754Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1373477Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1374227Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1374988Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1375719Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1376367Z   ISA Info:                
2024-12-18T03:37:07.1376839Z     ISA 1                    
2024-12-18T03:37:07.1377448Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1378234Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1378995Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1379732Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1380493Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1381203Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1381912Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1382586Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1383467Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1384085Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1384695Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1385350Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1385995Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1386551Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1387261Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1387852Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1388605Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1389337Z *******                  
2024-12-18T03:37:07.1389865Z Agent 4                  
2024-12-18T03:37:07.1390377Z *******                  
2024-12-18T03:37:07.1390888Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1391552Z   Uuid:                    GPU-2a0632e13f52b1f3               
2024-12-18T03:37:07.1392267Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1392992Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1393700Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1394397Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1395096Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1395803Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1396495Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1397184Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1397868Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1398522Z   Node:                    3                                  
2024-12-18T03:37:07.1399175Z   Device Type:             GPU                                
2024-12-18T03:37:07.1400131Z   Cache Info:              
2024-12-18T03:37:07.1400646Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1401249Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1401877Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1402673Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1403495Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1404202Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1404876Z   BDFID:                   13568                              
2024-12-18T03:37:07.1405548Z   Internal Node ID:        3                                  
2024-12-18T03:37:07.1406245Z   Compute Unit:            104                                
2024-12-18T03:37:07.1406922Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1407603Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1408324Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1409062Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1409799Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1410458Z   Memory Properties:       
2024-12-18T03:37:07.1410981Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1411642Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1412366Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1413086Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1413760Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1414323Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1415227Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1415831Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1416469Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1417176Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1417883Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1418516Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1419040Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1419642Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1420234Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1420939Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1421716Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1422478Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1423194Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1423830Z   Pool Info:               
2024-12-18T03:37:07.1424315Z     Pool 1                   
2024-12-18T03:37:07.1424912Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1425621Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1426310Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1427307Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1428063Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1428821Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1429554Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1430186Z     Pool 2                   
2024-12-18T03:37:07.1430795Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1431490Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1432529Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1433254Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1434001Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1434757Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1435487Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1436128Z     Pool 3                   
2024-12-18T03:37:07.1436707Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1437383Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1438058Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1438789Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1439534Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1440298Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1441034Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1441673Z     Pool 4                   
2024-12-18T03:37:07.1442228Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1442883Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1443562Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1444281Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1445027Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1445770Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1446826Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1447478Z   ISA Info:                
2024-12-18T03:37:07.1447957Z     ISA 1                    
2024-12-18T03:37:07.1448554Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1449334Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1450079Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1450823Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1451581Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1452290Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1452990Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1453655Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1454241Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1454958Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1455666Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1456436Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1457104Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1457654Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1458259Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1458856Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1459523Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1460151Z *******                  
2024-12-18T03:37:07.1460612Z Agent 5                  
2024-12-18T03:37:07.1461058Z *******                  
2024-12-18T03:37:07.1461565Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1462237Z   Uuid:                    GPU-811789ff388472b0               
2024-12-18T03:37:07.1462946Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1464011Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1464710Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1465402Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1466101Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1466931Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1467624Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1468311Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1468986Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1469626Z   Node:                    4                                  
2024-12-18T03:37:07.1470286Z   Device Type:             GPU                                
2024-12-18T03:37:07.1470896Z   Cache Info:              
2024-12-18T03:37:07.1471414Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1472024Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1472633Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1473293Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1473993Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1474693Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1475346Z   BDFID:                   4352                               
2024-12-18T03:37:07.1476008Z   Internal Node ID:        4                                  
2024-12-18T03:37:07.1476697Z   Compute Unit:            104                                
2024-12-18T03:37:07.1477696Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1478390Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1479102Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1479822Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1480554Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1481199Z   Memory Properties:       
2024-12-18T03:37:07.1481717Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1482439Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1483286Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1484041Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1484696Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1485257Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1485846Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1486413Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1487052Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1487756Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1488461Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1489088Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1489602Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1490188Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1490776Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1491451Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1492222Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1492953Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1493673Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1494300Z   Pool Info:               
2024-12-18T03:37:07.1495109Z     Pool 1                   
2024-12-18T03:37:07.1495694Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1496397Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1497078Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1497789Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1498530Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1499280Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1500083Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1500713Z     Pool 2                   
2024-12-18T03:37:07.1501292Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1501982Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1502661Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1503361Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1504091Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1504831Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1505550Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1506166Z     Pool 3                   
2024-12-18T03:37:07.1506825Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1507496Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1508163Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1508869Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1509926Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1510688Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1511422Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1512047Z     Pool 4                   
2024-12-18T03:37:07.1512592Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1513241Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1513907Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1514611Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1515344Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1516079Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1516832Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1517484Z   ISA Info:                
2024-12-18T03:37:07.1517954Z     ISA 1                    
2024-12-18T03:37:07.1518562Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1519464Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1520343Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1521172Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1521923Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1522625Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1523326Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1523996Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1524586Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1525208Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1525803Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1526802Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1527452Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1528005Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1528616Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1529217Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1529905Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1530529Z *******                  
2024-12-18T03:37:07.1530988Z Agent 6                  
2024-12-18T03:37:07.1531430Z *******                  
2024-12-18T03:37:07.1531927Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1532583Z   Uuid:                    GPU-271d47bdd095d3f3               
2024-12-18T03:37:07.1533300Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1534019Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1534720Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1535414Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1536114Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1536827Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1537520Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1538209Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1538885Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1539531Z   Node:                    5                                  
2024-12-18T03:37:07.1540183Z   Device Type:             GPU                                
2024-12-18T03:37:07.1541074Z   Cache Info:              
2024-12-18T03:37:07.1541593Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1542241Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1542974Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1543756Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1544457Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1545162Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1545828Z   BDFID:                   5120                               
2024-12-18T03:37:07.1546499Z   Internal Node ID:        5                                  
2024-12-18T03:37:07.1547554Z   Compute Unit:            104                                
2024-12-18T03:37:07.1548283Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1549123Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1549975Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1550795Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1551539Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1552191Z   Memory Properties:       
2024-12-18T03:37:07.1552725Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1553388Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1554111Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1554832Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1555494Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1556070Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1556637Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1557162Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1557734Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1558700Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1559334Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1559896Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1560378Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1560924Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1561451Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1562065Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1562875Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1563645Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1564306Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1564871Z   Pool Info:               
2024-12-18T03:37:07.1565297Z     Pool 1                   
2024-12-18T03:37:07.1565833Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1566467Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1567079Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1567709Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1568373Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1569060Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1569715Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1570284Z     Pool 2                   
2024-12-18T03:37:07.1570803Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1571712Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1572326Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1572965Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1573622Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1574282Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1574927Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1575492Z     Pool 3                   
2024-12-18T03:37:07.1575998Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1576595Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1577198Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1577821Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1578489Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1579163Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1579827Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1580387Z     Pool 4                   
2024-12-18T03:37:07.1580865Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1581439Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1582026Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1582651Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1583304Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1583962Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1584603Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1585167Z   ISA Info:                
2024-12-18T03:37:07.1585585Z     ISA 1                    
2024-12-18T03:37:07.1586105Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1587198Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1587852Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1588589Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1589370Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1590096Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1590763Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1591364Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1591890Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1592424Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1592949Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1593524Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1594100Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1594589Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1595119Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1595640Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1596233Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1596783Z *******                  
2024-12-18T03:37:07.1597187Z Agent 7                  
2024-12-18T03:37:07.1597571Z *******                  
2024-12-18T03:37:07.1598020Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1598617Z   Uuid:                    GPU-ffdd23a7d4aa1894               
2024-12-18T03:37:07.1599539Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1600194Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1600820Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1601437Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1602061Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1602794Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1603513Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1604170Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1604785Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1605357Z   Node:                    6                                  
2024-12-18T03:37:07.1605940Z   Device Type:             GPU                                
2024-12-18T03:37:07.1606485Z   Cache Info:              
2024-12-18T03:37:07.1606945Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1607471Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1608029Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1608630Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1609255Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1609880Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1610472Z   BDFID:                   44544                              
2024-12-18T03:37:07.1611064Z   Internal Node ID:        6                                  
2024-12-18T03:37:07.1611668Z   Compute Unit:            104                                
2024-12-18T03:37:07.1612265Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1612871Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1613503Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1614142Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1615124Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1615698Z   Memory Properties:       
2024-12-18T03:37:07.1616155Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1616723Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1617354Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1617977Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1618552Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1619048Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1619563Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1620071Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1620646Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1621282Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1621922Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1622481Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1622933Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1623454Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1623972Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1624579Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1625262Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1625938Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1626588Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1627231Z   Pool Info:               
2024-12-18T03:37:07.1628000Z     Pool 1                   
2024-12-18T03:37:07.1628532Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1629169Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1629782Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1630415Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1631076Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1631750Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1632406Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1632967Z     Pool 2                   
2024-12-18T03:37:07.1633486Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1634098Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1634700Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1635329Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1635988Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1636637Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1637279Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1637834Z     Pool 3                   
2024-12-18T03:37:07.1638339Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1638949Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1639547Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1640173Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1640824Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1641490Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1642128Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1642996Z     Pool 4                   
2024-12-18T03:37:07.1643481Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1644052Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1644645Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1645267Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1645916Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1646576Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1647216Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1647773Z   ISA Info:                
2024-12-18T03:37:07.1648184Z     ISA 1                    
2024-12-18T03:37:07.1648716Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1649402Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1650068Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1650717Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1651391Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1652004Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1652627Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1653223Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1653755Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1654290Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1654809Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1655638Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1656224Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1656712Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1657242Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1657775Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1658369Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1658917Z *******                  
2024-12-18T03:37:07.1659314Z Agent 8                  
2024-12-18T03:37:07.1659705Z *******                  
2024-12-18T03:37:07.1660147Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1660718Z   Uuid:                    GPU-13cb7b90220db844               
2024-12-18T03:37:07.1661339Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1661981Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1662589Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1663197Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1663814Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1664430Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1665040Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1665644Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1666233Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1667039Z   Node:                    7                                  
2024-12-18T03:37:07.1667609Z   Device Type:             GPU                                
2024-12-18T03:37:07.1668149Z   Cache Info:              
2024-12-18T03:37:07.1668674Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1669307Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1669941Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1670918Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1671535Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1672158Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1672741Z   BDFID:                   45824                              
2024-12-18T03:37:07.1673341Z   Internal Node ID:        7                                  
2024-12-18T03:37:07.1673949Z   Compute Unit:            104                                
2024-12-18T03:37:07.1674541Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1675150Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1675783Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1676435Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1677086Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1677667Z   Memory Properties:       
2024-12-18T03:37:07.1678123Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1678693Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1679319Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1679947Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1680520Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1681021Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1681543Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1682049Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1682671Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1683801Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1684496Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1685089Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1685551Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1686087Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1686615Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1687213Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1687893Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1688558Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1689191Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1689739Z   Pool Info:               
2024-12-18T03:37:07.1690150Z     Pool 1                   
2024-12-18T03:37:07.1690684Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1691306Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1691920Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1692551Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1693208Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1693877Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1694521Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1695081Z     Pool 2                   
2024-12-18T03:37:07.1695604Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1696213Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1696811Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1697446Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1698103Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1699032Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1699671Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1700222Z     Pool 3                   
2024-12-18T03:37:07.1700727Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1701324Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1701924Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1702546Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1703197Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1703859Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1704513Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1705064Z     Pool 4                   
2024-12-18T03:37:07.1705551Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1706117Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1706797Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1707435Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1708082Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1708744Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1709389Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1709944Z   ISA Info:                
2024-12-18T03:37:07.1710343Z     ISA 1                    
2024-12-18T03:37:07.1710863Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1711828Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1712491Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1713144Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1713806Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1714416Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1715027Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1715613Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1716127Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1716654Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1717175Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1717746Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1718324Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1718803Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1719337Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1719862Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1720450Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1720996Z *******                  
2024-12-18T03:37:07.1721393Z Agent 9                  
2024-12-18T03:37:07.1721777Z *******                  
2024-12-18T03:37:07.1722220Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1722798Z   Uuid:                    GPU-89e041d1b75d6f09               
2024-12-18T03:37:07.1723416Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1724048Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1724677Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1725283Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1726205Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1726831Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1727444Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1728057Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1728668Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1729235Z   Node:                    8                                  
2024-12-18T03:37:07.1729803Z   Device Type:             GPU                                
2024-12-18T03:37:07.1730337Z   Cache Info:              
2024-12-18T03:37:07.1730787Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1731321Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1731885Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1732486Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1733109Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1733739Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1734326Z   BDFID:                   36352                              
2024-12-18T03:37:07.1734925Z   Internal Node ID:        8                                  
2024-12-18T03:37:07.1735548Z   Compute Unit:            104                                
2024-12-18T03:37:07.1736154Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1736760Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1737387Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1738030Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1738919Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1739513Z   Memory Properties:       
2024-12-18T03:37:07.1739981Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1740560Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1741193Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1741837Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1742430Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1742927Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1743451Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1743963Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1744524Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1745152Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1745788Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1746353Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1746899Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1747425Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1747944Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1748547Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1749234Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1749887Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1750513Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1751054Z   Pool Info:               
2024-12-18T03:37:07.1751475Z     Pool 1                   
2024-12-18T03:37:07.1752071Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1752708Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1753312Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1754250Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1754915Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1755582Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1756223Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1756781Z     Pool 2                   
2024-12-18T03:37:07.1757297Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1757909Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1758504Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1759127Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1759789Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1760444Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1761094Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1761655Z     Pool 3                   
2024-12-18T03:37:07.1762159Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1762748Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1763344Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1763975Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1764624Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1765279Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1765917Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1766747Z     Pool 4                   
2024-12-18T03:37:07.1767254Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1767842Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1768435Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1769063Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1769706Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1770359Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1770998Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1771562Z   ISA Info:                
2024-12-18T03:37:07.1771968Z     ISA 1                    
2024-12-18T03:37:07.1772486Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1773161Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1773822Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1774479Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1775146Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1775767Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1776389Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1776991Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1777514Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1778053Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1778575Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1779147Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1779717Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1780218Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1780756Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1781550Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1782154Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1782701Z *******                  
2024-12-18T03:37:07.1783095Z Agent 10                 
2024-12-18T03:37:07.1783479Z *******                  
2024-12-18T03:37:07.1783922Z   Name:                    gfx90a                             
2024-12-18T03:37:07.1784522Z   Uuid:                    GPU-a911e24894f88cb2               
2024-12-18T03:37:07.1785153Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.1785802Z   Vendor Name:             AMD                                
2024-12-18T03:37:07.1786410Z   Feature:                 KERNEL_DISPATCH                    
2024-12-18T03:37:07.1787239Z   Profile:                 BASE_PROFILE                       
2024-12-18T03:37:07.1787864Z   Float Round Mode:        NEAR                               
2024-12-18T03:37:07.1788494Z   Max Queue Number:        128(0x80)                          
2024-12-18T03:37:07.1789113Z   Queue Min Size:          64(0x40)                           
2024-12-18T03:37:07.1789723Z   Queue Max Size:          131072(0x20000)                    
2024-12-18T03:37:07.1790331Z   Queue Type:              MULTI                              
2024-12-18T03:37:07.1790898Z   Node:                    9                                  
2024-12-18T03:37:07.1791477Z   Device Type:             GPU                                
2024-12-18T03:37:07.1792014Z   Cache Info:              
2024-12-18T03:37:07.1792457Z     L1:                      16(0x10) KB                        
2024-12-18T03:37:07.1792982Z     L2:                      8192(0x2000) KB                    
2024-12-18T03:37:07.1793865Z   Chip ID:                 29708(0x740c)                      
2024-12-18T03:37:07.1794474Z   ASIC Revision:           1(0x1)                             
2024-12-18T03:37:07.1795108Z   Cacheline Size:          64(0x40)                           
2024-12-18T03:37:07.1795738Z   Max Clock Freq. (MHz):   1700                               
2024-12-18T03:37:07.1796334Z   BDFID:                   37632                              
2024-12-18T03:37:07.1796922Z   Internal Node ID:        9                                  
2024-12-18T03:37:07.1797538Z   Compute Unit:            104                                
2024-12-18T03:37:07.1798133Z   SIMDs per CU:            4                                  
2024-12-18T03:37:07.1798741Z   Shader Engines:          8                                  
2024-12-18T03:37:07.1799377Z   Shader Arrs. per Eng.:   1                                  
2024-12-18T03:37:07.1800027Z   WatchPts on Addr. Ranges:4                                  
2024-12-18T03:37:07.1800706Z   Coherent Host Access:    FALSE                              
2024-12-18T03:37:07.1801297Z   Memory Properties:       
2024-12-18T03:37:07.1801757Z   Features:                KERNEL_DISPATCH 
2024-12-18T03:37:07.1802336Z   Fast F16 Operation:      TRUE                               
2024-12-18T03:37:07.1802977Z   Wavefront Size:          64(0x40)                           
2024-12-18T03:37:07.1803607Z   Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1804194Z   Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1804685Z     x                        1024(0x400)                        
2024-12-18T03:37:07.1805202Z     y                        1024(0x400)                        
2024-12-18T03:37:07.1805710Z     z                        1024(0x400)                        
2024-12-18T03:37:07.1806275Z   Max Waves Per CU:        32(0x20)                           
2024-12-18T03:37:07.1806903Z   Max Work-item Per CU:    2048(0x800)                        
2024-12-18T03:37:07.1807531Z   Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1808086Z   Grid Max Size per Dimension:
2024-12-18T03:37:07.1808545Z     x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1809367Z     y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1809886Z     z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1810482Z   Max fbarriers/Workgrp:   32                                 
2024-12-18T03:37:07.1811155Z   Packet Processor uCode:: 83                                 
2024-12-18T03:37:07.1811809Z   SDMA engine uCode::      8                                  
2024-12-18T03:37:07.1812437Z   IOMMU Support::          None                               
2024-12-18T03:37:07.1812987Z   Pool Info:               
2024-12-18T03:37:07.1813400Z     Pool 1                   
2024-12-18T03:37:07.1813918Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2024-12-18T03:37:07.1814540Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1815141Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1815778Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1816434Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1817095Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1817741Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1818305Z     Pool 2                   
2024-12-18T03:37:07.1818822Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2024-12-18T03:37:07.1819425Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1820016Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1820644Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1821531Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1822202Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1822849Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1823403Z     Pool 3                   
2024-12-18T03:37:07.1823898Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2024-12-18T03:37:07.1824492Z       Size:                    67092480(0x3ffc000) KB             
2024-12-18T03:37:07.1825077Z       Allocatable:             TRUE                               
2024-12-18T03:37:07.1825697Z       Alloc Granule:           4KB                                
2024-12-18T03:37:07.1826347Z       Alloc Recommended Granule:2048KB                             
2024-12-18T03:37:07.1827109Z       Alloc Alignment:         4KB                                
2024-12-18T03:37:07.1827742Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1828290Z     Pool 4                   
2024-12-18T03:37:07.1828777Z       Segment:                 GROUP                              
2024-12-18T03:37:07.1829346Z       Size:                    64(0x40) KB                        
2024-12-18T03:37:07.1829940Z       Allocatable:             FALSE                              
2024-12-18T03:37:07.1830559Z       Alloc Granule:           0KB                                
2024-12-18T03:37:07.1831212Z       Alloc Recommended Granule:0KB                                
2024-12-18T03:37:07.1831862Z       Alloc Alignment:         0KB                                
2024-12-18T03:37:07.1832501Z       Accessible by all:       FALSE                              
2024-12-18T03:37:07.1833053Z   ISA Info:                
2024-12-18T03:37:07.1833459Z     ISA 1                    
2024-12-18T03:37:07.1833977Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2024-12-18T03:37:07.1834656Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2024-12-18T03:37:07.1835324Z       Profiles:                HSA_PROFILE_BASE                   
2024-12-18T03:37:07.1835978Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1836959Z       Default Rounding Mode:   NEAR                               
2024-12-18T03:37:07.1837574Z       Fast f16:                TRUE                               
2024-12-18T03:37:07.1838183Z       Workgroup Max Size:      1024(0x400)                        
2024-12-18T03:37:07.1838770Z       Workgroup Max Size per Dimension:
2024-12-18T03:37:07.1839284Z         x                        1024(0x400)                        
2024-12-18T03:37:07.1839819Z         y                        1024(0x400)                        
2024-12-18T03:37:07.1840334Z         z                        1024(0x400)                        
2024-12-18T03:37:07.1840898Z       Grid Max Size:           4294967295(0xffffffff)             
2024-12-18T03:37:07.1841463Z       Grid Max Size per Dimension:
2024-12-18T03:37:07.1841950Z         x                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1842481Z         y                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1843012Z         z                        4294967295(0xffffffff)             
2024-12-18T03:37:07.1843593Z       FBarrier Max Size:       32                                 
2024-12-18T03:37:07.1844141Z *** Done ***             
2024-12-18T03:37:07.1844533Z + rocminfo
2024-12-18T03:37:07.1844904Z + grep -E 'Name:.*\sgfx|Marketing'
2024-12-18T03:37:07.2843952Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.2844862Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2024-12-18T03:37:07.2845631Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2846342Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2847037Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2847710Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2848917Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2849621Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2850315Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2850992Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2851663Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2852328Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2852993Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2853660Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2854328Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2854998Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.2855668Z   Name:                    gfx90a                             
2024-12-18T03:37:07.2856341Z   Marketing Name:          AMD Instinct MI250X/MI250          
2024-12-18T03:37:07.3012829Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]]
2024-12-18T03:37:07.3013651Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]]
2024-12-18T03:37:07.3014322Z + pip_install --user ninja==1.10.2
2024-12-18T03:37:07.3015058Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T03:37:07.3015940Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2024-12-18T03:37:07.8623569Z Collecting ninja==1.10.2
2024-12-18T03:37:07.9685010Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-12-18T03:37:07.9999172Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-12-18T03:37:08.3470925Z Installing collected packages: ninja
2024-12-18T03:37:08.3542666Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-12-18T03:37:08.3544434Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-12-18T03:37:08.3563385Z Successfully installed ninja-1.10.2
2024-12-18T03:37:08.4188032Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T03:37:08.4191515Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T03:37:08.4193492Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]]
2024-12-18T03:37:08.4194096Z + install_tlparse
2024-12-18T03:37:08.4194588Z + pip_install --user tlparse==0.3.25
2024-12-18T03:37:08.4195334Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T03:37:08.4196244Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25
2024-12-18T03:37:08.9674316Z Collecting tlparse==0.3.25
2024-12-18T03:37:09.0431791Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-12-18T03:37:09.0738040Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-12-18T03:37:09.5129413Z Installing collected packages: tlparse
2024-12-18T03:37:09.5427539Z Successfully installed tlparse-0.3.25
2024-12-18T03:37:09.6097921Z ++ python -m site --user-base
2024-12-18T03:37:09.6602294Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T03:37:09.6604572Z + [[ linux-focal-rocm6.2-py3.10 == *asan* ]]
2024-12-18T03:37:09.6605251Z + [[ linux-focal-rocm6.2-py3.10 == *-debug* ]]
2024-12-18T03:37:09.6606440Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]]
2024-12-18T03:37:09.6607383Z + echo 'We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass'
2024-12-18T03:37:09.6608546Z We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass
2024-12-18T03:37:09.6614701Z + cd test
2024-12-18T03:37:09.6615466Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-12-18T03:37:12.7526208Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-12-18T03:37:12.7527052Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-12-18T03:37:12.7527722Z + DYNAMO_BENCHMARK_FLAGS=()
2024-12-18T03:37:12.7528354Z + [[ distributed == *pr_time_benchmarks* ]]
2024-12-18T03:37:12.7529017Z + [[ distributed == *dynamo_eager* ]]
2024-12-18T03:37:12.7529636Z + [[ distributed == *aot_eager* ]]
2024-12-18T03:37:12.7530216Z + [[ distributed == *aot_inductor* ]]
2024-12-18T03:37:12.7530819Z + [[ distributed == *inductor* ]]
2024-12-18T03:37:12.7531637Z + [[ distributed == *dynamic* ]]
2024-12-18T03:37:12.7532207Z + [[ distributed == *cpu* ]]
2024-12-18T03:37:12.7532830Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-12-18T03:37:12.7566909Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]]
2024-12-18T03:37:12.7567748Z + [[ linux-focal-rocm6.2-py3.10 == *-bazel-* ]]
2024-12-18T03:37:12.7574364Z + cd test
2024-12-18T03:37:12.7575002Z + python -c 'import torch; print(torch.__config__.show())'
2024-12-18T03:37:15.2654324Z PyTorch built with:
2024-12-18T03:37:15.2654955Z   - GCC 9.4
2024-12-18T03:37:15.2655392Z   - C++ Version: 201703
2024-12-18T03:37:15.2656423Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T03:37:15.2657741Z   - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T03:37:15.2658555Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T03:37:15.2659184Z   - LAPACK is enabled (usually provided by MKL)
2024-12-18T03:37:15.2659846Z   - NNPACK is enabled
2024-12-18T03:37:15.2660336Z   - CPU capability usage: AVX2
2024-12-18T03:37:15.2660862Z   - HIP Runtime 6.2.41134
2024-12-18T03:37:15.2662166Z   - MIOpen 3.2.0
2024-12-18T03:37:15.2662582Z   - Magma 2.7.2
2024-12-18T03:37:15.2671035Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-12-18T03:37:15.2679564Z 
2024-12-18T03:37:15.7763761Z + cd test
2024-12-18T03:37:15.7764497Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-12-18T03:37:17.0521614Z ATen/Parallel:
2024-12-18T03:37:17.0522237Z 	at::get_num_threads() : 128
2024-12-18T03:37:17.0522806Z 	at::get_num_interop_threads() : 128
2024-12-18T03:37:17.0523396Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T03:37:17.0523934Z 	omp_get_max_threads() : 128
2024-12-18T03:37:17.0525714Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T03:37:17.0526796Z 	mkl_get_max_threads() : 128
2024-12-18T03:37:17.0527518Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T03:37:17.0528355Z std::thread::hardware_concurrency() : 128
2024-12-18T03:37:17.0528931Z Environment variables:
2024-12-18T03:37:17.0529398Z 	OMP_NUM_THREADS : [not set]
2024-12-18T03:37:17.0529880Z 	MKL_NUM_THREADS : [not set]
2024-12-18T03:37:17.0530383Z ATen parallel backend: OpenMP
2024-12-18T03:37:17.0530725Z 
2024-12-18T03:37:18.8288373Z + [[ distributed == *numpy_2* ]]
2024-12-18T03:37:18.8289120Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]]
2024-12-18T03:37:18.8289804Z + [[ distributed == *backward* ]]
2024-12-18T03:37:18.8290354Z + [[ distributed == *xla* ]]
2024-12-18T03:37:18.8290876Z + [[ distributed == *executorch* ]]
2024-12-18T03:37:18.8291454Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2024-12-18T03:37:18.8292088Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]]
2024-12-18T03:37:18.8292742Z + [[ distributed == distributed ]]
2024-12-18T03:37:18.8293268Z + test_distributed
2024-12-18T03:37:18.8293754Z + echo 'Testing distributed python tests'
2024-12-18T03:37:18.8294379Z Testing distributed python tests
2024-12-18T03:37:18.8295126Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose
2024-12-18T03:37:18.9465601Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
2024-12-18T03:37:18.9467281Z   import pkg_resources
2024-12-18T03:37:23.4093314Z Excluding distributed/rpc/test_faulty_agent on ROCm
2024-12-18T03:37:23.4094253Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm
2024-12-18T03:37:23.4095027Z Excluding distributed/rpc/test_share_memory on ROCm
2024-12-18T03:37:23.4095811Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm
2024-12-18T03:37:23.4096743Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm
2024-12-18T03:37:23.4097783Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm
2024-12-18T03:37:23.4098800Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm
2024-12-18T03:37:23.4100564Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm
2024-12-18T03:37:23.4101487Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm
2024-12-18T03:37:23.4102330Z Excluding distributed/_tensor/test_attention on ROCm
2024-12-18T03:37:24.3942881Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2024-12-18T03:37:24.6323578Z Ignoring disabled issues:  ['']
2024-12-18T03:37:24.6618551Z Found test times from artifacts
2024-12-18T03:37:24.7245208Z Found test times from artifacts
2024-12-18T03:37:24.7259835Z Running all tests
2024-12-18T03:37:24.7342479Z Running parallel tests on 8 processes
2024-12-18T03:37:24.7345706Z Name: tests to run (est. time: 142.49min)
2024-12-18T03:37:24.7346444Z   Serial tests (82):
2024-12-18T03:37:24.7347143Z     distributed/test_inductor_collectives 1/1
2024-12-18T03:37:24.7347872Z     distributed/_tensor/test_common_rules 1/1
2024-12-18T03:37:24.7348510Z     distributed/_tensor/test_init 1/1
2024-12-18T03:37:24.7349161Z     distributed/_tensor/test_experimental_ops 1/1
2024-12-18T03:37:24.7349829Z     distributed/_tensor/test_tensor_ops 1/1
2024-12-18T03:37:24.7350507Z     distributed/fsdp/test_fsdp_clip_grad_norm 1/1
2024-12-18T03:37:24.7351133Z     distributed/test_c10d_common 1/1
2024-12-18T03:37:24.7351785Z     distributed/flight_recorder/test_fr_analysis 1/1
2024-12-18T03:37:24.7352502Z     distributed/_tensor/test_dtensor_compile 1/1
2024-12-18T03:37:24.7353171Z     distributed/checkpoint/test_nested_dict 1/1
2024-12-18T03:37:24.7353829Z     distributed/pipelining/test_transformer 1/1
2024-12-18T03:37:24.7354483Z     distributed/pipelining/test_backward 1/1
2024-12-18T03:37:24.7360425Z     distributed/_shard/sharded_tensor/test_logger 1/1
2024-12-18T03:37:24.7361153Z     distributed/pipelining/test_pipe 1/1
2024-12-18T03:37:24.7361877Z     distributed/_composable/fsdp/test_fully_shard_compile 1/1
2024-12-18T03:37:24.7362657Z     distributed/checkpoint/test_save_load_api 1/1
2024-12-18T03:37:24.7363359Z     distributed/checkpoint/test_dedup_tensors 1/1
2024-12-18T03:37:24.7364028Z     distributed/_composable/test_contract 1/1
2024-12-18T03:37:24.7364685Z     distributed/pipelining/test_microbatch 1/1
2024-12-18T03:37:24.7365463Z     distributed/checkpoint/e2e/test_fsdp_ep 1/1
2024-12-18T03:37:24.7366352Z     distributed/_tensor/experimental/test_register_sharding 1/1
2024-12-18T03:37:24.7367128Z     distributed/_tools/test_mem_tracker 1/1
2024-12-18T03:37:24.7367729Z     distributed/fsdp/test_fsdp_fx 1/1
2024-12-18T03:37:24.7368305Z     distributed/_tensor/test_api 1/1
2024-12-18T03:37:24.7368902Z     distributed/pipelining/test_unflatten 1/1
2024-12-18T03:37:24.7369547Z     distributed/checkpoint/fsdp/test_fsdp_dsd 1/1
2024-12-18T03:37:24.7370306Z     distributed/_composable/fsdp/test_fully_shard_logging 1/1
2024-12-18T03:37:24.7371074Z     distributed/pipelining/test_stage 1/1
2024-12-18T03:37:24.7371698Z     distributed/_tools/test_memory_tracker 1/1
2024-12-18T03:37:24.7372507Z     distributed/_composable/test_composability/test_pp_composability 1/1
2024-12-18T03:37:24.7373324Z     distributed/test_distributed_spawn 2/12
2024-12-18T03:37:24.7373938Z     distributed/test_distributed_spawn 5/12
2024-12-18T03:37:24.7374543Z     distributed/test_distributed_spawn 8/12
2024-12-18T03:37:24.7375153Z     distributed/test_distributed_spawn 11/12
2024-12-18T03:37:24.7375774Z     distributed/checkpoint/test_fsspec 1/1
2024-12-18T03:37:24.7376418Z     distributed/tensor/parallel/test_tp_style 1/1
2024-12-18T03:37:24.7377136Z     distributed/_tensor/debug/test_comm_mode_features 1/1
2024-12-18T03:37:24.7377857Z     distributed/checkpoint/e2e/test_pipeline 1/1
2024-12-18T03:37:24.7378565Z     distributed/checkpoint/test_dtensor_checkpoint 1/1
2024-12-18T03:37:24.7379289Z     distributed/checkpoint/test_tp_checkpoint 1/1
2024-12-18T03:37:24.7380043Z     distributed/_composable/fsdp/test_fully_shard_overlap 1/1
2024-12-18T03:37:24.7381204Z     distributed/checkpoint/test_planner 1/1
2024-12-18T03:37:24.7381949Z     distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1
2024-12-18T03:37:24.7382771Z     distributed/checkpoint/test_state_dict_utils 1/1
2024-12-18T03:37:24.7383425Z     distributed/fsdp/test_shard_utils 1/1
2024-12-18T03:37:24.7384060Z     distributed/checkpoint/test_compatibility 1/1
2024-12-18T03:37:24.7384743Z     distributed/checkpoint/test_hsdp_checkpoint 1/1
2024-12-18T03:37:24.7385477Z     distributed/_composable/fsdp/test_fully_shard_state 1/1
2024-12-18T03:37:24.7386183Z     distributed/test_control_collectives 1/1
2024-12-18T03:37:24.7386962Z     distributed/_tensor/debug/test_comm_mode 1/1
2024-12-18T03:37:24.7387611Z     distributed/_composable/test_replicate 1/1
2024-12-18T03:37:24.7388246Z     distributed/fsdp/test_fsdp_state_dict 3/3
2024-12-18T03:37:24.7388842Z     distributed/test_c10d_nccl 1/3
2024-12-18T03:37:24.7389409Z     distributed/fsdp/test_fsdp_overlap 1/1
2024-12-18T03:37:24.7390049Z     distributed/fsdp/test_fsdp_optim_state 2/2
2024-12-18T03:37:24.7390686Z     distributed/nn/jit/test_instantiator 1/1
2024-12-18T03:37:24.7391340Z     distributed/fsdp/test_fsdp_backward_prefetch 1/1
2024-12-18T03:37:24.7391996Z     distributed/fsdp/test_fsdp_pure_fp16 1/1
2024-12-18T03:37:24.7392591Z     distributed/fsdp/test_fsdp_memory 1/1
2024-12-18T03:37:24.7393281Z     distributed/checkpoint/test_file_system_checkpoint 1/1
2024-12-18T03:37:24.7393982Z     distributed/fsdp/test_fsdp_apply 1/1
2024-12-18T03:37:24.7394612Z     distributed/fsdp/test_fsdp_flatten_params 1/1
2024-12-18T03:37:24.7395325Z     distributed/optim/test_zero_redundancy_optimizer 1/1
2024-12-18T03:37:24.7396001Z     distributed/test_multi_threaded_pg 1/1
2024-12-18T03:37:24.7396593Z     distributed/fsdp/test_fsdp_uneven 1/1
2024-12-18T03:37:24.7397574Z     distributed/fsdp/test_fsdp_input 1/1
2024-12-18T03:37:24.7398169Z     distributed/test_c10d_ucc 1/1
2024-12-18T03:37:24.7398779Z     distributed/fsdp/test_fsdp_ignored_modules 1/1
2024-12-18T03:37:24.7399613Z     distributed/_composable/test_composability/test_2d_composability 1/1
2024-12-18T03:37:24.7400414Z     distributed/test_pg_wrapper 1/1
2024-12-18T03:37:24.7401010Z     distributed/fsdp/test_fsdp_hybrid_shard 1/1
2024-12-18T03:37:24.7401755Z     distributed/checkpoint/test_file_system_checkpoint_cpu 1/1
2024-12-18T03:37:24.7402477Z     distributed/test_data_parallel 1/1
2024-12-18T03:37:24.7403070Z     distributed/fsdp/test_fsdp_core 1/3
2024-12-18T03:37:24.7403749Z     distributed/_shard/sharding_plan/test_sharding_plan 1/1
2024-12-18T03:37:24.7404499Z     distributed/fsdp/test_fsdp_multiple_forward 1/1
2024-12-18T03:37:24.7405205Z     distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1
2024-12-18T03:37:24.7405865Z     distributed/_shard/test_sharder 1/1
2024-12-18T03:37:24.7406448Z     distributed/fsdp/test_fsdp_comm 1/1
2024-12-18T03:37:24.7407089Z     distributed/fsdp/test_distributed_checkpoint 1/1
2024-12-18T03:37:24.7407793Z     distributed/fsdp/test_fsdp_use_orig_params 1/1
2024-12-18T03:37:24.7408419Z     distributed/test_c10d_gloo 2/2
2024-12-18T03:37:24.7408944Z   Parallel tests (0):
2024-12-18T03:37:24.7409419Z Name: excluded (est. time: 0.0min)
2024-12-18T03:37:24.7409938Z   Serial tests (0):
2024-12-18T03:37:24.7410368Z   Parallel tests (0):
2024-12-18T03:37:24.7494998Z Running distributed/test_inductor_collectives 1/1 ... [2024-12-18 03:37:24.748736]
2024-12-18T03:37:24.7496044Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:37:24.7498056Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:37:24.749029]
2024-12-18T03:40:19.2234031Z 
2024-12-18T03:40:19.2235992Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_68a7a5fbad4e9b59_.log
2024-12-18T03:40:19.2278809Z Running 48 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta
2024-12-18T03:40:19.2319179Z 
2024-12-18T03:40:19.2319688Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-12-18 03:40:19.224425]
2024-12-18T03:40:19.2320605Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:40:19.2322968Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:40:19.225054]
2024-12-18T03:41:22.8900180Z 
2024-12-18T03:41:22.8901826Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_96c062f2a0ead921_.log
2024-12-18T03:41:22.8913833Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion
2024-12-18T03:41:22.8920625Z 
2024-12-18T03:41:22.8921043Z Running distributed/_tensor/test_init 1/1 ... [2024-12-18 03:41:22.889493]
2024-12-18T03:41:22.8921852Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:41:22.8923736Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:41:22.890092]
2024-12-18T03:42:10.1155140Z 
2024-12-18T03:42:10.1156755Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_3b5fc1a45b7e03e8_.log
2024-12-18T03:42:10.1161905Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh
2024-12-18T03:42:10.1166931Z 
2024-12-18T03:42:10.1167431Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-12-18 03:42:10.115376]
2024-12-18T03:42:10.1168342Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:42:10.1170311Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:42:10.115976]
2024-12-18T03:42:36.9448742Z 
2024-12-18T03:42:36.9450547Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_aa04315bd3df3f31_.log
2024-12-18T03:42:36.9453813Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice
2024-12-18T03:42:36.9455835Z 
2024-12-18T03:42:36.9456304Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-12-18 03:42:36.944814]
2024-12-18T03:42:36.9457245Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:42:36.9460041Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:42:36.945391]
2024-12-18T03:48:22.6767724Z 
2024-12-18T03:48:22.6769331Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_c3244eed237fa88e_.log
2024-12-18T03:48:22.6784602Z Running 25 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum
2024-12-18T03:48:22.6800867Z 
2024-12-18T03:48:22.6801390Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2024-12-18 03:48:22.676871]
2024-12-18T03:48:22.6802301Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:48:22.6804271Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:48:22.677443]
2024-12-18T03:49:37.5673129Z 
2024-12-18T03:49:37.5674620Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_cbd7fb56e06583d7_.log
2024-12-18T03:49:37.5678830Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2024-12-18T03:49:37.5681839Z 
2024-12-18T03:49:37.5682327Z Running distributed/test_c10d_common 1/1 ... [2024-12-18 03:49:37.567749]
2024-12-18T03:49:37.5683143Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:49:37.5690448Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:49:37.568336]
2024-12-18T03:53:40.8732253Z 
2024-12-18T03:53:40.8734047Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_b5bad9f56c5f78c9_.log
2024-12-18T03:53:40.8753016Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2024-12-18T03:53:40.8771680Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier
2024-12-18T03:53:40.8773469Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype
2024-12-18T03:53:40.8775268Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype
2024-12-18T03:53:40.8777019Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype
2024-12-18T03:53:40.8778786Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype
2024-12-18T03:53:40.8780335Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level
2024-12-18T03:53:40.8781863Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr
2024-12-18T03:53:40.8783640Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config
2024-12-18T03:53:40.8785332Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives
2024-12-18T03:53:40.8788772Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name
2024-12-18T03:53:40.8791076Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends
2024-12-18T03:53:40.8793067Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available
2024-12-18T03:53:40.8794787Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv
2024-12-18T03:53:40.8796744Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends
2024-12-18T03:53:40.8799005Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2024-12-18T03:53:40.8801174Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop
2024-12-18T03:53:40.8802948Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable
2024-12-18T03:53:40.8804608Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal
2024-12-18T03:53:40.8806028Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle
2024-12-18T03:53:40.8807426Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank
2024-12-18T03:53:40.8808939Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback
2024-12-18T03:53:40.8810439Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv
2024-12-18T03:53:40.8811860Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2024-12-18T03:53:40.8812713Z 
2024-12-18T03:53:40.8813277Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2024-12-18 03:53:40.873953]
2024-12-18T03:53:40.8814222Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:53:40.8816673Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:53:40.874577]
2024-12-18T03:53:46.4543172Z 
2024-12-18T03:53:46.4545119Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_58c2639846c056da_.log
2024-12-18T03:53:46.4548142Z Running 1 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event
2024-12-18T03:53:46.4549184Z 
2024-12-18T03:53:46.4550242Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-12-18 03:53:46.454624]
2024-12-18T03:53:46.4551226Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:53:46.4558353Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:53:46.455234]
2024-12-18T03:54:49.5708798Z 
2024-12-18T03:54:49.5710588Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_e0b8ca91de6d7d7f_.log
2024-12-18T03:54:49.5740058Z Running 30 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True
2024-12-18T03:54:49.5766516Z 
2024-12-18T03:54:49.5767090Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-12-18 03:54:49.571360]
2024-12-18T03:54:49.5768046Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:54:49.5770142Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:54:49.572008]
2024-12-18T03:54:55.2519508Z 
2024-12-18T03:54:55.2521189Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_b8f7cb8d908381ba_.log
2024-12-18T03:54:55.2524026Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping
2024-12-18T03:54:55.2525519Z 
2024-12-18T03:54:55.2528742Z Running distributed/pipelining/test_transformer 1/1 ... [2024-12-18 03:54:55.252320]
2024-12-18T03:54:55.2529745Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:54:55.2536349Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:54:55.252942]
2024-12-18T03:55:01.2840418Z 
2024-12-18T03:55:01.2842188Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_131b27f0513507d0_.log
2024-12-18T03:55:01.2844366Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir
2024-12-18T03:55:01.2845199Z 
2024-12-18T03:55:01.2849145Z Running distributed/pipelining/test_backward 1/1 ... [2024-12-18 03:55:01.284235]
2024-12-18T03:55:01.2850112Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:55:01.2854163Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:55:01.284783]
2024-12-18T03:55:07.9674988Z 
2024-12-18T03:55:07.9676661Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_d090e960195e3a60_.log
2024-12-18T03:55:07.9681055Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_input, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight_multiple_iters
2024-12-18T03:55:07.9684538Z 
2024-12-18T03:55:07.9685058Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-12-18 03:55:07.967020]
2024-12-18T03:55:07.9686707Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:55:07.9688681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:55:07.967636]
2024-12-18T03:55:13.5973528Z 
2024-12-18T03:55:13.5975268Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_b550bd56b4fe7412_.log
2024-12-18T03:55:13.5977722Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger
2024-12-18T03:55:13.5978742Z 
2024-12-18T03:55:13.5981182Z Running distributed/pipelining/test_pipe 1/1 ... [2024-12-18 03:55:13.597658]
2024-12-18T03:55:13.5982073Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:55:13.5988544Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:55:13.598245]
2024-12-18T03:55:20.1801343Z 
2024-12-18T03:55:20.1803223Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_fea425310abf1c0b_.log
2024-12-18T03:55:20.1806556Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2
2024-12-18T03:55:20.1809757Z 
2024-12-18T03:55:20.1810480Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2024-12-18 03:55:20.180376]
2024-12-18T03:55:20.1811478Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:55:20.1813943Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:55:20.180916]
2024-12-18T03:57:14.8117421Z 
2024-12-18T03:57:14.8119832Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_1dfde7e8e53f89bc_.log
2024-12-18T03:57:14.8136070Z Running 16 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True
2024-12-18T03:57:14.8152050Z 
2024-12-18T03:57:14.8152613Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-12-18 03:57:14.812038]
2024-12-18T03:57:14.8153517Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:14.8155475Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:14.812622]
2024-12-18T03:57:25.5547244Z 
2024-12-18T03:57:25.5549078Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_f3d6436631659283_.log
2024-12-18T03:57:25.5551453Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect
2024-12-18T03:57:25.5552555Z 
2024-12-18T03:57:25.5562937Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-12-18 03:57:25.555147]
2024-12-18T03:57:25.5566690Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:25.5568741Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:25.555818]
2024-12-18T03:57:31.2368873Z 
2024-12-18T03:57:31.2370710Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_783abe034de69c7c_.log
2024-12-18T03:57:31.2372986Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards
2024-12-18T03:57:31.2373934Z 
2024-12-18T03:57:31.2378551Z Running distributed/_composable/test_contract 1/1 ... [2024-12-18 03:57:31.237274]
2024-12-18T03:57:31.2379569Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:31.2385196Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:31.237879]
2024-12-18T03:57:36.8176872Z 
2024-12-18T03:57:36.8178960Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_881d7a5f728e5e35_.log
2024-12-18T03:57:36.8183154Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state
2024-12-18T03:57:36.8186059Z 
2024-12-18T03:57:36.8186768Z Running distributed/pipelining/test_microbatch 1/1 ... [2024-12-18 03:57:36.817940]
2024-12-18T03:57:36.8187673Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:36.8191671Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:36.818542]
2024-12-18T03:57:43.3510716Z 
2024-12-18T03:57:43.3512430Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_18278882ef4449aa_.log
2024-12-18T03:57:43.3515217Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_chunk_spec, test/distributed/pipelining/test_microbatch.py::MicrobatchTests::test_split_and_merge
2024-12-18T03:57:43.3516683Z 
2024-12-18T03:57:43.3517243Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-12-18 03:57:43.350959]
2024-12-18T03:57:43.3518134Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:43.3521058Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:43.351558]
2024-12-18T03:57:54.5443919Z 
2024-12-18T03:57:54.5446001Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_ad6628436927b910_.log
2024-12-18T03:57:54.5448429Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e
2024-12-18T03:57:54.5449414Z 
2024-12-18T03:57:54.5450022Z Running distributed/_tensor/experimental/test_register_sharding 1/1 ... [2024-12-18 03:57:54.544513]
2024-12-18T03:57:54.5451964Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:57:54.5458113Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:57:54.545115]
2024-12-18T03:58:14.3584107Z 
2024-12-18T03:58:14.3586180Z distributed/_tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.experimental.test_register_sharding_1.1_4fc198bb76fef3f6_.log
2024-12-18T03:58:14.3589797Z Running 2 items in this shard: test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/_tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd
2024-12-18T03:58:14.3591585Z 
2024-12-18T03:58:14.3592090Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-12-18 03:58:14.358192]
2024-12-18T03:58:14.3592930Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:58:14.3594862Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:58:14.358811]
2024-12-18T03:58:26.7066569Z 
2024-12-18T03:58:26.7068713Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_deda8e26e3c22323_.log
2024-12-18T03:58:26.7072443Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_cuda_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing
2024-12-18T03:58:26.7075083Z 
2024-12-18T03:58:26.7075532Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-12-18 03:58:26.706516]
2024-12-18T03:58:26.7077395Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:58:26.7079626Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:58:26.707115]
2024-12-18T03:58:33.0387004Z 
2024-12-18T03:58:33.0388494Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_4805875a53de68ce_.log
2024-12-18T03:58:33.0390606Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracingCUDA::test_symbolic_tracing_outputs_cuda
2024-12-18T03:58:33.0391614Z 
2024-12-18T03:58:33.0392528Z Running distributed/_tensor/test_api 1/1 ... [2024-12-18 03:58:33.038751]
2024-12-18T03:58:33.0393395Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:58:33.0399519Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:58:33.039304]
2024-12-18T03:59:26.3735944Z 
2024-12-18T03:59:26.3737966Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_267f8fc3de9c1ceb_.log
2024-12-18T03:59:26.3745414Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding
2024-12-18T03:59:26.3750885Z 
2024-12-18T03:59:26.3751422Z Running distributed/pipelining/test_unflatten 1/1 ... [2024-12-18 03:59:26.373133]
2024-12-18T03:59:26.3752353Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:59:26.3754418Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:59:26.373742]
2024-12-18T03:59:32.4053399Z 
2024-12-18T03:59:32.4055341Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_177835b6c088b14b_.log
2024-12-18T03:59:32.4057690Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTests::test_unflatten
2024-12-18T03:59:32.4058601Z 
2024-12-18T03:59:32.4059152Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2024-12-18 03:59:32.405392]
2024-12-18T03:59:32.4060094Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T03:59:32.4066991Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:59:32.406037]
2024-12-18T04:00:31.8657180Z 
2024-12-18T04:00:31.8659240Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_b37722d16af97e99_.log
2024-12-18T04:00:31.8668848Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp
2024-12-18T04:00:31.8676664Z 
2024-12-18T04:00:31.8677371Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2024-12-18 04:00:31.865857]
2024-12-18T04:00:31.8678417Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:00:31.8680619Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:00:31.866429]
2024-12-18T04:00:37.1240335Z 
2024-12-18T04:00:37.1242590Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_6b5902679e757c56_.log
2024-12-18T04:00:37.1244721Z Running 0 items in this shard:
2024-12-18T04:00:37.1245094Z 
2024-12-18T04:00:37.1246282Z Running distributed/pipelining/test_stage 1/1 ... [2024-12-18 04:00:37.124160]
2024-12-18T04:00:37.1247201Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:00:37.1254479Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:00:37.124699]
2024-12-18T04:01:02.1506927Z 
2024-12-18T04:01:02.1512609Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_8b171b6457337b35_.log
2024-12-18T04:01:02.1517580Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0
2024-12-18T04:01:02.1525371Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0
2024-12-18T04:01:02.1528867Z 
2024-12-18T04:01:02.1529348Z Running distributed/_tools/test_memory_tracker 1/1 ... [2024-12-18 04:01:02.150418]
2024-12-18T04:01:02.1530207Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:01:02.1532359Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:01:02.151027]
2024-12-18T04:01:16.9535410Z 
2024-12-18T04:01:16.9537120Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_fcda1e259621697f_.log
2024-12-18T04:01:16.9540391Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model
2024-12-18T04:01:16.9541262Z 
2024-12-18T04:01:16.9541939Z Running distributed/_composable/test_composability/test_pp_composability 1/1 ... [2024-12-18 04:01:16.953287]
2024-12-18T04:01:16.9543016Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:01:16.9545153Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_pp_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:01:16.953879]
2024-12-18T04:03:18.8979849Z 
2024-12-18T04:03:18.8982317Z distributed/_composable/test_composability/test_pp_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_dc9931ecfbdd7a09_.log
2024-12-18T04:03:18.9006542Z Running 21 items in this shard: test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass0_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass0_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass1_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass1_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass2_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass2_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass3_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass3_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass4_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_DDP_ScheduleClass4_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass0_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass0_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass1_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass1_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass2_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass2_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass3_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass3_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass4_use_new_runtime_False, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_manual_with_data_parallel_dp_type_FSDP_ScheduleClass4_use_new_runtime_True, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_pp_and_dcp
2024-12-18T04:03:18.9032283Z 
2024-12-18T04:03:18.9032774Z Running distributed/test_distributed_spawn 2/12 ... [2024-12-18 04:03:18.897647]
2024-12-18T04:03:18.9033922Z MPI not available -- MPI backend tests will be skipped
2024-12-18T04:03:18.9034926Z Running distributed tests for the test backend with env init_method
2024-12-18T04:03:18.9035864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:03:18.9037707Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:03:18.902372]
2024-12-18T04:03:24.4324892Z 
2024-12-18T04:03:24.4326813Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_421ecc5ef85b394e_.log
2024-12-18T04:03:24.4328445Z Running 0 items in this shard:
2024-12-18T04:03:24.4328810Z 
2024-12-18T04:03:24.4340763Z Running distributed tests for the test backend with file init_method
2024-12-18T04:03:24.4342598Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:03:24.4348498Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:03:24.434277]
2024-12-18T04:03:29.9900888Z 
2024-12-18T04:03:29.9902745Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_b29323564d0c2bba_.log
2024-12-18T04:03:29.9904338Z Running 0 items in this shard:
2024-12-18T04:03:29.9904693Z 
2024-12-18T04:03:29.9915071Z Running distributed tests for the nccl backend with env init_method
2024-12-18T04:03:29.9917802Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:03:29.9925107Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:03:29.991807]
2024-12-18T04:07:53.7108857Z 
2024-12-18T04:07:53.7110934Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_5adb684c22e30129_.log
2024-12-18T04:07:53.7143328Z Running 25 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:07:53.7163999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2024-12-18T04:07:53.7166159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2024-12-18T04:07:53.7168260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2024-12-18T04:07:53.7170113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2024-12-18T04:07:53.7171923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2024-12-18T04:07:53.7173650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2024-12-18T04:07:53.7175396Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2024-12-18T04:07:53.7177088Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2024-12-18T04:07:53.7178842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2024-12-18T04:07:53.7180630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2024-12-18T04:07:53.7182227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2024-12-18T04:07:53.7184015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2024-12-18T04:07:53.7185869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2024-12-18T04:07:53.7188280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2024-12-18T04:07:53.7189991Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2024-12-18T04:07:53.7192035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2024-12-18T04:07:53.7193972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2024-12-18T04:07:53.7195551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2024-12-18T04:07:53.7197193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2024-12-18T04:07:53.7199162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2024-12-18T04:07:53.7201353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2024-12-18T04:07:53.7203226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2024-12-18T04:07:53.7204866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2024-12-18T04:07:53.7206554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2024-12-18T04:07:53.7208600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:07:53.7209582Z 
2024-12-18T04:07:53.7209978Z Running distributed tests for the nccl backend with file init_method
2024-12-18T04:07:53.7210782Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:07:53.7212802Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:07:53.713598]
2024-12-18T04:12:17.5008905Z 
2024-12-18T04:12:17.5011652Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_1701e152ded10549_.log
2024-12-18T04:12:17.5031743Z Running 25 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:12:17.5053014Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2024-12-18T04:12:17.5055079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2024-12-18T04:12:17.5057125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2024-12-18T04:12:17.5059318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2024-12-18T04:12:17.5061099Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2024-12-18T04:12:17.5062827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2024-12-18T04:12:17.5064549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2024-12-18T04:12:17.5066220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2024-12-18T04:12:17.5068144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2024-12-18T04:12:17.5070284Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2024-12-18T04:12:17.5071963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2024-12-18T04:12:17.5073737Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2024-12-18T04:12:17.5075559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2024-12-18T04:12:17.5077222Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2024-12-18T04:12:17.5079126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2024-12-18T04:12:17.5081515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2024-12-18T04:12:17.5083662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2024-12-18T04:12:17.5085575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2024-12-18T04:12:17.5087187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2024-12-18T04:12:17.5089351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2024-12-18T04:12:17.5091727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2024-12-18T04:12:17.5093562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2024-12-18T04:12:17.5095178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2024-12-18T04:12:17.5096843Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2024-12-18T04:12:17.5098534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:12:17.5099466Z 
2024-12-18T04:12:17.5099850Z Running distributed tests for the gloo backend with env init_method
2024-12-18T04:12:17.5100616Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:12:17.5102612Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:12:17.503108]
2024-12-18T04:16:55.4811829Z 
2024-12-18T04:16:55.4815141Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_f7be46cd1af7782b_.log
2024-12-18T04:16:55.4835903Z Running 25 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:16:55.4856688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2024-12-18T04:16:55.4858769Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2024-12-18T04:16:55.4860808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2024-12-18T04:16:55.4862601Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2024-12-18T04:16:55.4864420Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2024-12-18T04:16:55.4866204Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2024-12-18T04:16:55.4868032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2024-12-18T04:16:55.4869692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2024-12-18T04:16:55.4871946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2024-12-18T04:16:55.4874108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2024-12-18T04:16:55.4875992Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2024-12-18T04:16:55.4877974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2024-12-18T04:16:55.4879777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2024-12-18T04:16:55.4881424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2024-12-18T04:16:55.4883111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2024-12-18T04:16:55.4885125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2024-12-18T04:16:55.4887363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2024-12-18T04:16:55.4888916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2024-12-18T04:16:55.4890517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2024-12-18T04:16:55.4892452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2024-12-18T04:16:55.4894611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2024-12-18T04:16:55.4896788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2024-12-18T04:16:55.4898399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2024-12-18T04:16:55.4900093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2024-12-18T04:16:55.4901776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:16:55.4902704Z 
2024-12-18T04:16:55.4903110Z Running distributed tests for the gloo backend with file init_method
2024-12-18T04:16:55.4903901Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:16:55.4905901Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:16:55.482830]
2024-12-18T04:21:32.6939008Z 
2024-12-18T04:21:32.6940910Z distributed/test_distributed_spawn 2/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.12_23741254553e6064_.log
2024-12-18T04:21:32.6962693Z Running 25 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:21:32.6983250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2024-12-18T04:21:32.6985724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2024-12-18T04:21:32.6987923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2024-12-18T04:21:32.6990012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2024-12-18T04:21:32.6991845Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2024-12-18T04:21:32.6993551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2024-12-18T04:21:32.6995256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2024-12-18T04:21:32.6996975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2024-12-18T04:21:32.6998860Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2024-12-18T04:21:32.7000958Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2024-12-18T04:21:32.7002834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2024-12-18T04:21:32.7004699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2024-12-18T04:21:32.7006859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2024-12-18T04:21:32.7008553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2024-12-18T04:21:32.7010260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2024-12-18T04:21:32.7012267Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2024-12-18T04:21:32.7014205Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2024-12-18T04:21:32.7015772Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2024-12-18T04:21:32.7017390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2024-12-18T04:21:32.7019334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2024-12-18T04:21:32.7021495Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2024-12-18T04:21:32.7023342Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2024-12-18T04:21:32.7024969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2024-12-18T04:21:32.7026942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2024-12-18T04:21:32.7028714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2024-12-18T04:21:32.7029837Z 
2024-12-18T04:21:32.7030815Z Running distributed/test_distributed_spawn 5/12 ... [2024-12-18 04:21:32.695882]
2024-12-18T04:21:32.7031807Z MPI not available -- MPI backend tests will be skipped
2024-12-18T04:21:32.7032683Z Running distributed tests for the test backend with env init_method
2024-12-18T04:21:32.7033461Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:21:32.7035444Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:21:32.700153]
2024-12-18T04:21:38.1428228Z 
2024-12-18T04:21:38.1429820Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_073b4a4190748e85_.log
2024-12-18T04:21:38.1431459Z Running 0 items in this shard:
2024-12-18T04:21:38.1431826Z 
2024-12-18T04:21:38.1442789Z Running distributed tests for the test backend with file init_method
2024-12-18T04:21:38.1443731Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:21:38.1452895Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:21:38.144597]
2024-12-18T04:21:43.6519558Z 
2024-12-18T04:21:43.6521101Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_3f292d01448a2652_.log
2024-12-18T04:21:43.6522689Z Running 0 items in this shard:
2024-12-18T04:21:43.6523049Z 
2024-12-18T04:21:43.6534346Z Running distributed tests for the nccl backend with env init_method
2024-12-18T04:21:43.6538421Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:21:43.6544752Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:21:43.653813]
2024-12-18T04:26:10.5352459Z 
2024-12-18T04:26:10.5358207Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_512f0284a560d9e6_.log
2024-12-18T04:26:10.5375717Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:26:10.5394493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2024-12-18T04:26:10.5396615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2024-12-18T04:26:10.5398553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2024-12-18T04:26:10.5400221Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda
2024-12-18T04:26:10.5402007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2024-12-18T04:26:10.5403928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2024-12-18T04:26:10.5406179Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs
2024-12-18T04:26:10.5407880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2024-12-18T04:26:10.5409577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2024-12-18T04:26:10.5411499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params
2024-12-18T04:26:10.5413552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2024-12-18T04:26:10.5415405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features
2024-12-18T04:26:10.5417125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group
2024-12-18T04:26:10.5418812Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2024-12-18T04:26:10.5420485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2024-12-18T04:26:10.5422082Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler
2024-12-18T04:26:10.5423872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2024-12-18T04:26:10.5425732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2024-12-18T04:26:10.5427808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2024-12-18T04:26:10.5429556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2024-12-18T04:26:10.5431558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2024-12-18T04:26:10.5433249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:26:10.5434228Z 
2024-12-18T04:26:10.5434648Z Running distributed tests for the nccl backend with file init_method
2024-12-18T04:26:10.5435431Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:26:10.5437432Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:26:10.537641]
2024-12-18T04:30:36.4592965Z 
2024-12-18T04:30:36.4594697Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_b4a5685261ccdc7b_.log
2024-12-18T04:30:36.4613799Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:30:36.4631897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2024-12-18T04:30:36.4634408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2024-12-18T04:30:36.4636748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2024-12-18T04:30:36.4638440Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda
2024-12-18T04:30:36.4640663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2024-12-18T04:30:36.4642582Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2024-12-18T04:30:36.4644403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs
2024-12-18T04:30:36.4646095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2024-12-18T04:30:36.4647782Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2024-12-18T04:30:36.4649739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params
2024-12-18T04:30:36.4651834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2024-12-18T04:30:36.4653723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features
2024-12-18T04:30:36.4655433Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group
2024-12-18T04:30:36.4657167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2024-12-18T04:30:36.4658833Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2024-12-18T04:30:36.4660771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler
2024-12-18T04:30:36.4662595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2024-12-18T04:30:36.4664475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2024-12-18T04:30:36.4666260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2024-12-18T04:30:36.4668137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2024-12-18T04:30:36.4669787Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2024-12-18T04:30:36.4671490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:30:36.4672478Z 
2024-12-18T04:30:36.4672894Z Running distributed tests for the gloo backend with env init_method
2024-12-18T04:30:36.4673721Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:30:36.4675722Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:30:36.461715]
2024-12-18T04:35:14.6486274Z 
2024-12-18T04:35:14.6488306Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_12850680cf337135_.log
2024-12-18T04:35:14.6507224Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:35:14.6526042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2024-12-18T04:35:14.6528161Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2024-12-18T04:35:14.6530119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2024-12-18T04:35:14.6531795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda
2024-12-18T04:35:14.6533583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2024-12-18T04:35:14.6535514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2024-12-18T04:35:14.6537323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs
2024-12-18T04:35:14.6539030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2024-12-18T04:35:14.6540727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2024-12-18T04:35:14.6542679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params
2024-12-18T04:35:14.6544747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2024-12-18T04:35:14.6546748Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features
2024-12-18T04:35:14.6548686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group
2024-12-18T04:35:14.6550970Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2024-12-18T04:35:14.6552630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2024-12-18T04:35:14.6554245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler
2024-12-18T04:35:14.6556042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2024-12-18T04:35:14.6558032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2024-12-18T04:35:14.6560165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2024-12-18T04:35:14.6562248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2024-12-18T04:35:14.6563903Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2024-12-18T04:35:14.6565606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:35:14.6566580Z 
2024-12-18T04:35:14.6566991Z Running distributed tests for the gloo backend with file init_method
2024-12-18T04:35:14.6567817Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:35:14.6570534Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:35:14.651073]
2024-12-18T04:39:53.0415493Z 
2024-12-18T04:39:53.0417980Z distributed/test_distributed_spawn 5/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.12_cdf7a38acf662072_.log
2024-12-18T04:39:53.0436276Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:39:53.0456022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2024-12-18T04:39:53.0458101Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2024-12-18T04:39:53.0460024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex
2024-12-18T04:39:53.0461676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda
2024-12-18T04:39:53.0463427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex
2024-12-18T04:39:53.0465307Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2024-12-18T04:39:53.0467249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs
2024-12-18T04:39:53.0468927Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2024-12-18T04:39:53.0471150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2024-12-18T04:39:53.0473275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params
2024-12-18T04:39:53.0475336Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2024-12-18T04:39:53.0477194Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features
2024-12-18T04:39:53.0478872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group
2024-12-18T04:39:53.0480561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars
2024-12-18T04:39:53.0482210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2024-12-18T04:39:53.0483817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler
2024-12-18T04:39:53.0485685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2024-12-18T04:39:53.0487556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2024-12-18T04:39:53.0489335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2024-12-18T04:39:53.0491061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2024-12-18T04:39:53.0492729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice
2024-12-18T04:39:53.0494426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2024-12-18T04:39:53.0495747Z 
2024-12-18T04:39:53.0496225Z Running distributed/test_distributed_spawn 8/12 ... [2024-12-18 04:39:53.043383]
2024-12-18T04:39:53.0497145Z MPI not available -- MPI backend tests will be skipped
2024-12-18T04:39:53.0497988Z Running distributed tests for the test backend with env init_method
2024-12-18T04:39:53.0498762Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:39:53.0500752Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:39:53.047500]
2024-12-18T04:39:58.5544948Z 
2024-12-18T04:39:58.5546978Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_d21f7cac556f544a_.log
2024-12-18T04:39:58.5547854Z Running 0 items in this shard:
2024-12-18T04:39:58.5548044Z 
2024-12-18T04:39:58.5558239Z Running distributed tests for the test backend with file init_method
2024-12-18T04:39:58.5562365Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:39:58.5569197Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:39:58.556207]
2024-12-18T04:40:04.0650224Z 
2024-12-18T04:40:04.0652301Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_4a785e41ae60ef0c_.log
2024-12-18T04:40:04.0653928Z Running 0 items in this shard:
2024-12-18T04:40:04.0655197Z 
2024-12-18T04:40:04.0663142Z Running distributed tests for the nccl backend with env init_method
2024-12-18T04:40:04.0664503Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:40:04.0672701Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:40:04.066643]
2024-12-18T04:44:18.3479266Z 
2024-12-18T04:44:18.3480921Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_e01b4ab38607f772_.log
2024-12-18T04:44:18.3498594Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:44:18.3520110Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2024-12-18T04:44:18.3522105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2024-12-18T04:44:18.3523870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex
2024-12-18T04:44:18.3525574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2024-12-18T04:44:18.3527306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2024-12-18T04:44:18.3529518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2024-12-18T04:44:18.3531502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda
2024-12-18T04:44:18.3533513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group
2024-12-18T04:44:18.3535330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2024-12-18T04:44:18.3537079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2024-12-18T04:44:18.3538847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2024-12-18T04:44:18.3540814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2024-12-18T04:44:18.3542705Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2024-12-18T04:44:18.3544321Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2024-12-18T04:44:18.3546095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2024-12-18T04:44:18.3548237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2024-12-18T04:44:18.3550113Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module
2024-12-18T04:44:18.3551912Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2024-12-18T04:44:18.3553729Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload
2024-12-18T04:44:18.3555857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum
2024-12-18T04:44:18.3557517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda
2024-12-18T04:44:18.3559312Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:44:18.3560358Z 
2024-12-18T04:44:18.3560752Z Running distributed tests for the nccl backend with file init_method
2024-12-18T04:44:18.3561534Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:44:18.3563538Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:44:18.350405]
2024-12-18T04:48:32.6889248Z 
2024-12-18T04:48:32.6891317Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_b14d5910861700aa_.log
2024-12-18T04:48:32.6911440Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:48:32.6929997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2024-12-18T04:48:32.6931966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2024-12-18T04:48:32.6933734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex
2024-12-18T04:48:32.6936040Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2024-12-18T04:48:32.6937987Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2024-12-18T04:48:32.6939799Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2024-12-18T04:48:32.6941753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda
2024-12-18T04:48:32.6943805Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group
2024-12-18T04:48:32.6945685Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2024-12-18T04:48:32.6947562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2024-12-18T04:48:32.6949333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2024-12-18T04:48:32.6951141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2024-12-18T04:48:32.6952988Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2024-12-18T04:48:32.6954628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2024-12-18T04:48:32.6957111Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2024-12-18T04:48:32.6959486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2024-12-18T04:48:32.6961640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module
2024-12-18T04:48:32.6963467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2024-12-18T04:48:32.6965556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload
2024-12-18T04:48:32.6967333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum
2024-12-18T04:48:32.6968995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda
2024-12-18T04:48:32.6970808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:48:32.6971862Z 
2024-12-18T04:48:32.6972273Z Running distributed tests for the gloo backend with env init_method
2024-12-18T04:48:32.6973054Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:48:32.6975058Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:48:32.691364]
2024-12-18T04:52:43.0187536Z 
2024-12-18T04:52:43.0193474Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_becf32c5e9057910_.log
2024-12-18T04:52:43.0213190Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:52:43.0231456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2024-12-18T04:52:43.0233580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2024-12-18T04:52:43.0235672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex
2024-12-18T04:52:43.0237651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2024-12-18T04:52:43.0239363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2024-12-18T04:52:43.0241114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2024-12-18T04:52:43.0243130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda
2024-12-18T04:52:43.0245501Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group
2024-12-18T04:52:43.0247323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2024-12-18T04:52:43.0249355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2024-12-18T04:52:43.0251149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2024-12-18T04:52:43.0252949Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2024-12-18T04:52:43.0255094Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2024-12-18T04:52:43.0256695Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2024-12-18T04:52:43.0258469Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2024-12-18T04:52:43.0260440Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2024-12-18T04:52:43.0262306Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module
2024-12-18T04:52:43.0264400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2024-12-18T04:52:43.0266379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload
2024-12-18T04:52:43.0268474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum
2024-12-18T04:52:43.0270131Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda
2024-12-18T04:52:43.0271922Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:52:43.0272968Z 
2024-12-18T04:52:43.0273377Z Running distributed tests for the gloo backend with file init_method
2024-12-18T04:52:43.0274588Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:52:43.0276587Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:52:43.021182]
2024-12-18T04:56:52.4748955Z 
2024-12-18T04:56:52.4750612Z distributed/test_distributed_spawn 8/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.12_7a118cae8e9e51c5_.log
2024-12-18T04:56:52.4769587Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:56:52.4788658Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2024-12-18T04:56:52.4790866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2024-12-18T04:56:52.4792614Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex
2024-12-18T04:56:52.4794296Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2024-12-18T04:56:52.4796126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda
2024-12-18T04:56:52.4798196Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2024-12-18T04:56:52.4800877Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda
2024-12-18T04:56:52.4802876Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group
2024-12-18T04:56:52.4804688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2024-12-18T04:56:52.4806427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2024-12-18T04:56:52.4808186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2024-12-18T04:56:52.4809957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler
2024-12-18T04:56:52.4811723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2024-12-18T04:56:52.4813328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2024-12-18T04:56:52.4815104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2024-12-18T04:56:52.4817079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2024-12-18T04:56:52.4818981Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module
2024-12-18T04:56:52.4820768Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager
2024-12-18T04:56:52.4822562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload
2024-12-18T04:56:52.4824375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum
2024-12-18T04:56:52.4827093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda
2024-12-18T04:56:52.4828891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler
2024-12-18T04:56:52.4829934Z 
2024-12-18T04:56:52.4830386Z Running distributed/test_distributed_spawn 11/12 ... [2024-12-18 04:56:52.476638]
2024-12-18T04:56:52.4831319Z MPI not available -- MPI backend tests will be skipped
2024-12-18T04:56:52.4832167Z Running distributed tests for the test backend with env init_method
2024-12-18T04:56:52.4832949Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:56:52.4835128Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:56:52.480888]
2024-12-18T04:56:58.0126819Z 
2024-12-18T04:56:58.0128259Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_94d06be4ed4dfe85_.log
2024-12-18T04:56:58.0129830Z Running 0 items in this shard:
2024-12-18T04:56:58.0140120Z 
2024-12-18T04:56:58.0140708Z Running distributed tests for the test backend with file init_method
2024-12-18T04:56:58.0141608Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:56:58.0150273Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:56:58.014398]
2024-12-18T04:57:03.4906831Z 
2024-12-18T04:57:03.4909488Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_19f8e2c62c8a0f13_.log
2024-12-18T04:57:03.4911147Z Running 0 items in this shard:
2024-12-18T04:57:03.4911499Z 
2024-12-18T04:57:03.4921625Z Running distributed tests for the nccl backend with env init_method
2024-12-18T04:57:03.4925643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T04:57:03.4932250Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:57:03.492551]
2024-12-18T05:01:06.4632108Z 
2024-12-18T05:01:06.4633382Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_04c57fed3e24adef_.log
2024-12-18T05:01:06.4645651Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:01:06.4663913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2024-12-18T05:01:06.4665817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2024-12-18T05:01:06.4667891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2024-12-18T05:01:06.4669608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min
2024-12-18T05:01:06.4671727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2024-12-18T05:01:06.4673864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda
2024-12-18T05:01:06.4676037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2024-12-18T05:01:06.4677234Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2024-12-18T05:01:06.4678269Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2024-12-18T05:01:06.4679276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2024-12-18T05:01:06.4680369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2024-12-18T05:01:06.4681479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2024-12-18T05:01:06.4682683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none
2024-12-18T05:01:06.4683931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2024-12-18T05:01:06.4684982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2024-12-18T05:01:06.4686015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2024-12-18T05:01:06.4687105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2024-12-18T05:01:06.4688184Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static
2024-12-18T05:01:06.4689523Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync
2024-12-18T05:01:06.4690602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product
2024-12-18T05:01:06.4691585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2024-12-18T05:01:06.4692535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:01:06.4693080Z 
2024-12-18T05:01:06.4693326Z Running distributed tests for the nccl backend with file init_method
2024-12-18T05:01:06.4693814Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:01:06.4695029Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:01:06.465471]
2024-12-18T05:05:08.6122437Z 
2024-12-18T05:05:08.6123591Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_0897f813d4066607_.log
2024-12-18T05:05:08.6143991Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:05:08.6161073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2024-12-18T05:05:08.6162241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2024-12-18T05:05:08.6163652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2024-12-18T05:05:08.6164717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min
2024-12-18T05:05:08.6165726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2024-12-18T05:05:08.6166803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda
2024-12-18T05:05:08.6167992Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2024-12-18T05:05:08.6169136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2024-12-18T05:05:08.6170178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2024-12-18T05:05:08.6171199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2024-12-18T05:05:08.6172298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2024-12-18T05:05:08.6173410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2024-12-18T05:05:08.6174613Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none
2024-12-18T05:05:08.6176054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2024-12-18T05:05:08.6177102Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2024-12-18T05:05:08.6178129Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2024-12-18T05:05:08.6179197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2024-12-18T05:05:08.6180275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static
2024-12-18T05:05:08.6181399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync
2024-12-18T05:05:08.6182479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product
2024-12-18T05:05:08.6183452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2024-12-18T05:05:08.6184395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:05:08.6185025Z 
2024-12-18T05:05:08.6185441Z Running distributed tests for the gloo backend with env init_method
2024-12-18T05:05:08.6186353Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:05:08.6188637Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:05:08.614523]
2024-12-18T05:09:32.5580701Z 
2024-12-18T05:09:32.5582491Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_a49f665d90b21db1_.log
2024-12-18T05:09:32.5597719Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:09:32.5612725Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2024-12-18T05:09:32.5614613Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2024-12-18T05:09:32.5616441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2024-12-18T05:09:32.5618169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min
2024-12-18T05:09:32.5619897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2024-12-18T05:09:32.5621810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda
2024-12-18T05:09:32.5624151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2024-12-18T05:09:32.5626041Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2024-12-18T05:09:32.5627873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2024-12-18T05:09:32.5629549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2024-12-18T05:09:32.5631347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2024-12-18T05:09:32.5633515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2024-12-18T05:09:32.5635489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none
2024-12-18T05:09:32.5637580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2024-12-18T05:09:32.5639585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2024-12-18T05:09:32.5641591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2024-12-18T05:09:32.5643566Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2024-12-18T05:09:32.5644565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static
2024-12-18T05:09:32.5645518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync
2024-12-18T05:09:32.5646505Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product
2024-12-18T05:09:32.5647652Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2024-12-18T05:09:32.5648665Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:09:32.5649201Z 
2024-12-18T05:09:32.5649658Z Running distributed tests for the gloo backend with file init_method
2024-12-18T05:09:32.5650144Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:09:32.5651357Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=11', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:09:32.559850]
2024-12-18T05:13:57.1558580Z 
2024-12-18T05:13:57.1560901Z distributed/test_distributed_spawn 11/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_11.12_c875f0a93646c526_.log
2024-12-18T05:13:57.1579582Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:13:57.1598995Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2024-12-18T05:13:57.1601372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops
2024-12-18T05:13:57.1603710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2024-12-18T05:13:57.1605648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min
2024-12-18T05:13:57.1607400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2024-12-18T05:13:57.1609251Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda
2024-12-18T05:13:57.1611763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2024-12-18T05:13:57.1617249Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global
2024-12-18T05:13:57.1619217Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2024-12-18T05:13:57.1620996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2024-12-18T05:13:57.1622907Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2024-12-18T05:13:57.1624879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2024-12-18T05:13:57.1627172Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none
2024-12-18T05:13:57.1629381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2024-12-18T05:13:57.1631452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2024-12-18T05:13:57.1633437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2024-12-18T05:13:57.1635300Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2024-12-18T05:13:57.1637181Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static
2024-12-18T05:13:57.1639141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync
2024-12-18T05:13:57.1641260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product
2024-12-18T05:13:57.1643781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max
2024-12-18T05:13:57.1645555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2024-12-18T05:13:57.1646483Z 
2024-12-18T05:13:57.1646961Z Running distributed/checkpoint/test_fsspec 1/1 ... [2024-12-18 05:13:57.157696]
2024-12-18T05:13:57.1647845Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:13:57.1649936Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:13:57.158276]
2024-12-18T05:14:22.7859651Z 
2024-12-18T05:14:22.7861314Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_d451223279eac0d6_.log
2024-12-18T05:14:22.7864028Z Running 2 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_fsspec, test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_overwrite
2024-12-18T05:14:22.7865371Z 
2024-12-18T05:14:22.7870965Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-12-18 05:14:22.786363]
2024-12-18T05:14:22.7872035Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:14:22.7876375Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:14:22.787010]
2024-12-18T05:15:52.6644097Z 
2024-12-18T05:15:52.6645933Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_6d534a248fa46aba_.log
2024-12-18T05:15:52.6654540Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style
2024-12-18T05:15:52.6661720Z 
2024-12-18T05:15:52.6662307Z Running distributed/_tensor/debug/test_comm_mode_features 1/1 ... [2024-12-18 05:15:52.664595]
2024-12-18T05:15:52.6663270Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:15:52.6665293Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode_features.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:15:52.665193]
2024-12-18T05:16:49.2173162Z 
2024-12-18T05:16:49.2175229Z distributed/_tensor/debug/test_comm_mode_features 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_features_1.1_fb98f3983969ac97_.log
2024-12-18T05:16:49.2180318Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLPStacked_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_distributed_sharding_display, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_module_tracing, test/distributed/_tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_transformer_module_tracing
2024-12-18T05:16:49.2184779Z 
2024-12-18T05:16:49.2185305Z Running distributed/checkpoint/e2e/test_pipeline 1/1 ... [2024-12-18 05:16:49.217672]
2024-12-18T05:16:49.2186213Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:16:49.2188476Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_pipeline.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:16:49.218265]
2024-12-18T05:17:01.7648402Z 
2024-12-18T05:17:01.7650288Z distributed/checkpoint/e2e/test_pipeline 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_pipeline_1.1_32e11c31b00dfc90_.log
2024-12-18T05:17:01.7652531Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_pipeline.py::TestPipeline::test_pipeline
2024-12-18T05:17:01.7653380Z 
2024-12-18T05:17:01.7655560Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2024-12-18 05:17:01.765043]
2024-12-18T05:17:01.7656696Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:17:01.7663283Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:17:01.765667]
2024-12-18T05:17:14.4119895Z 
2024-12-18T05:17:14.4121857Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_3115d5b8213561ba_.log
2024-12-18T05:17:14.4124380Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner
2024-12-18T05:17:14.4125466Z 
2024-12-18T05:17:14.4126714Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2024-12-18 05:17:14.412199]
2024-12-18T05:17:14.4127646Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:17:14.4133494Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:17:14.412799]
2024-12-18T05:17:39.9408250Z 
2024-12-18T05:17:39.9410123Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_826ec111d74705a2_.log
2024-12-18T05:17:39.9413424Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device
2024-12-18T05:17:39.9415236Z 
2024-12-18T05:17:39.9415917Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-12-18 05:17:39.940811]
2024-12-18T05:17:39.9416991Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:17:39.9419193Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:17:39.941350]
2024-12-18T05:17:45.6702872Z 
2024-12-18T05:17:45.6705303Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_a61e987ed79db039_.log
2024-12-18T05:17:45.6719581Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap
2024-12-18T05:17:45.6722012Z 
2024-12-18T05:17:45.6722532Z Running distributed/checkpoint/test_planner 1/1 ... [2024-12-18 05:17:45.669575]
2024-12-18T05:17:45.6723562Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:17:45.6730460Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:17:45.670117]
2024-12-18T05:17:51.6513277Z 
2024-12-18T05:17:51.6515468Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_1893b97db2277fb9_.log
2024-12-18T05:17:51.6523317Z Running 8 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_load_different_sizes_throws, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_strict
2024-12-18T05:17:51.6528749Z 
2024-12-18T05:17:51.6529469Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2024-12-18 05:17:51.650822]
2024-12-18T05:17:51.6530488Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:17:51.6532550Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:17:51.651378]
2024-12-18T05:18:10.1116670Z 
2024-12-18T05:18:10.1119459Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_cc4d02b6069cc354_.log
2024-12-18T05:18:10.1122828Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp
2024-12-18T05:18:10.1124041Z 
2024-12-18T05:18:10.1124916Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2024-12-18 05:18:10.111929]
2024-12-18T05:18:10.1126169Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:18:10.1134837Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:18:10.112531]
2024-12-18T05:18:51.8757437Z 
2024-12-18T05:18:51.8759148Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_66b63c2b994d5e0d_.log
2024-12-18T05:18:51.8766204Z Running 6 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_create_cpu_state_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_state_dict_util_distribute_tensors
2024-12-18T05:18:51.8771679Z 
2024-12-18T05:18:51.8772162Z Running distributed/fsdp/test_shard_utils 1/1 ... [2024-12-18 05:18:51.876028]
2024-12-18T05:18:51.8773025Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:18:51.8774950Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:18:51.876601]
2024-12-18T05:19:11.1882676Z 
2024-12-18T05:19:11.1884738Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_38875b136b29f813_.log
2024-12-18T05:19:11.1887818Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor
2024-12-18T05:19:11.1889688Z 
2024-12-18T05:19:11.1890236Z Running distributed/checkpoint/test_compatibility 1/1 ... [2024-12-18 05:19:11.188499]
2024-12-18T05:19:11.1891147Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:19:11.1897570Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_compatibility.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:11.189079]
2024-12-18T05:19:17.0199977Z 
2024-12-18T05:19:17.0201624Z distributed/checkpoint/test_compatibility 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_compatibility_1.1_42c5e06b8bcabfbc_.log
2024-12-18T05:19:17.0206030Z Running 4 items in this shard: test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_metadata, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_sharded_tensor_dependency, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_storage_meta, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_with_v_2_3
2024-12-18T05:19:17.0209009Z 
2024-12-18T05:19:17.0209555Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2024-12-18 05:19:17.019955]
2024-12-18T05:19:17.0210465Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:19:17.0212446Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:17.020524]
2024-12-18T05:19:44.4996932Z 
2024-12-18T05:19:44.4998776Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_55562f309f454b97_.log
2024-12-18T05:19:44.5004256Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True
2024-12-18T05:19:44.5008175Z 
2024-12-18T05:19:44.5009537Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-12-18 05:19:44.499858]
2024-12-18T05:19:44.5010492Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:19:44.5012520Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:44.500452]
2024-12-18T05:19:50.6313678Z 
2024-12-18T05:19:50.6315678Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_cc2be410b0328950_.log
2024-12-18T05:19:50.6321494Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls
2024-12-18T05:19:50.6325719Z 
2024-12-18T05:19:50.6326203Z Running distributed/test_control_collectives 1/1 ... [2024-12-18 05:19:50.631706]
2024-12-18T05:19:50.6327075Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:19:50.6329280Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_control_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:50.632389]
2024-12-18T05:19:56.2626557Z 
2024-12-18T05:19:56.2628520Z distributed/test_control_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_control_collectives_1.1_13620c377268e761_.log
2024-12-18T05:19:56.2637416Z Running 13 items in this shard: test/distributed/test_control_collectives.py::TestCollectives::test_all_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_barrier, test/distributed/test_control_collectives.py::TestCollectives::test_barrier_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_gather, test/distributed/test_control_collectives.py::TestCollectives::test_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_scatter, test/distributed/test_control_collectives.py::TestCollectives::test_scatter_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_simple_user_func, test/distributed/test_control_collectives.py::TestCollectives::test_unique
2024-12-18T05:19:56.2645504Z 
2024-12-18T05:19:56.2646011Z Running distributed/_tensor/debug/test_comm_mode 1/1 ... [2024-12-18 05:19:56.262685]
2024-12-18T05:19:56.2646901Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:19:56.2648850Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:56.263312]
2024-12-18T05:20:07.9105234Z 
2024-12-18T05:20:07.9106877Z distributed/_tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_comm_mode_1.1_c55c36c8de5194f4_.log
2024-12-18T05:20:07.9111449Z Running 4 items in this shard: test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_coalesced, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_c10d, test/distributed/_tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor
2024-12-18T05:20:07.9114063Z 
2024-12-18T05:20:07.9114582Z Running distributed/_composable/test_replicate 1/1 ... [2024-12-18 05:20:07.910432]
2024-12-18T05:20:07.9115467Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:20:07.9117424Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:20:07.911049]
2024-12-18T05:22:31.7599263Z 
2024-12-18T05:22:31.7602053Z distributed/_composable/test_replicate 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_1.1_4d1818ad5b598a3d_.log
2024-12-18T05:22:31.7617438Z Running 17 items in this shard: test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_non_root_multiple_save_load, test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_single_module_save_load, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_wrong_device_id_type, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_fully_shard_init, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_wrong_device_id_type
2024-12-18T05:22:31.7630477Z 
2024-12-18T05:22:31.7630988Z Running distributed/fsdp/test_fsdp_state_dict 3/3 ... [2024-12-18 05:22:31.760026]
2024-12-18T05:22:31.7631876Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:22:31.7633805Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:22:31.760713]
2024-12-18T05:32:13.2397203Z 
2024-12-18T05:32:13.2399749Z distributed/fsdp/test_fsdp_state_dict 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_3.3_b7394df93be5e8de_.log
2024-12-18T05:32:13.2508060Z Running 68 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config
2024-12-18T05:32:13.2597425Z 
2024-12-18T05:32:13.2597859Z Running distributed/test_c10d_nccl 1/3 ... [2024-12-18 05:32:13.241592]
2024-12-18T05:32:13.2598664Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:32:13.2600582Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:32:13.242149]
2024-12-18T05:50:34.1007419Z 
2024-12-18T05:50:34.1009941Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_ea5cde5b5a6ea9e4_.log
2024-12-18T05:50:34.1067315Z Running 82 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2024-12-18T05:50:34.1125427Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors
2024-12-18T05:50:34.1126901Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus
2024-12-18T05:50:34.1128713Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs
2024-12-18T05:50:34.1130847Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg
2024-12-18T05:50:34.1132389Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered
2024-12-18T05:50:34.1134024Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False
2024-12-18T05:50:34.1135633Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True
2024-12-18T05:50:34.1137183Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group
2024-12-18T05:50:34.1138834Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl
2024-12-18T05:50:34.1140469Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check
2024-12-18T05:50:34.1142089Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout
2024-12-18T05:50:34.1143718Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16
2024-12-18T05:50:34.1145280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2
2024-12-18T05:50:34.1147041Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True
2024-12-18T05:50:34.1148613Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init
2024-12-18T05:50:34.1150218Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module
2024-12-18T05:50:34.1151927Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view
2024-12-18T05:50:34.1153540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output
2024-12-18T05:50:34.1155592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing
2024-12-18T05:50:34.1157495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False
2024-12-18T05:50:34.1159825Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2024-12-18T05:50:34.1162174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl
2024-12-18T05:50:34.1163933Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params
2024-12-18T05:50:34.1165552Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters
2024-12-18T05:50:34.1167327Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view
2024-12-18T05:50:34.1169526Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info
2024-12-18T05:50:34.1171566Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off
2024-12-18T05:50:34.1173517Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail
2024-12-18T05:50:34.1175544Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off
2024-12-18T05:50:34.1177559Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16
2024-12-18T05:50:34.1179128Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl
2024-12-18T05:50:34.1180735Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view
2024-12-18T05:50:34.1182474Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess
2024-12-18T05:50:34.1184266Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module
2024-12-18T05:50:34.1185926Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module
2024-12-18T05:50:34.1187845Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None
2024-12-18T05:50:34.1189838Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids
2024-12-18T05:50:34.1191685Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason
2024-12-18T05:50:34.1193239Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad
2024-12-18T05:50:34.1194792Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error
2024-12-18T05:50:34.1196468Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl
2024-12-18T05:50:34.1198088Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object
2024-12-18T05:50:34.1199601Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast
2024-12-18T05:50:34.1201433Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq
2024-12-18T05:50:34.1202849Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp
2024-12-18T05:50:34.1204283Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result
2024-12-18T05:50:34.1205834Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env
2024-12-18T05:50:34.1207481Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit
2024-12-18T05:50:34.1209109Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm
2024-12-18T05:50:34.1210760Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier
2024-12-18T05:50:34.1212385Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors
2024-12-18T05:50:34.1213831Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config
2024-12-18T05:50:34.1215235Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced
2024-12-18T05:50:34.1216706Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default
2024-12-18T05:50:34.1218213Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup
2024-12-18T05:50:34.1219686Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group
2024-12-18T05:50:34.1221338Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor
2024-12-18T05:50:34.1222927Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base
2024-12-18T05:50:34.1224914Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2
2024-12-18T05:50:34.1227362Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2024-12-18T05:50:34.1229287Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True
2024-12-18T05:50:34.1231045Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False
2024-12-18T05:50:34.1232923Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True
2024-12-18T05:50:34.1234617Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True
2024-12-18T05:50:34.1236121Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False
2024-12-18T05:50:34.1237722Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False
2024-12-18T05:50:34.1239252Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True
2024-12-18T05:50:34.1240741Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False
2024-12-18T05:50:34.1242378Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False
2024-12-18T05:50:34.1243993Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False
2024-12-18T05:50:34.1245872Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False
2024-12-18T05:50:34.1247278Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata
2024-12-18T05:50:34.1248475Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe
2024-12-18T05:50:34.1249811Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True
2024-12-18T05:50:34.1251117Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long
2024-12-18T05:50:34.1252480Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False
2024-12-18T05:50:34.1254132Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False
2024-12-18T05:50:34.1255768Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True
2024-12-18T05:50:34.1257394Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False
2024-12-18T05:50:34.1258997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True
2024-12-18T05:50:34.1260451Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired
2024-12-18T05:50:34.1262182Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2024-12-18T05:50:34.1263088Z 
2024-12-18T05:50:34.1263501Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2024-12-18 05:50:34.103314]
2024-12-18T05:50:34.1264239Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:50:34.1265914Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:50:34.103863]
2024-12-18T05:51:36.5540611Z 
2024-12-18T05:51:36.5542290Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_99d461af1f5f2687_.log
2024-12-18T05:51:36.5545385Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeTwo::test_forward_overlap, test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOneCUDA::test_forward_overlap_cuda
2024-12-18T05:51:36.5547838Z 
2024-12-18T05:51:36.5548322Z Running distributed/fsdp/test_fsdp_optim_state 2/2 ... [2024-12-18 05:51:36.554182]
2024-12-18T05:51:36.5549257Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:51:36.5553393Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:51:36.554784]
2024-12-18T05:57:15.6662254Z 
2024-12-18T05:57:15.6664121Z distributed/fsdp/test_fsdp_optim_state 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_2.2_a78d7a5f72ac2d4b_.log
2024-12-18T05:57:15.6683733Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params
2024-12-18T05:57:15.6704731Z 
2024-12-18T05:57:15.6705253Z Running distributed/nn/jit/test_instantiator 1/1 ... [2024-12-18 05:57:15.666922]
2024-12-18T05:57:15.6706130Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:57:15.6708646Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:57:15.667468]
2024-12-18T05:57:21.2477697Z 
2024-12-18T05:57:21.2479400Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_990f92e64ea4f623_.log
2024-12-18T05:57:21.2483479Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template
2024-12-18T05:57:21.2486127Z 
2024-12-18T05:57:21.2486680Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2024-12-18 05:57:21.247496]
2024-12-18T05:57:21.2487611Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:57:21.2489588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:57:21.248060]
2024-12-18T05:57:39.8099240Z 
2024-12-18T05:57:39.8101290Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_ab99644de447c71b_.log
2024-12-18T05:57:39.8103701Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch
2024-12-18T05:57:39.8105655Z 
2024-12-18T05:57:39.8106139Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2024-12-18 05:57:39.809144]
2024-12-18T05:57:39.8107195Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:57:39.8109143Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:57:39.809754]
2024-12-18T05:58:11.5027436Z 
2024-12-18T05:58:11.5029198Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_e4791377985c4016_.log
2024-12-18T05:58:11.5032847Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_fp16_dtypes_cuda, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_pure_fp16_training_cuda
2024-12-18T05:58:11.5034529Z 
2024-12-18T05:58:11.5035677Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2024-12-18 05:58:11.503059]
2024-12-18T05:58:11.5036608Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:58:11.5042533Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:58:11.503654]
2024-12-18T05:58:59.9288306Z 
2024-12-18T05:58:59.9289665Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_19f4ebcd9986d7a7_.log
2024-12-18T05:58:59.9291792Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt
2024-12-18T05:58:59.9292933Z 
2024-12-18T05:58:59.9293454Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2024-12-18 05:58:59.928746]
2024-12-18T05:58:59.9294995Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:58:59.9300917Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:58:59.929337]
2024-12-18T05:59:41.1445475Z 
2024-12-18T05:59:41.1447400Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_77b59d7825b8bdc1_.log
2024-12-18T05:59:41.1454727Z Running 6 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor
2024-12-18T05:59:41.1460518Z 
2024-12-18T05:59:41.1460988Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2024-12-18 05:59:41.144888]
2024-12-18T05:59:41.1461838Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T05:59:41.1464530Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:59:41.145503]
2024-12-18T06:00:09.0269505Z 
2024-12-18T06:00:09.0271127Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_0db9726ed0dc7d6b_.log
2024-12-18T06:00:09.0274472Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_apply_in_summon_raises_error_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_nested_module_apply_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_transformer_module_apply_cuda
2024-12-18T06:00:09.0276630Z 
2024-12-18T06:00:09.0278223Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2024-12-18 06:00:09.027365]
2024-12-18T06:00:09.0279137Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:00:09.0286515Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:00:09.027963]
2024-12-18T06:01:57.0962464Z 
2024-12-18T06:01:57.0964300Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_1860cab972ae4d1a_.log
2024-12-18T06:01:57.0975735Z Running 13 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format0, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format1, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params
2024-12-18T06:01:57.0987276Z 
2024-12-18T06:01:57.0987900Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2024-12-18 06:01:57.096330]
2024-12-18T06:01:57.0988864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:01:57.0990867Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:01:57.096924]
2024-12-18T06:06:31.9167719Z 
2024-12-18T06:06:31.9174467Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_4462c1a15e31e0b9_.log
2024-12-18T06:06:31.9227874Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True
2024-12-18T06:06:31.9278860Z 
2024-12-18T06:06:31.9279342Z Running distributed/test_multi_threaded_pg 1/1 ... [2024-12-18 06:06:31.917254]
2024-12-18T06:06:31.9280197Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:06:31.9282084Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:06:31.917894]
2024-12-18T06:06:38.2498633Z 
2024-12-18T06:06:38.2500234Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_583d66d95fabba57_.log
2024-12-18T06:06:38.2517764Z Running 22 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_none, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_tensor, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread
2024-12-18T06:06:38.2533218Z 
2024-12-18T06:06:38.2533670Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2024-12-18 06:06:38.249802]
2024-12-18T06:06:38.2534531Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:06:38.2536424Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:06:38.250354]
2024-12-18T06:06:57.7140722Z 
2024-12-18T06:06:57.7142751Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_b64fa1292448eb2d_.log
2024-12-18T06:06:57.7144893Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda
2024-12-18T06:06:57.7145816Z 
2024-12-18T06:06:57.7146256Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2024-12-18 06:06:57.713605]
2024-12-18T06:06:57.7147277Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:06:57.7149158Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:06:57.714195]
2024-12-18T06:07:28.7553714Z 
2024-12-18T06:07:28.7555267Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_20e87d171e10826a_.log
2024-12-18T06:07:28.7557905Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda, test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2024-12-18T06:07:28.7559329Z 
2024-12-18T06:07:28.7560040Z Running distributed/test_c10d_ucc 1/1 ... [2024-12-18 06:07:28.755614]
2024-12-18T06:07:28.7560822Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:07:28.7568683Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:07:28.756212]
2024-12-18T06:07:31.7683975Z 
2024-12-18T06:07:31.7686393Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_6dde9664adeca665_.log
2024-12-18T06:07:31.7687667Z 
2024-12-18T06:07:31.7692983Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2024-12-18 06:07:31.768690]
2024-12-18T06:07:31.7696735Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:07:31.7698903Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:07:31.769213]
2024-12-18T06:09:01.7014630Z 
2024-12-18T06:09:01.7016509Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_fa112a30850403f2_.log
2024-12-18T06:09:01.7027108Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check
2024-12-18T06:09:01.7034239Z 
2024-12-18T06:09:01.7034968Z Running distributed/_composable/test_composability/test_2d_composability 1/1 ... [2024-12-18 06:09:01.701712]
2024-12-18T06:09:01.7036054Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:09:01.7038949Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:09:01.702334]
2024-12-18T06:10:50.7691183Z 
2024-12-18T06:10:50.7693269Z distributed/_composable/test_composability/test_2d_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_1.1_7775d95b75730559_.log
2024-12-18T06:10:50.7716332Z Running 20 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_2d_uneven_shard_raise_error, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DStateDict::test_fully_shard_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::Test2dFSDP1ParallelIntegration::test_2d_ddp_integration_functionality, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_raise_invalid_tp_composition, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp1_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp_2d_extension
2024-12-18T06:10:50.7737722Z 
2024-12-18T06:10:50.7738136Z Running distributed/test_pg_wrapper 1/1 ... [2024-12-18 06:10:50.769056]
2024-12-18T06:10:50.7738943Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:10:50.7740898Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:10:50.769649]
2024-12-18T06:14:16.5733700Z 
2024-12-18T06:14:16.5734856Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_fd41e2510b139fd9_.log
2024-12-18T06:14:16.5751417Z Running 17 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode
2024-12-18T06:14:16.5764285Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail
2024-12-18T06:14:16.5765372Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang
2024-12-18T06:14:16.5766448Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail
2024-12-18T06:14:16.5767637Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off
2024-12-18T06:14:16.5768724Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch
2024-12-18T06:14:16.5769797Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode
2024-12-18T06:14:16.5770879Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo
2024-12-18T06:14:16.5771877Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo
2024-12-18T06:14:16.5772822Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang
2024-12-18T06:14:16.5773835Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda
2024-12-18T06:14:16.5774951Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode
2024-12-18T06:14:16.5776096Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode
2024-12-18T06:14:16.5777234Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off
2024-12-18T06:14:16.5778586Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch
2024-12-18T06:14:16.5779612Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda
2024-12-18T06:14:16.5780745Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode
2024-12-18T06:14:16.5781875Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode
2024-12-18T06:14:16.5782497Z 
2024-12-18T06:14:16.5782797Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2024-12-18 06:14:16.574045]
2024-12-18T06:14:16.5783331Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:14:16.5784496Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:14:16.574637]
2024-12-18T06:15:22.1974440Z 
2024-12-18T06:15:22.1980463Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_9eaf62d24972eee0_.log
2024-12-18T06:15:22.1987305Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy
2024-12-18T06:15:22.1992185Z 
2024-12-18T06:15:22.1992944Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2024-12-18 06:15:22.197801]
2024-12-18T06:15:22.1994126Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:15:22.1996573Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:15:22.198402]
2024-12-18T06:16:24.3599039Z 
2024-12-18T06:16:24.3601208Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_200ae0c109d966aa_.log
2024-12-18T06:16:24.3616673Z Running 12 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2
2024-12-18T06:16:24.3630843Z 
2024-12-18T06:16:24.3631289Z Running distributed/test_data_parallel 1/1 ... [2024-12-18 06:16:24.360472]
2024-12-18T06:16:24.3632132Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:16:24.3634019Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:16:24.361086]
2024-12-18T06:16:46.7316761Z 
2024-12-18T06:16:46.7322858Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_c31892192fe3784f_.log
2024-12-18T06:16:46.7357411Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64
2024-12-18T06:16:46.7390352Z 
2024-12-18T06:16:46.7390793Z Running distributed/fsdp/test_fsdp_core 1/3 ... [2024-12-18 06:16:46.732128]
2024-12-18T06:16:46.7391638Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:16:46.7393543Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:16:46.732770]
2024-12-18T06:25:16.9032119Z 
2024-12-18T06:25:16.9033565Z distributed/fsdp/test_fsdp_core 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.3_7ce471b0543491f5_.log
2024-12-18T06:25:16.9044880Z Running 19 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_after_state_dict_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_False_cuda
2024-12-18T06:25:16.9056063Z 
2024-12-18T06:25:16.9056443Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2024-12-18 06:25:16.903458]
2024-12-18T06:25:16.9057079Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:25:16.9058388Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:25:16.903857]
2024-12-18T06:25:41.6861461Z 
2024-12-18T06:25:41.6863189Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_fd5f30d2335df409_.log
2024-12-18T06:25:41.6868044Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors
2024-12-18T06:25:41.6870455Z 
2024-12-18T06:25:41.6870920Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2024-12-18 06:25:41.686389]
2024-12-18T06:25:41.6871723Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:25:41.6876177Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:25:41.687037]
2024-12-18T06:25:47.3674897Z 
2024-12-18T06:25:47.3676794Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_98ae0e4821d8a094_.log
2024-12-18T06:25:47.3678489Z Running 0 items in this shard:
2024-12-18T06:25:47.3678844Z 
2024-12-18T06:25:47.3687271Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2024-12-18 06:25:47.367514]
2024-12-18T06:25:47.3688299Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:25:47.3690391Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:25:47.368113]
2024-12-18T06:29:50.5275096Z 
2024-12-18T06:29:50.5277417Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_06fc35097033ea5c_.log
2024-12-18T06:29:50.5306081Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf
2024-12-18T06:29:50.5323701Z 
2024-12-18T06:29:50.5324207Z Running distributed/_shard/test_sharder 1/1 ... [2024-12-18 06:29:50.527747]
2024-12-18T06:29:50.5325054Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:29:50.5326942Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:29:50.528379]
2024-12-18T06:30:07.1335854Z 
2024-12-18T06:30:07.1337388Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_225e8d078d88c2f7_.log
2024-12-18T06:30:07.1340275Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors
2024-12-18T06:30:07.1343104Z 
2024-12-18T06:30:07.1343592Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2024-12-18 06:30:07.133549]
2024-12-18T06:30:07.1344442Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:30:07.1347082Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:30:07.134197]
2024-12-18T06:32:26.1861196Z 
2024-12-18T06:32:26.1863280Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_4b47b21aae6e0ad9_.log
2024-12-18T06:32:26.1875130Z Running 10 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda
2024-12-18T06:32:26.1885840Z 
2024-12-18T06:32:26.1886563Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2024-12-18 06:32:26.185173]
2024-12-18T06:32:26.1887502Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:32:26.1889476Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:32:26.185770]
2024-12-18T06:32:47.0523825Z 
2024-12-18T06:32:47.0525828Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_7e68bd5d5f0334ff_.log
2024-12-18T06:32:47.0529549Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda
2024-12-18T06:32:47.0531993Z 
2024-12-18T06:32:47.0532503Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2024-12-18 06:32:47.052728]
2024-12-18T06:32:47.0533410Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:32:47.0539502Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:32:47.053311]
2024-12-18T06:40:51.7774901Z 
2024-12-18T06:40:51.7777656Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_09b3cbc767959d19_.log
2024-12-18T06:40:51.7805274Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda
2024-12-18T06:40:51.7830089Z 
2024-12-18T06:40:51.7830573Z Running distributed/test_c10d_gloo 2/2 ... [2024-12-18 06:40:51.778206]
2024-12-18T06:40:51.7831410Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2024-12-18T06:40:51.7833452Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:40:51.778818]
2024-12-18T06:51:59.0898906Z 
2024-12-18T06:51:59.0900829Z distributed/test_c10d_gloo 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_2.2_7d8a4275f972531e_.log
2024-12-18T06:51:59.0941859Z Running 57 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2024-12-18T06:51:59.0980505Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init
2024-12-18T06:51:59.0981932Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics
2024-12-18T06:51:59.0983402Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda
2024-12-18T06:51:59.0985191Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks
2024-12-18T06:51:59.0986965Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode
2024-12-18T06:51:59.0988558Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input
2024-12-18T06:51:59.0990275Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async
2024-12-18T06:51:59.0991998Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks
2024-12-18T06:51:59.0993539Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress
2024-12-18T06:51:59.0995078Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda
2024-12-18T06:51:59.0996559Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks
2024-12-18T06:51:59.0997982Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics
2024-12-18T06:51:59.0999443Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda
2024-12-18T06:51:59.1001116Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks
2024-12-18T06:51:59.1002874Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input
2024-12-18T06:51:59.1004352Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress
2024-12-18T06:51:59.1005760Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda
2024-12-18T06:51:59.1007162Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks
2024-12-18T06:51:59.1008607Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor
2024-12-18T06:51:59.1010748Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced
2024-12-18T06:51:59.1012384Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks
2024-12-18T06:51:59.1013806Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda
2024-12-18T06:51:59.1015280Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks
2024-12-18T06:51:59.1016854Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched
2024-12-18T06:51:59.1018546Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param
2024-12-18T06:51:59.1020482Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False
2024-12-18T06:51:59.1022472Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing
2024-12-18T06:51:59.1024426Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2024-12-18T06:51:59.1026490Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False
2024-12-18T06:51:59.1028500Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once
2024-12-18T06:51:59.1030554Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init
2024-12-18T06:51:59.1032294Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type
2024-12-18T06:51:59.1034199Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty
2024-12-18T06:51:59.1036204Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph
2024-12-18T06:51:59.1038176Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list
2024-12-18T06:51:59.1040310Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list
2024-12-18T06:51:59.1042506Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module
2024-12-18T06:51:59.1044210Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module
2024-12-18T06:51:59.1045949Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters
2024-12-18T06:51:59.1047679Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state
2024-12-18T06:51:59.1049280Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint
2024-12-18T06:51:59.1051234Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input
2024-12-18T06:51:59.1052890Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer
2024-12-18T06:51:59.1054372Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters
2024-12-18T06:51:59.1056168Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket
2024-12-18T06:51:59.1057465Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors
2024-12-18T06:51:59.1058745Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group
2024-12-18T06:51:59.1060151Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default
2024-12-18T06:51:59.1061643Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup
2024-12-18T06:51:59.1063066Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch
2024-12-18T06:51:59.1064761Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single
2024-12-18T06:51:59.1067105Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced
2024-12-18T06:51:59.1069051Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives
2024-12-18T06:51:59.1070985Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2024-12-18T06:51:59.1072800Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier
2024-12-18T06:51:59.1074261Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync
2024-12-18T06:51:59.1075887Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2024-12-18T06:51:59.1076674Z 
2024-12-18T06:52:00.2527760Z Running test batch 'tests to run' cost 11675.52 seconds
2024-12-18T06:52:01.0707353Z 
2024-12-18T06:52:01.0708001Z real	194m42.242s
2024-12-18T06:52:01.0708592Z user	664m4.816s
2024-12-18T06:52:01.0709091Z sys	365m31.507s
2024-12-18T06:52:01.0709523Z + assert_git_not_dirty
2024-12-18T06:52:01.0710073Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]]
2024-12-18T06:52:01.0710734Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]]
2024-12-18T06:52:01.0711457Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]]
2024-12-18T06:52:01.0712135Z + [[ 3 == 1 ]]
2024-12-18T06:52:01.0712629Z + [[ 3 == 1 ]]
2024-12-18T06:52:01.0894574Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2024-12-18T06:52:01.0896173Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2024-12-18T06:52:01.0897969Z docker exec -t "ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2024-12-18T06:52:01.0957877Z shell: /usr/bin/bash -e {0}
2024-12-18T06:52:01.0958496Z env:
2024-12-18T06:52:01.0958964Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:01.0959668Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:01.0960779Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:01.0961470Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:01.0961855Z   AWS_REGION: us-east-1
2024-12-18T06:52:01.0962235Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:01.0962700Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:01.0970995Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:01.0971796Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:01.0972645Z ##[endgroup]
2024-12-18T06:52:01.2198019Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-12-18T06:52:01.3022939Z ##[group]Run cat test/**/*_toprint.log || true
2024-12-18T06:52:01.3023851Z cat test/**/*_toprint.log || true
2024-12-18T06:52:01.3073520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T06:52:01.3074233Z env:
2024-12-18T06:52:01.3074655Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:01.3075358Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:01.3076664Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:01.3077814Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:01.3078427Z   AWS_REGION: us-east-1
2024-12-18T06:52:01.3079084Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:01.3079578Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:01.3086928Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:01.3087857Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:01.3088716Z ##[endgroup]
2024-12-18T06:52:01.3489728Z cat: 'test/**/*_toprint.log': No such file or directory
2024-12-18T06:52:01.3726841Z Prepare all required actions
2024-12-18T06:52:01.3727608Z Getting action download info
2024-12-18T06:52:01.6786936Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-12-18T06:52:02.2564214Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-12-18T06:52:02.2564489Z with:
2024-12-18T06:52:02.2564657Z   use-gha: true
2024-12-18T06:52:02.2564919Z   file-suffix: test-distributed-3-3-linux.rocm.gpu_34566687343
2024-12-18T06:52:02.2565234Z   s3-bucket: gha-artifacts
2024-12-18T06:52:02.2565436Z env:
2024-12-18T06:52:02.2565612Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:02.2565846Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:02.2566258Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:02.2566644Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:02.2566871Z   AWS_REGION: us-east-1
2024-12-18T06:52:02.2567172Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:02.2567462Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:02.2571930Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:02.2572263Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:02.2572616Z ##[endgroup]
2024-12-18T06:52:02.2633048Z ##[group]Run actions/upload-artifact@v4
2024-12-18T06:52:02.2633290Z with:
2024-12-18T06:52:02.2633610Z   name: test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip
2024-12-18T06:52:02.2634009Z   retention-days: 14
2024-12-18T06:52:02.2634214Z   if-no-files-found: warn
2024-12-18T06:52:02.2634422Z   path: test/**/*.json
2024-12-18T06:52:02.2634621Z   compression-level: 6
2024-12-18T06:52:02.2634817Z   overwrite: false
2024-12-18T06:52:02.2635014Z   include-hidden-files: false
2024-12-18T06:52:02.2635223Z env:
2024-12-18T06:52:02.2635397Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:02.2635644Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:02.2636077Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:02.2636475Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:02.2636696Z   AWS_REGION: us-east-1
2024-12-18T06:52:02.2636934Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:02.2637222Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:02.2641689Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:02.2642023Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:02.2642400Z ##[endgroup]
2024-12-18T06:52:02.9897049Z With the provided path, there will be 6 files uploaded
2024-12-18T06:52:02.9903055Z Artifact name is valid!
2024-12-18T06:52:02.9903653Z Root directory input is valid!
2024-12-18T06:52:03.1599853Z Beginning upload of artifact content to blob storage
2024-12-18T06:52:03.4003732Z Uploaded bytes 44674
2024-12-18T06:52:03.4434314Z Finished uploading artifact content to blob storage!
2024-12-18T06:52:03.4435707Z SHA256 hash of uploaded artifact zip is 20a843f3a3200cb3894ba9414da539efe0789c8787fae9cc486de89a20c82b14
2024-12-18T06:52:03.4437138Z Finalizing artifact upload
2024-12-18T06:52:03.5453894Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip.zip successfully finalized. Artifact ID 2336049747
2024-12-18T06:52:03.5456139Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip has been successfully uploaded! Final size is 44674 bytes. Artifact ID is 2336049747
2024-12-18T06:52:03.5471262Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336049747
2024-12-18T06:52:03.5793078Z ##[group]Run actions/upload-artifact@v4
2024-12-18T06:52:03.5793345Z with:
2024-12-18T06:52:03.5793689Z   name: test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip
2024-12-18T06:52:03.5794089Z   retention-days: 14
2024-12-18T06:52:03.5794296Z   if-no-files-found: ignore
2024-12-18T06:52:03.5794523Z   path: test/**/*.xml
test/**/*.csv

2024-12-18T06:52:03.5795046Z   compression-level: 6
2024-12-18T06:52:03.5795267Z   overwrite: false
2024-12-18T06:52:03.5795469Z   include-hidden-files: false
2024-12-18T06:52:03.5795708Z env:
2024-12-18T06:52:03.5795886Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:03.5796134Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:03.5796570Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:03.5796961Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:03.5797178Z   AWS_REGION: us-east-1
2024-12-18T06:52:03.5797448Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:03.5797736Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:03.5802167Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:03.5802504Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:03.5802867Z ##[endgroup]
2024-12-18T06:52:04.3834157Z With the provided path, there will be 619 files uploaded
2024-12-18T06:52:04.3838778Z Artifact name is valid!
2024-12-18T06:52:04.3839458Z Root directory input is valid!
2024-12-18T06:52:04.5507882Z Beginning upload of artifact content to blob storage
2024-12-18T06:52:05.1533361Z Uploaded bytes 412508
2024-12-18T06:52:05.2013092Z Finished uploading artifact content to blob storage!
2024-12-18T06:52:05.2018537Z SHA256 hash of uploaded artifact zip is 515a3715e6f19815a7124e4a16be3b7ac4181dc3c4fec1dedc9f0492de254424
2024-12-18T06:52:05.2021313Z Finalizing artifact upload
2024-12-18T06:52:05.2988743Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip.zip successfully finalized. Artifact ID 2336049823
2024-12-18T06:52:05.2991103Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip has been successfully uploaded! Final size is 412508 bytes. Artifact ID is 2336049823
2024-12-18T06:52:05.3001973Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336049823
2024-12-18T06:52:05.3423733Z ##[group]Run actions/upload-artifact@v4
2024-12-18T06:52:05.3424357Z with:
2024-12-18T06:52:05.3425060Z   name: logs-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip
2024-12-18T06:52:05.3425944Z   retention-days: 14
2024-12-18T06:52:05.3426417Z   if-no-files-found: ignore
2024-12-18T06:52:05.3427115Z   path: usage_log.txt
test/**/*.log

2024-12-18T06:52:05.3427691Z   compression-level: 6
2024-12-18T06:52:05.3428181Z   overwrite: false
2024-12-18T06:52:05.3428640Z   include-hidden-files: false
2024-12-18T06:52:05.3429134Z env:
2024-12-18T06:52:05.3429526Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:05.3430122Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:05.3431326Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:05.3432402Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:05.3432919Z   AWS_REGION: us-east-1
2024-12-18T06:52:05.3433513Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:05.3434198Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:05.3444243Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:05.3444725Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:05.3445569Z ##[endgroup]
2024-12-18T06:52:06.1210577Z Multiple search paths detected. Calculating the least common ancestor of all paths
2024-12-18T06:52:06.1212772Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2024-12-18T06:52:06.1214134Z With the provided path, there will be 102 files uploaded
2024-12-18T06:52:06.1216946Z Artifact name is valid!
2024-12-18T06:52:06.1218053Z Root directory input is valid!
2024-12-18T06:52:06.2753304Z Beginning upload of artifact content to blob storage
2024-12-18T06:52:06.6447572Z Uploaded bytes 185028
2024-12-18T06:52:06.6922330Z Finished uploading artifact content to blob storage!
2024-12-18T06:52:06.6928012Z SHA256 hash of uploaded artifact zip is 2e4eced7ee5cd3204dc6e1ae3dea7107ff4e10327560a0ff439680dd436afe4a
2024-12-18T06:52:06.6931828Z Finalizing artifact upload
2024-12-18T06:52:06.7921905Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip.zip successfully finalized. Artifact ID 2336049899
2024-12-18T06:52:06.7924096Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu_34566687343.zip has been successfully uploaded! Final size is 185028 bytes. Artifact ID is 2336049899
2024-12-18T06:52:06.7936915Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336049899
2024-12-18T06:52:06.8310726Z ##[group]Run # shellcheck disable=SC2156
2024-12-18T06:52:06.8311500Z # shellcheck disable=SC2156
2024-12-18T06:52:06.8312530Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-12-18T06:52:06.8367432Z shell: /usr/bin/bash -e {0}
2024-12-18T06:52:06.8367957Z env:
2024-12-18T06:52:06.8368363Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:06.8368971Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:06.8370002Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:06.8370995Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:06.8371517Z   AWS_REGION: us-east-1
2024-12-18T06:52:06.8372096Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:06.8372770Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:06.8383642Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:06.8384490Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:06.8385339Z ##[endgroup]
2024-12-18T06:52:08.9748862Z Prepare all required actions
2024-12-18T06:52:08.9750378Z Getting action download info
2024-12-18T06:52:08.9821947Z ##[group]Run ./.github/actions/teardown-rocm
2024-12-18T06:52:08.9822869Z env:
2024-12-18T06:52:08.9823717Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:08.9824640Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:08.9826244Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:08.9827738Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:08.9828578Z   AWS_REGION: us-east-1
2024-12-18T06:52:08.9829599Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:08.9830634Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:08.9842353Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:08.9843670Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:08.9844824Z ##[endgroup]
2024-12-18T06:52:08.9887240Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T06:52:08.9888459Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T06:52:08.9889381Z # shellcheck disable=SC2046
2024-12-18T06:52:08.9890101Z docker stop $(docker ps -q) || true
2024-12-18T06:52:08.9891058Z # Prune all stopped containers.
2024-12-18T06:52:08.9891801Z docker container prune -f
2024-12-18T06:52:08.9943036Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T06:52:08.9943847Z env:
2024-12-18T06:52:08.9945159Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:08.9945903Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:08.9947372Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:08.9948479Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:08.9949190Z   AWS_REGION: us-east-1
2024-12-18T06:52:08.9950024Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:08.9950849Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:08.9960838Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:08.9961813Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:08.9962825Z ##[endgroup]
2024-12-18T06:52:09.8944350Z ce9b65b64934
2024-12-18T06:52:15.6712298Z Deleted Containers:
2024-12-18T06:52:15.6713138Z ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:15.6713768Z 
2024-12-18T06:52:15.6714006Z Total reclaimed space: 7.173GB
2024-12-18T06:52:15.6824330Z Prepare all required actions
2024-12-18T06:52:15.6880650Z ##[group]Run ./.github/actions/diskspace-cleanup
2024-12-18T06:52:15.6881275Z with:
2024-12-18T06:52:15.6881696Z   diskspace-cutoff: 70
2024-12-18T06:52:15.6882146Z env:
2024-12-18T06:52:15.6882545Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:15.6883114Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:15.6884157Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:15.6885125Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:15.6885648Z   AWS_REGION: us-east-1
2024-12-18T06:52:15.6886227Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:15.6886908Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:15.6897856Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:15.6898674Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:15.6899536Z ##[endgroup]
2024-12-18T06:52:15.6930536Z ##[group]Run set -ex
2024-12-18T06:52:15.6931094Z set -ex
2024-12-18T06:52:15.6931547Z diskspace_cutoff=70
2024-12-18T06:52:15.6932253Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2024-12-18T06:52:15.6933333Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2024-12-18T06:52:15.6934959Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2024-12-18T06:52:15.6936834Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2024-12-18T06:52:15.6937572Z     docker system prune -af
2024-12-18T06:52:15.6938495Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2024-12-18T06:52:15.6939511Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2024-12-18T06:52:15.6940594Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2024-12-18T06:52:15.6941550Z         echo "$msg"
2024-12-18T06:52:15.6942051Z         exit 1
2024-12-18T06:52:15.6942502Z     else
2024-12-18T06:52:15.6943027Z         difference=$((diskspace - diskspace_new))
2024-12-18T06:52:15.6943760Z         echo "Diskspace saved: $difference percent"
2024-12-18T06:52:15.6944377Z     fi
2024-12-18T06:52:15.6944774Z fi
2024-12-18T06:52:15.6999023Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T06:52:15.6999659Z env:
2024-12-18T06:52:15.7000021Z   GIT_DEFAULT_BRANCH: main
2024-12-18T06:52:15.7000552Z   DOCKER_HOST: unix:///run/user/1307800118/docker.sock
2024-12-18T06:52:15.7001449Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon
2024-12-18T06:52:15.7002277Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T06:52:15.7002734Z   AWS_REGION: us-east-1
2024-12-18T06:52:15.7003225Z   AWS_ACCESS_KEY_ID: ***
2024-12-18T06:52:15.7003835Z   AWS_SECRET_ACCESS_KEY: ***
2024-12-18T06:52:15.7013899Z   AWS_SESSION_TOKEN: ***
2024-12-18T06:52:15.7014597Z   CONTAINER_NAME: ce9b65b64934a4e20f84a53cfd5b7fc3e855be2cb79e0e4591a2a60746dffbea
2024-12-18T06:52:15.7015345Z ##[endgroup]
2024-12-18T06:52:15.7104174Z + diskspace_cutoff=70
2024-12-18T06:52:15.7114284Z ++ docker info -f '{{.DockerRootDir}}'
2024-12-18T06:52:15.8161921Z + docker_root_dir=/media/4TB/docker-rootless
2024-12-18T06:52:15.8177498Z ++ df -H --output=pcent /media/4TB/docker-rootless
2024-12-18T06:52:15.8180642Z ++ sed -n 2p
2024-12-18T06:52:15.8184269Z ++ sed s/%//
2024-12-18T06:52:15.8187101Z ++ sed 's/ //'
2024-12-18T06:52:15.8234066Z + diskspace=31
2024-12-18T06:52:15.8235308Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2024-12-18T06:52:15.8236557Z + [[ 31 -ge 70 ]]
2024-12-18T06:52:15.8294399Z Post job cleanup.
2024-12-18T06:52:15.8326529Z Post job cleanup.
2024-12-18T06:52:15.8987961Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T06:52:15.9554798Z Post job cleanup.
2024-12-18T06:52:16.1014903Z Post job cleanup.
2024-12-18T06:52:16.1104682Z Post job cleanup.
2024-12-18T06:52:16.1172332Z Post job cleanup.
2024-12-18T06:52:16.2230989Z [command]/usr/bin/git version
2024-12-18T06:52:16.2283090Z git version 2.34.1
2024-12-18T06:52:16.2322506Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/95cfd387-2f93-4c1e-a0fc-91110703cbb5' before making global git config changes
2024-12-18T06:52:16.2324342Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T06:52:16.2325755Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2024-12-18T06:52:16.2393949Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T06:52:16.2444496Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T06:52:16.3116713Z Entering 'android/libs/fbjni'
2024-12-18T06:52:16.3246545Z Entering 'third_party/FP16'
2024-12-18T06:52:16.3376181Z Entering 'third_party/FXdiv'
2024-12-18T06:52:16.3498775Z Entering 'third_party/NNPACK'
2024-12-18T06:52:16.3627492Z Entering 'third_party/NVTX'
2024-12-18T06:52:16.3748518Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T06:52:16.3868072Z Entering 'third_party/XNNPACK'
2024-12-18T06:52:16.4028562Z Entering 'third_party/benchmark'
2024-12-18T06:52:16.4148622Z Entering 'third_party/composable_kernel'
2024-12-18T06:52:16.4280404Z Entering 'third_party/cpp-httplib'
2024-12-18T06:52:16.4401007Z Entering 'third_party/cpuinfo'
2024-12-18T06:52:16.4526011Z Entering 'third_party/cudnn_frontend'
2024-12-18T06:52:16.4650012Z Entering 'third_party/cutlass'
2024-12-18T06:52:16.4785413Z Entering 'third_party/eigen'
2024-12-18T06:52:16.4908717Z Entering 'third_party/fbgemm'
2024-12-18T06:52:16.5028008Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T06:52:16.5146538Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T06:52:16.5264815Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T06:52:16.5400331Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T06:52:16.5517403Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T06:52:16.5640314Z Entering 'third_party/flatbuffers'
2024-12-18T06:52:16.5767811Z Entering 'third_party/fmt'
2024-12-18T06:52:16.5890429Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T06:52:16.6013468Z Entering 'third_party/gloo'
2024-12-18T06:52:16.6138698Z Entering 'third_party/googletest'
2024-12-18T06:52:16.6260234Z Entering 'third_party/ideep'
2024-12-18T06:52:16.6374303Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T06:52:16.6508948Z Entering 'third_party/ittapi'
2024-12-18T06:52:16.6625794Z Entering 'third_party/kineto'
2024-12-18T06:52:16.6740724Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T06:52:16.6848961Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T06:52:16.6964653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T06:52:16.7082695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T06:52:16.7196808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T06:52:16.7305085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T06:52:16.7429595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T06:52:16.7547605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T06:52:16.7663391Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T06:52:16.7781619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T06:52:16.7902408Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T06:52:16.8015252Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T06:52:16.8136549Z Entering 'third_party/mimalloc'
2024-12-18T06:52:16.8255593Z Entering 'third_party/nccl/nccl'
2024-12-18T06:52:16.8379586Z Entering 'third_party/nlohmann'
2024-12-18T06:52:16.8500688Z Entering 'third_party/onnx'
2024-12-18T06:52:16.8647682Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T06:52:16.8773356Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T06:52:16.8891750Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T06:52:16.9006832Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T06:52:16.9118599Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T06:52:16.9229337Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T06:52:16.9344961Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T06:52:16.9457729Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T06:52:16.9570170Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T06:52:16.9682689Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T06:52:16.9800050Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T06:52:16.9918664Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T06:52:17.0080601Z Entering 'third_party/pocketfft'
2024-12-18T06:52:17.0198339Z Entering 'third_party/protobuf'
2024-12-18T06:52:17.0323133Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T06:52:17.0434982Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T06:52:17.0556544Z Entering 'third_party/psimd'
2024-12-18T06:52:17.0674662Z Entering 'third_party/pthreadpool'
2024-12-18T06:52:17.0795521Z Entering 'third_party/pybind11'
2024-12-18T06:52:17.0917481Z Entering 'third_party/python-peachpy'
2024-12-18T06:52:17.1034425Z Entering 'third_party/sleef'
2024-12-18T06:52:17.1151585Z Entering 'third_party/tensorpipe'
2024-12-18T06:52:17.1264305Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T06:52:17.1375588Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T06:52:17.1486496Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T06:52:17.1600324Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T06:52:17.1707690Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T06:52:17.1868989Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T06:52:17.1917681Z http.https://github.com/.extraheader
2024-12-18T06:52:17.1939794Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-18T06:52:17.2017674Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T06:52:17.2695246Z Entering 'android/libs/fbjni'
2024-12-18T06:52:17.2763298Z http.https://github.com/.extraheader
2024-12-18T06:52:17.2874652Z Entering 'third_party/FP16'
2024-12-18T06:52:17.2940757Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3039518Z Entering 'third_party/FXdiv'
2024-12-18T06:52:17.3104809Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3199714Z Entering 'third_party/NNPACK'
2024-12-18T06:52:17.3263931Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3355496Z Entering 'third_party/NVTX'
2024-12-18T06:52:17.3420955Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3516808Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T06:52:17.3580093Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3674343Z Entering 'third_party/XNNPACK'
2024-12-18T06:52:17.3737085Z http.https://github.com/.extraheader
2024-12-18T06:52:17.3866348Z Entering 'third_party/benchmark'
2024-12-18T06:52:17.3931287Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4026509Z Entering 'third_party/composable_kernel'
2024-12-18T06:52:17.4090848Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4199162Z Entering 'third_party/cpp-httplib'
2024-12-18T06:52:17.4263558Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4357627Z Entering 'third_party/cpuinfo'
2024-12-18T06:52:17.4421970Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4518318Z Entering 'third_party/cudnn_frontend'
2024-12-18T06:52:17.4582370Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4677265Z Entering 'third_party/cutlass'
2024-12-18T06:52:17.4741303Z http.https://github.com/.extraheader
2024-12-18T06:52:17.4854390Z Entering 'third_party/eigen'
2024-12-18T06:52:17.4919260Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5018266Z Entering 'third_party/fbgemm'
2024-12-18T06:52:17.5085503Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5178698Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T06:52:17.5239429Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5345301Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T06:52:17.5407319Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5500671Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T06:52:17.5563057Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5676527Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T06:52:17.5737697Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5833570Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T06:52:17.5895005Z http.https://github.com/.extraheader
2024-12-18T06:52:17.5995634Z Entering 'third_party/flatbuffers'
2024-12-18T06:52:17.6060826Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6162049Z Entering 'third_party/fmt'
2024-12-18T06:52:17.6228573Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6324496Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T06:52:17.6389310Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6484455Z Entering 'third_party/gloo'
2024-12-18T06:52:17.6547688Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6642245Z Entering 'third_party/googletest'
2024-12-18T06:52:17.6704751Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6797420Z Entering 'third_party/ideep'
2024-12-18T06:52:17.6862696Z http.https://github.com/.extraheader
2024-12-18T06:52:17.6951546Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T06:52:17.7013514Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7130963Z Entering 'third_party/ittapi'
2024-12-18T06:52:17.7195939Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7290756Z Entering 'third_party/kineto'
2024-12-18T06:52:17.7355780Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7453538Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T06:52:17.7515943Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7606118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T06:52:17.7666459Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7765856Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T06:52:17.7825202Z http.https://github.com/.extraheader
2024-12-18T06:52:17.7919845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T06:52:17.7980625Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8074330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T06:52:17.8134802Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8222241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T06:52:17.8283539Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8384282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T06:52:17.8447702Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8541354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T06:52:17.8602180Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8695378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T06:52:17.8756494Z http.https://github.com/.extraheader
2024-12-18T06:52:17.8850134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T06:52:17.8908625Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9006916Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T06:52:17.9068930Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9160744Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T06:52:17.9220763Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9319032Z Entering 'third_party/mimalloc'
2024-12-18T06:52:17.9383880Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9476281Z Entering 'third_party/nccl/nccl'
2024-12-18T06:52:17.9540694Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9633011Z Entering 'third_party/nlohmann'
2024-12-18T06:52:17.9695609Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9790805Z Entering 'third_party/onnx'
2024-12-18T06:52:17.9853627Z http.https://github.com/.extraheader
2024-12-18T06:52:17.9976530Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T06:52:18.0043375Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0156108Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T06:52:18.0219919Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0316465Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T06:52:18.0377581Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0472586Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T06:52:18.0531944Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0620919Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T06:52:18.0682048Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0770943Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T06:52:18.0831350Z http.https://github.com/.extraheader
2024-12-18T06:52:18.0926744Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T06:52:18.0985985Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1075171Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T06:52:18.1136930Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1227314Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T06:52:18.1285604Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1369201Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T06:52:18.1429448Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1528947Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T06:52:18.1590209Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1688481Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T06:52:18.1749300Z http.https://github.com/.extraheader
2024-12-18T06:52:18.1886794Z Entering 'third_party/pocketfft'
2024-12-18T06:52:18.1952257Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2044993Z Entering 'third_party/protobuf'
2024-12-18T06:52:18.2109229Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2209267Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T06:52:18.2270304Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2363766Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T06:52:18.2425372Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2525405Z Entering 'third_party/psimd'
2024-12-18T06:52:18.2590106Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2683100Z Entering 'third_party/pthreadpool'
2024-12-18T06:52:18.2746769Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2839555Z Entering 'third_party/pybind11'
2024-12-18T06:52:18.2901856Z http.https://github.com/.extraheader
2024-12-18T06:52:18.2993813Z Entering 'third_party/python-peachpy'
2024-12-18T06:52:18.3059416Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3154439Z Entering 'third_party/sleef'
2024-12-18T06:52:18.3216788Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3308278Z Entering 'third_party/tensorpipe'
2024-12-18T06:52:18.3372317Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3461576Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T06:52:18.3520062Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3608601Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T06:52:18.3667513Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3756957Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T06:52:18.3817326Z http.https://github.com/.extraheader
2024-12-18T06:52:18.3906052Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T06:52:18.3965760Z http.https://github.com/.extraheader
2024-12-18T06:52:18.4050513Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T06:52:18.4110728Z http.https://github.com/.extraheader
2024-12-18T06:52:18.4526377Z Cleaning up orphan processes