2024-06-26T06:29:32.9909570Z Current runner version: '2.317.0' 2024-06-26T06:29:32.9918497Z Runner name: 'i-02a7758459db2295e' 2024-06-26T06:29:32.9919660Z Runner group name: 'Default' 2024-06-26T06:29:32.9920791Z Machine name: 'ip-10-0-44-37' 2024-06-26T06:29:32.9926978Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T06:29:32.9929603Z Actions: read 2024-06-26T06:29:32.9930432Z Attestations: read 2024-06-26T06:29:32.9931068Z Checks: read 2024-06-26T06:29:32.9931710Z Contents: read 2024-06-26T06:29:32.9932460Z Deployments: read 2024-06-26T06:29:32.9933085Z Discussions: read 2024-06-26T06:29:32.9933777Z Issues: read 2024-06-26T06:29:32.9934450Z Metadata: read 2024-06-26T06:29:32.9935050Z Packages: read 2024-06-26T06:29:32.9935717Z Pages: read 2024-06-26T06:29:32.9936442Z PullRequests: read 2024-06-26T06:29:32.9937113Z RepositoryProjects: read 2024-06-26T06:29:32.9937891Z SecurityEvents: read 2024-06-26T06:29:32.9938624Z Statuses: read 2024-06-26T06:29:32.9939243Z ##[endgroup] 2024-06-26T06:29:32.9944573Z Secret source: Actions 2024-06-26T06:29:32.9945668Z Prepare workflow directory 2024-06-26T06:29:33.4758140Z Prepare all required actions 2024-06-26T06:29:33.5003837Z Getting action download info 2024-06-26T06:29:33.7067020Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T06:29:34.0921220Z Download action repository 'pytorch/pytorch@main' (SHA:000f2d637bf383567bd62289f12c868c3afea02d) 2024-06-26T06:29:37.8429630Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T06:29:37.9755559Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T06:29:38.2650630Z Getting action download info 2024-06-26T06:29:38.3670147Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T06:29:38.5570224Z Getting action download info 2024-06-26T06:29:38.6511376Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T06:29:38.7994593Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/129470/merge (4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d) 2024-06-26T06:29:38.7998344Z ##[group] Inputs 2024-06-26T06:29:38.7998933Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:29:38.8001050Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-06-26T06:29:38.8003825Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:29:38.8005183Z sync-tag: 2024-06-26T06:29:38.8006406Z timeout-minutes: 360 2024-06-26T06:29:38.8006863Z use-gha: 2024-06-26T06:29:38.8007234Z dashboard-tag: 2024-06-26T06:29:38.8007609Z s3-bucket: gha-artifacts 2024-06-26T06:29:38.8008060Z aws-role-to-assume: 2024-06-26T06:29:38.8008489Z ##[endgroup] 2024-06-26T06:29:38.8009794Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:29:38.9000672Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T06:29:38.9220429Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:29:38.9235150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:29:38.9235841Z ##[endgroup] 2024-06-26T06:29:39.7228510Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T06:29:39.7229247Z with: 2024-06-26T06:29:39.7230062Z github-secret: *** 2024-06-26T06:29:39.7231296Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T06:29:39.7232834Z activate-with-label: false 2024-06-26T06:29:39.7233283Z label: with-ssh 2024-06-26T06:29:39.7233662Z remove-existing-keys: true 2024-06-26T06:29:39.7234111Z fail-silently: true 2024-06-26T06:29:39.7234504Z env: 2024-06-26T06:29:39.7234819Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:29:39.7235248Z ##[endgroup] 2024-06-26T06:29:39.8463811Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T06:29:40.1389951Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:29:40.2044417Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:29:40.2045354Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:29:40.2716430Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:29:40.2858722Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T06:29:40.2859438Z with: 2024-06-26T06:29:40.2859780Z submodules: recursive 2024-06-26T06:29:40.2860216Z fetch-depth: 0 2024-06-26T06:29:40.2860553Z env: 2024-06-26T06:29:40.2860879Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:29:40.2861303Z ##[endgroup] 2024-06-26T06:29:40.3133516Z ##[group]Run retry () { 2024-06-26T06:29:40.3134006Z retry () { 2024-06-26T06:29:40.3134636Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T06:29:40.3135342Z } 2024-06-26T06:29:40.3135705Z echo "${GITHUB_WORKSPACE}" 2024-06-26T06:29:40.3136232Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T06:29:40.3136834Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:29:40.3137386Z else 2024-06-26T06:29:40.3137811Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:29:40.3138346Z fi 2024-06-26T06:29:40.3138765Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T06:29:40.3148045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:29:40.3148704Z env: 2024-06-26T06:29:40.3149066Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:29:40.3149478Z NO_SUDO: 2024-06-26T06:29:40.3149818Z ##[endgroup] 2024-06-26T06:29:40.3177529Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:29:44.0666019Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T06:29:44.0666623Z with: 2024-06-26T06:29:44.0667004Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:29:44.0667555Z fetch-depth: 0 2024-06-26T06:29:44.0667934Z submodules: recursive 2024-06-26T06:29:44.0668328Z quiet-checkout: true 2024-06-26T06:29:44.0668756Z repository: pytorch/pytorch 2024-06-26T06:29:44.0669372Z token: *** 2024-06-26T06:29:44.0669707Z ssh-strict: true 2024-06-26T06:29:44.0670108Z persist-credentials: true 2024-06-26T06:29:44.0670547Z clean: true 2024-06-26T06:29:44.0670932Z sparse-checkout-cone-mode: true 2024-06-26T06:29:44.0671408Z lfs: false 2024-06-26T06:29:44.0671775Z set-safe-directory: true 2024-06-26T06:29:44.0672170Z env: 2024-06-26T06:29:44.0672498Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:29:44.0672932Z ##[endgroup] 2024-06-26T06:29:44.2009902Z Syncing repository: pytorch/pytorch 2024-06-26T06:29:44.2013123Z ##[group]Getting Git version info 2024-06-26T06:29:44.2014048Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:29:44.2015177Z [command]/usr/bin/git version 2024-06-26T06:29:44.2015636Z git version 2.40.1 2024-06-26T06:29:44.2018247Z ##[endgroup] 2024-06-26T06:29:44.2035950Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/126f9aeb-d02d-41ff-a11d-8022b773ae2a' before making global git config changes 2024-06-26T06:29:44.2037589Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:29:44.2042587Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:29:44.2087224Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:29:44.2093285Z ##[group]Initializing the repository 2024-06-26T06:29:44.2097440Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:29:44.2127714Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T06:29:44.2129443Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T06:29:44.2130723Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T06:29:44.2132134Z hint: 2024-06-26T06:29:44.2132691Z hint: git config --global init.defaultBranch 2024-06-26T06:29:44.2133252Z hint: 2024-06-26T06:29:44.2133889Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T06:29:44.2134947Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T06:29:44.2136320Z hint: 2024-06-26T06:29:44.2137160Z hint: git branch -m 2024-06-26T06:29:44.2138106Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T06:29:44.2141080Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T06:29:44.2175569Z ##[endgroup] 2024-06-26T06:29:44.2176288Z ##[group]Disabling automatic garbage collection 2024-06-26T06:29:44.2179558Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T06:29:44.2211984Z ##[endgroup] 2024-06-26T06:29:44.2212624Z ##[group]Setting up auth 2024-06-26T06:29:44.2221116Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:29:44.2256004Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:29:44.2547794Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:29:44.2578047Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:29:44.2868879Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:29:44.2917306Z ##[endgroup] 2024-06-26T06:29:44.2918300Z ##[group]Fetching the repository 2024-06-26T06:29:44.2926208Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T06:29:46.5396000Z remote: Enumerating objects: 987625 2024-06-26T06:29:46.5397413Z remote: Enumerating objects: 990184, done. 2024-06-26T06:29:46.5398931Z remote: Counting objects: 0% (1/2559) 2024-06-26T06:29:46.5400209Z remote: Counting objects: 1% (26/2559) 2024-06-26T06:29:46.5401617Z remote: Counting objects: 2% (52/2559) 2024-06-26T06:29:46.5402907Z remote: Counting objects: 3% (77/2559) 2024-06-26T06:29:46.5404242Z remote: Counting objects: 4% (103/2559) 2024-06-26T06:29:46.5405621Z remote: Counting objects: 5% (128/2559) 2024-06-26T06:29:46.5406846Z remote: Counting objects: 6% (154/2559) 2024-06-26T06:29:46.5408141Z remote: Counting objects: 7% (180/2559) 2024-06-26T06:29:46.5409497Z remote: Counting objects: 8% (205/2559) 2024-06-26T06:29:46.5410632Z remote: Counting objects: 9% (231/2559) 2024-06-26T06:29:46.5411886Z remote: Counting objects: 10% (256/2559) 2024-06-26T06:29:46.5413397Z remote: Counting objects: 11% (282/2559) 2024-06-26T06:29:46.5414619Z remote: Counting objects: 12% (308/2559) 2024-06-26T06:29:46.5415892Z remote: Counting objects: 13% (333/2559) 2024-06-26T06:29:46.5417216Z remote: Counting objects: 14% (359/2559) 2024-06-26T06:29:46.5418219Z remote: Counting objects: 15% (384/2559) 2024-06-26T06:29:46.5419532Z remote: Counting objects: 16% (410/2559) 2024-06-26T06:29:46.5420378Z remote: Counting objects: 17% (436/2559) 2024-06-26T06:29:46.5422713Z remote: Counting objects: 18% (461/2559) 2024-06-26T06:29:46.5423484Z remote: Counting objects: 19% (487/2559) 2024-06-26T06:29:46.5424124Z remote: Counting objects: 20% (512/2559) 2024-06-26T06:29:46.5424761Z remote: Counting objects: 21% (538/2559) 2024-06-26T06:29:46.5425375Z remote: Counting objects: 22% (563/2559) 2024-06-26T06:29:46.5426009Z remote: Counting objects: 23% (589/2559) 2024-06-26T06:29:46.5427150Z remote: Counting objects: 24% (615/2559) 2024-06-26T06:29:46.5428479Z remote: Counting objects: 25% (640/2559) 2024-06-26T06:29:46.5429622Z remote: Counting objects: 26% (666/2559) 2024-06-26T06:29:46.5430953Z remote: Counting objects: 27% (691/2559) 2024-06-26T06:29:46.5432581Z remote: Counting objects: 28% (717/2559) 2024-06-26T06:29:46.5433877Z remote: Counting objects: 29% (743/2559) 2024-06-26T06:29:46.5435047Z remote: Counting objects: 30% (768/2559) 2024-06-26T06:29:46.5436363Z remote: Counting objects: 31% (794/2559) 2024-06-26T06:29:46.5437499Z remote: Counting objects: 32% (819/2559) 2024-06-26T06:29:46.5438794Z remote: Counting objects: 33% (845/2559) 2024-06-26T06:29:46.5439972Z remote: Counting objects: 34% (871/2559) 2024-06-26T06:29:46.5441180Z remote: Counting objects: 35% (896/2559) 2024-06-26T06:29:46.5441833Z remote: Counting objects: 36% (922/2559) 2024-06-26T06:29:46.5442712Z remote: Counting objects: 37% (947/2559) 2024-06-26T06:29:46.5444010Z remote: Counting objects: 38% (973/2559) 2024-06-26T06:29:46.5445161Z remote: Counting objects: 39% (999/2559) 2024-06-26T06:29:46.5446198Z remote: Counting objects: 40% (1024/2559) 2024-06-26T06:29:46.5447333Z remote: Counting objects: 41% (1050/2559) 2024-06-26T06:29:46.5448647Z remote: Counting objects: 42% (1075/2559) 2024-06-26T06:29:46.5449549Z remote: Counting objects: 43% (1101/2559) 2024-06-26T06:29:46.5450427Z remote: Counting objects: 44% (1126/2559) 2024-06-26T06:29:46.5451628Z remote: Counting objects: 45% (1152/2559) 2024-06-26T06:29:46.5452294Z remote: Counting objects: 46% (1178/2559) 2024-06-26T06:29:46.5452950Z remote: Counting objects: 47% (1203/2559) 2024-06-26T06:29:46.5453605Z remote: Counting objects: 48% (1229/2559) 2024-06-26T06:29:46.5454249Z remote: Counting objects: 49% (1254/2559) 2024-06-26T06:29:46.5454903Z remote: Counting objects: 50% (1280/2559) 2024-06-26T06:29:46.5455553Z remote: Counting objects: 51% (1306/2559) 2024-06-26T06:29:46.5456188Z remote: Counting objects: 52% (1331/2559) 2024-06-26T06:29:46.5457400Z remote: Counting objects: 53% (1357/2559) 2024-06-26T06:29:46.5458211Z remote: Counting objects: 54% (1382/2559) 2024-06-26T06:29:46.5458840Z remote: Counting objects: 55% (1408/2559) 2024-06-26T06:29:46.5459508Z remote: Counting objects: 56% (1434/2559) 2024-06-26T06:29:46.5460160Z remote: Counting objects: 57% (1459/2559) 2024-06-26T06:29:46.5460794Z remote: Counting objects: 58% (1485/2559) 2024-06-26T06:29:46.5461444Z remote: Counting objects: 59% (1510/2559) 2024-06-26T06:29:46.5462102Z remote: Counting objects: 60% (1536/2559) 2024-06-26T06:29:46.5463006Z remote: Counting objects: 61% (1561/2559) 2024-06-26T06:29:46.5463659Z remote: Counting objects: 62% (1587/2559) 2024-06-26T06:29:46.5464333Z remote: Counting objects: 63% (1613/2559) 2024-06-26T06:29:46.5464985Z remote: Counting objects: 64% (1638/2559) 2024-06-26T06:29:46.5465621Z remote: Counting objects: 65% (1664/2559) 2024-06-26T06:29:46.5466282Z remote: Counting objects: 66% (1689/2559) 2024-06-26T06:29:46.5466931Z remote: Counting objects: 67% (1715/2559) 2024-06-26T06:29:46.5467563Z remote: Counting objects: 68% (1741/2559) 2024-06-26T06:29:46.5468485Z remote: Counting objects: 69% (1766/2559) 2024-06-26T06:29:46.5469132Z remote: Counting objects: 70% (1792/2559) 2024-06-26T06:29:46.5469782Z remote: Counting objects: 71% (1817/2559) 2024-06-26T06:29:46.5470410Z remote: Counting objects: 72% (1843/2559) 2024-06-26T06:29:46.5471061Z remote: Counting objects: 73% (1869/2559) 2024-06-26T06:29:46.5471718Z remote: Counting objects: 74% (1894/2559) 2024-06-26T06:29:46.5472356Z remote: Counting objects: 75% (1920/2559) 2024-06-26T06:29:46.5473010Z remote: Counting objects: 76% (1945/2559) 2024-06-26T06:29:46.5473712Z remote: Counting objects: 77% (1971/2559) 2024-06-26T06:29:46.5474344Z remote: Counting objects: 78% (1997/2559) 2024-06-26T06:29:46.5475132Z remote: Counting objects: 79% (2022/2559) 2024-06-26T06:29:46.5475809Z remote: Counting objects: 80% (2048/2559) 2024-06-26T06:29:46.5476433Z remote: Counting objects: 81% (2073/2559) 2024-06-26T06:29:46.5477088Z remote: Counting objects: 82% (2099/2559) 2024-06-26T06:29:46.5477740Z remote: Counting objects: 83% (2124/2559) 2024-06-26T06:29:46.5478376Z remote: Counting objects: 84% (2150/2559) 2024-06-26T06:29:46.5479024Z remote: Counting objects: 85% (2176/2559) 2024-06-26T06:29:46.5479676Z remote: Counting objects: 86% (2201/2559) 2024-06-26T06:29:46.5480307Z remote: Counting objects: 87% (2227/2559) 2024-06-26T06:29:46.5480963Z remote: Counting objects: 88% (2252/2559) 2024-06-26T06:29:46.5481610Z remote: Counting objects: 89% (2278/2559) 2024-06-26T06:29:46.5482258Z remote: Counting objects: 90% (2304/2559) 2024-06-26T06:29:46.5482891Z remote: Counting objects: 91% (2329/2559) 2024-06-26T06:29:46.5483549Z remote: Counting objects: 92% (2355/2559) 2024-06-26T06:29:46.5484201Z remote: Counting objects: 93% (2380/2559) 2024-06-26T06:29:46.5484827Z remote: Counting objects: 94% (2406/2559) 2024-06-26T06:29:46.5485496Z remote: Counting objects: 95% (2432/2559) 2024-06-26T06:29:46.5486147Z remote: Counting objects: 96% (2457/2559) 2024-06-26T06:29:46.5486776Z remote: Counting objects: 97% (2483/2559) 2024-06-26T06:29:46.5487427Z remote: Counting objects: 98% (2508/2559) 2024-06-26T06:29:46.5488081Z remote: Counting objects: 99% (2534/2559) 2024-06-26T06:29:46.5488711Z remote: Counting objects: 100% (2559/2559) 2024-06-26T06:29:46.5489414Z remote: Counting objects: 100% (2559/2559), done. 2024-06-26T06:29:46.5618116Z remote: Compressing objects: 0% (1/1092) 2024-06-26T06:29:46.6005373Z remote: Compressing objects: 1% (11/1092) 2024-06-26T06:29:46.6322744Z remote: Compressing objects: 2% (22/1092) 2024-06-26T06:29:46.6723152Z remote: Compressing objects: 3% (33/1092) 2024-06-26T06:29:46.6906435Z remote: Compressing objects: 4% (44/1092) 2024-06-26T06:29:46.7151621Z remote: Compressing objects: 5% (55/1092) 2024-06-26T06:29:46.7330129Z remote: Compressing objects: 6% (66/1092) 2024-06-26T06:29:46.7617911Z remote: Compressing objects: 7% (77/1092) 2024-06-26T06:29:46.8049331Z remote: Compressing objects: 8% (88/1092) 2024-06-26T06:29:46.8702330Z remote: Compressing objects: 9% (99/1092) 2024-06-26T06:29:46.9604285Z remote: Compressing objects: 10% (110/1092) 2024-06-26T06:29:47.0733056Z remote: Compressing objects: 11% (121/1092) 2024-06-26T06:29:47.1119938Z remote: Compressing objects: 12% (132/1092) 2024-06-26T06:29:47.1492066Z remote: Compressing objects: 13% (142/1092) 2024-06-26T06:29:47.1779160Z remote: Compressing objects: 14% (153/1092) 2024-06-26T06:29:47.1912909Z remote: Compressing objects: 15% (164/1092) 2024-06-26T06:29:47.2087330Z remote: Compressing objects: 16% (175/1092) 2024-06-26T06:29:47.2158977Z remote: Compressing objects: 17% (186/1092) 2024-06-26T06:29:47.2206588Z remote: Compressing objects: 18% (197/1092) 2024-06-26T06:29:47.2211134Z remote: Compressing objects: 19% (208/1092) 2024-06-26T06:29:47.2235115Z remote: Compressing objects: 20% (219/1092) 2024-06-26T06:29:47.2248424Z remote: Compressing objects: 21% (230/1092) 2024-06-26T06:29:47.2267411Z remote: Compressing objects: 22% (241/1092) 2024-06-26T06:29:47.2277949Z remote: Compressing objects: 23% (252/1092) 2024-06-26T06:29:47.2300590Z remote: Compressing objects: 24% (263/1092) 2024-06-26T06:29:47.2301292Z remote: Compressing objects: 25% (273/1092) 2024-06-26T06:29:47.2306730Z remote: Compressing objects: 26% (284/1092) 2024-06-26T06:29:47.2314493Z remote: Compressing objects: 27% (295/1092) 2024-06-26T06:29:47.2318173Z remote: Compressing objects: 28% (306/1092) 2024-06-26T06:29:47.2322188Z remote: Compressing objects: 29% (317/1092) 2024-06-26T06:29:47.2333572Z remote: Compressing objects: 30% (328/1092) 2024-06-26T06:29:47.2336577Z remote: Compressing objects: 31% (339/1092) 2024-06-26T06:29:47.2350477Z remote: Compressing objects: 32% (350/1092) 2024-06-26T06:29:47.2356699Z remote: Compressing objects: 33% (361/1092) 2024-06-26T06:29:47.2362323Z remote: Compressing objects: 34% (372/1092) 2024-06-26T06:29:47.2368243Z remote: Compressing objects: 35% (383/1092) 2024-06-26T06:29:47.2382144Z remote: Compressing objects: 36% (394/1092) 2024-06-26T06:29:47.2392507Z remote: Compressing objects: 37% (405/1092) 2024-06-26T06:29:47.2410798Z remote: Compressing objects: 38% (415/1092) 2024-06-26T06:29:47.2423431Z remote: Compressing objects: 39% (426/1092) 2024-06-26T06:29:47.2439526Z remote: Compressing objects: 40% (437/1092) 2024-06-26T06:29:47.2453478Z remote: Compressing objects: 41% (448/1092) 2024-06-26T06:29:47.2469729Z remote: Compressing objects: 42% (459/1092) 2024-06-26T06:29:47.2484018Z remote: Compressing objects: 43% (470/1092) 2024-06-26T06:29:47.2502682Z remote: Compressing objects: 44% (481/1092) 2024-06-26T06:29:47.2506924Z remote: Compressing objects: 45% (492/1092) 2024-06-26T06:29:47.2509951Z remote: Compressing objects: 46% (503/1092) 2024-06-26T06:29:47.2514141Z remote: Compressing objects: 47% (514/1092) 2024-06-26T06:29:47.2520346Z remote: Compressing objects: 48% (525/1092) 2024-06-26T06:29:47.2538168Z remote: Compressing objects: 49% (536/1092) 2024-06-26T06:29:47.2553093Z remote: Compressing objects: 50% (546/1092) 2024-06-26T06:29:47.2558011Z remote: Compressing objects: 51% (557/1092) 2024-06-26T06:29:47.2570040Z remote: Compressing objects: 52% (568/1092) 2024-06-26T06:29:47.2585618Z remote: Compressing objects: 53% (579/1092) 2024-06-26T06:29:47.2586319Z remote: Compressing objects: 54% (590/1092) 2024-06-26T06:29:47.2591837Z remote: Compressing objects: 55% (601/1092) 2024-06-26T06:29:47.2609681Z remote: Compressing objects: 56% (612/1092) 2024-06-26T06:29:47.2627160Z remote: Compressing objects: 57% (623/1092) 2024-06-26T06:29:47.2630964Z remote: Compressing objects: 58% (634/1092) 2024-06-26T06:29:47.2642165Z remote: Compressing objects: 59% (645/1092) 2024-06-26T06:29:47.2654230Z remote: Compressing objects: 60% (656/1092) 2024-06-26T06:29:47.2665299Z remote: Compressing objects: 61% (667/1092) 2024-06-26T06:29:47.2676060Z remote: Compressing objects: 62% (678/1092) 2024-06-26T06:29:47.2689190Z remote: Compressing objects: 63% (688/1092) 2024-06-26T06:29:47.2693185Z remote: Compressing objects: 64% (699/1092) 2024-06-26T06:29:47.2699030Z remote: Compressing objects: 65% (710/1092) 2024-06-26T06:29:47.2711656Z remote: Compressing objects: 66% (721/1092) 2024-06-26T06:29:47.2719016Z remote: Compressing objects: 67% (732/1092) 2024-06-26T06:29:47.2723445Z remote: Compressing objects: 68% (743/1092) 2024-06-26T06:29:47.2725457Z remote: Compressing objects: 69% (754/1092) 2024-06-26T06:29:47.2735600Z remote: Compressing objects: 70% (765/1092) 2024-06-26T06:29:47.2741043Z remote: Compressing objects: 71% (776/1092) 2024-06-26T06:29:47.2745825Z remote: Compressing objects: 72% (787/1092) 2024-06-26T06:29:47.2750277Z remote: Compressing objects: 73% (798/1092) 2024-06-26T06:29:47.2753425Z remote: Compressing objects: 74% (809/1092) 2024-06-26T06:29:47.2755226Z remote: Compressing objects: 75% (819/1092) 2024-06-26T06:29:47.2756703Z remote: Compressing objects: 76% (830/1092) 2024-06-26T06:29:47.2758460Z remote: Compressing objects: 77% (841/1092) 2024-06-26T06:29:47.2759604Z remote: Compressing objects: 78% (852/1092) 2024-06-26T06:29:47.2761005Z remote: Compressing objects: 79% (863/1092) 2024-06-26T06:29:47.2762161Z remote: Compressing objects: 80% (874/1092) 2024-06-26T06:29:47.2770059Z remote: Compressing objects: 81% (885/1092) 2024-06-26T06:29:47.2776120Z remote: Compressing objects: 82% (896/1092) 2024-06-26T06:29:47.2780192Z remote: Compressing objects: 83% (907/1092) 2024-06-26T06:29:47.2784596Z remote: Compressing objects: 84% (918/1092) 2024-06-26T06:29:47.2785743Z remote: Compressing objects: 85% (929/1092) 2024-06-26T06:29:47.2788664Z remote: Compressing objects: 86% (940/1092) 2024-06-26T06:29:47.2792531Z remote: Compressing objects: 87% (951/1092) 2024-06-26T06:29:47.2793263Z remote: Compressing objects: 88% (961/1092) 2024-06-26T06:29:47.2796689Z remote: Compressing objects: 89% (972/1092) 2024-06-26T06:29:47.2798111Z remote: Compressing objects: 90% (983/1092) 2024-06-26T06:29:47.2799599Z remote: Compressing objects: 91% (994/1092) 2024-06-26T06:29:47.2801785Z remote: Compressing objects: 92% (1005/1092) 2024-06-26T06:29:47.2805168Z remote: Compressing objects: 93% (1016/1092) 2024-06-26T06:29:47.2806171Z remote: Compressing objects: 94% (1027/1092) 2024-06-26T06:29:47.2806885Z remote: Compressing objects: 95% (1038/1092) 2024-06-26T06:29:47.2810403Z remote: Compressing objects: 96% (1049/1092) 2024-06-26T06:29:47.2811801Z remote: Compressing objects: 97% (1060/1092) 2024-06-26T06:29:47.2812543Z remote: Compressing objects: 98% (1071/1092) 2024-06-26T06:29:47.2813250Z remote: Compressing objects: 99% (1082/1092) 2024-06-26T06:29:47.2813927Z remote: Compressing objects: 100% (1092/1092) 2024-06-26T06:29:47.2814675Z remote: Compressing objects: 100% (1092/1092), done. 2024-06-26T06:30:14.3039143Z remote: Total 990184 (delta 2041), reused 1828 (delta 1466), pack-reused 987625 2024-06-26T06:30:36.9470200Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T06:30:36.9497681Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:30:36.9503614Z ##[endgroup] 2024-06-26T06:30:36.9504390Z ##[group]Determining the checkout info 2024-06-26T06:30:36.9507582Z ##[endgroup] 2024-06-26T06:30:36.9508387Z ##[group]Checking out the ref 2024-06-26T06:30:36.9510240Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:30:38.7700961Z ##[endgroup] 2024-06-26T06:30:38.7701704Z ##[group]Setting up auth for fetching submodules 2024-06-26T06:30:38.7708475Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:30:38.7764648Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T06:30:38.7797159Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T06:30:38.7830647Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T06:30:38.7859320Z ##[endgroup] 2024-06-26T06:30:38.7859955Z ##[group]Fetching submodules 2024-06-26T06:30:38.7865137Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T06:30:38.8190445Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T06:30:38.8497197Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T06:30:38.8499372Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T06:30:38.8503242Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T06:30:38.8507625Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T06:30:38.8510934Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T06:30:38.8513704Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T06:30:38.8517670Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T06:30:38.8521507Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T06:30:38.8525288Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T06:30:38.8529451Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T06:30:38.8533552Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T06:30:38.8537609Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T06:30:38.8541952Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T06:30:38.8547187Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T06:30:38.8551570Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T06:30:38.8556358Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T06:30:38.8561463Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:30:38.8566294Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T06:30:38.8571547Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T06:30:38.8576749Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T06:30:38.8582687Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T06:30:38.8588573Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T06:30:38.8594479Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T06:30:38.8600232Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T06:30:38.8606244Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T06:30:38.8612180Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T06:30:38.8618860Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T06:30:38.8625481Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T06:30:38.8631916Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T06:30:38.8638586Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T06:30:38.8645376Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T06:30:38.8652274Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T06:30:38.8659516Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T06:30:38.8667427Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T06:30:38.8674676Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T06:30:38.8702526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T06:30:39.1115619Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T06:30:39.3414916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T06:30:39.5362132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T06:30:39.7859612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T06:30:41.9203674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T06:30:53.0431820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T06:30:53.4503731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T06:30:53.9978824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T06:30:54.6276435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T06:30:56.1040630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T06:30:58.3309128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T06:31:05.6173610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T06:31:07.1250136Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T06:31:09.0293233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T06:31:10.4117083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T06:31:10.5724854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T06:31:11.0394657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T06:31:11.3713705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T06:31:13.1694685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T06:31:13.5183920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T06:31:13.7830128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T06:31:15.6539334Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T06:31:16.4438750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T06:31:17.1074907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T06:31:24.1130294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T06:31:26.8680025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T06:31:33.4772882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T06:31:33.6949693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T06:31:43.4990121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T06:31:43.6621894Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T06:31:43.8962980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T06:31:45.0464211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T06:31:45.3508594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T06:31:46.0002083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T06:31:46.4700281Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T06:31:46.4826919Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T06:31:46.4921650Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T06:31:46.5200551Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T06:31:46.5624466Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T06:31:47.7703597Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T06:31:47.7965740Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T06:31:47.8444043Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T06:31:47.9646487Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T06:31:48.0042831Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T06:31:48.6649920Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T06:31:48.9759456Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T06:31:49.0680400Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T06:31:49.0698772Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:31:49.0702160Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:31:49.0706331Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:31:49.0709866Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:31:49.0713835Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:31:49.0740215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T06:31:50.0583377Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T06:31:50.6871542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T06:31:52.8899762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T06:31:54.0023233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T06:31:54.3416764Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T06:31:54.4592923Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T06:31:54.9775803Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T06:31:55.0536017Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T06:31:55.0672010Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T06:31:55.2091563Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T06:31:55.2508168Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T06:31:55.2611428Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T06:31:55.3089589Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T06:31:55.3373998Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T06:31:55.3907166Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T06:31:55.4048739Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T06:31:55.4065780Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T06:31:55.4091369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T06:32:09.5000998Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T06:32:09.5202734Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T06:32:09.6374005Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T06:32:09.6392920Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:09.6395443Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:09.6399303Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:09.6426784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T06:32:10.2452878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T06:32:11.5918767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T06:32:12.7887140Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T06:32:12.7904428Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:12.7907925Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:12.7911637Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:12.7915083Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:12.7918718Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:12.7922773Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:12.7925939Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:12.7930162Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:12.7957937Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T06:32:14.0163723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T06:32:14.3956918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T06:32:15.7558906Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T06:32:16.0559797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T06:32:16.6186979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T06:32:17.7477492Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T06:32:24.8112852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T06:32:25.2353723Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T06:32:25.2558208Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T06:32:25.2998281Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T06:32:25.3143663Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T06:32:25.3159729Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:25.3185919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T06:32:25.6307718Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T06:32:25.6511033Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T06:32:25.6984820Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T06:32:25.8219821Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T06:32:25.8400608Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T06:32:25.8837760Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T06:32:25.9548443Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T06:32:25.9981822Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T06:32:26.0270431Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T06:32:26.1575673Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T06:32:26.5859349Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T06:32:26.5896369Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:26.5898851Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:26.5926752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T06:32:27.0070711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T06:32:28.1770522Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T06:32:28.2161941Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T06:32:28.2967615Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T06:32:28.2988893Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:28.2991406Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:28.2995124Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:28.2998608Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:28.3002427Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:28.3005711Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:28.3009771Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:28.3013553Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:28.3041046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T06:32:28.7103882Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T06:32:29.8596585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T06:32:30.3121244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T06:32:37.3284365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T06:32:37.6325208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T06:32:37.8114844Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T06:32:38.1111225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T06:32:46.5843615Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T06:32:46.6326764Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T06:32:46.6500676Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T06:32:46.7753727Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T06:32:46.7894934Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T06:32:46.8055931Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T06:32:46.8230593Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T06:32:46.8247847Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:46.8250887Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:46.8278017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T06:32:48.8341276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T06:32:50.2148467Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T06:32:50.2706445Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T06:32:50.8319873Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T06:32:50.8444475Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T06:32:51.1819481Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T06:32:51.1843343Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:51.1846926Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:51.1874170Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T06:32:51.5919683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T06:32:52.7490337Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T06:32:52.8365971Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T06:32:52.8460405Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T06:32:52.8599503Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T06:32:52.9011646Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T06:32:52.9341973Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T06:32:52.9841592Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T06:32:53.0142990Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T06:32:53.0160694Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:53.0164135Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:53.0167907Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:53.0171744Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:53.0198459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T06:32:54.1287590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T06:32:54.3484057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T06:32:55.6148384Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T06:32:56.8105746Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T06:32:56.8272689Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T06:32:56.9105451Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T06:32:56.9435915Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T06:32:56.9451960Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:56.9478627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T06:32:57.1681859Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T06:32:57.1721536Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T06:32:57.2052219Z Entering 'android/libs/fbjni' 2024-06-26T06:32:57.2095346Z Entering 'third_party/FP16' 2024-06-26T06:32:57.2137550Z Entering 'third_party/FXdiv' 2024-06-26T06:32:57.2180113Z Entering 'third_party/NNPACK' 2024-06-26T06:32:57.2223095Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:57.2265780Z Entering 'third_party/XNNPACK' 2024-06-26T06:32:57.2324971Z Entering 'third_party/benchmark' 2024-06-26T06:32:57.2367691Z Entering 'third_party/cpp-httplib' 2024-06-26T06:32:57.2409383Z Entering 'third_party/cpuinfo' 2024-06-26T06:32:57.2451293Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:32:57.2492971Z Entering 'third_party/cutlass' 2024-06-26T06:32:57.2542140Z Entering 'third_party/eigen' 2024-06-26T06:32:57.2587043Z Entering 'third_party/fbgemm' 2024-06-26T06:32:57.2629586Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:32:57.2670320Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:32:57.2711422Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:32:57.2758438Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:32:57.2799052Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:32:57.2841099Z Entering 'third_party/flatbuffers' 2024-06-26T06:32:57.2886503Z Entering 'third_party/fmt' 2024-06-26T06:32:57.2929891Z Entering 'third_party/foxi' 2024-06-26T06:32:57.2971798Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:57.3013798Z Entering 'third_party/gloo' 2024-06-26T06:32:57.3055547Z Entering 'third_party/googletest' 2024-06-26T06:32:57.3097777Z Entering 'third_party/ideep' 2024-06-26T06:32:57.3138765Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:32:57.3190047Z Entering 'third_party/ittapi' 2024-06-26T06:32:57.3231045Z Entering 'third_party/kineto' 2024-06-26T06:32:57.3272898Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:57.3314343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:57.3356033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:57.3396467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:57.3437119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:57.3476536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:57.3518174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:57.3559244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:57.3599527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:57.3641118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:57.3683676Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:57.3724274Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:57.3766457Z Entering 'third_party/mimalloc' 2024-06-26T06:32:57.3808246Z Entering 'third_party/nccl/nccl' 2024-06-26T06:32:57.3849202Z Entering 'third_party/nlohmann' 2024-06-26T06:32:57.3891973Z Entering 'third_party/onnx' 2024-06-26T06:32:57.3951294Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:57.3994548Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:57.4039050Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:32:57.4083534Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:57.4124765Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:57.4165518Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:57.4205461Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:57.4248410Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:57.4288278Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:57.4328889Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:57.4368318Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:57.4412099Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:57.4454320Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:57.4517105Z Entering 'third_party/pocketfft' 2024-06-26T06:32:57.4558884Z Entering 'third_party/protobuf' 2024-06-26T06:32:57.4603771Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:57.4644774Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:57.4687369Z Entering 'third_party/psimd' 2024-06-26T06:32:57.4729374Z Entering 'third_party/pthreadpool' 2024-06-26T06:32:57.4771101Z Entering 'third_party/pybind11' 2024-06-26T06:32:57.4813283Z Entering 'third_party/python-peachpy' 2024-06-26T06:32:57.4854246Z Entering 'third_party/sleef' 2024-06-26T06:32:57.4895609Z Entering 'third_party/tensorpipe' 2024-06-26T06:32:57.4937561Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:57.4977900Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:57.5017973Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:57.5059864Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:57.5100193Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:57.5159111Z ##[endgroup] 2024-06-26T06:32:57.5160325Z ##[group]Persisting credentials for submodules 2024-06-26T06:32:57.5164468Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T06:32:57.5476555Z Entering 'android/libs/fbjni' 2024-06-26T06:32:57.5527942Z Entering 'third_party/FP16' 2024-06-26T06:32:57.5579196Z Entering 'third_party/FXdiv' 2024-06-26T06:32:57.5630999Z Entering 'third_party/NNPACK' 2024-06-26T06:32:57.5682508Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:57.5733714Z Entering 'third_party/XNNPACK' 2024-06-26T06:32:57.5803213Z Entering 'third_party/benchmark' 2024-06-26T06:32:57.5855259Z Entering 'third_party/cpp-httplib' 2024-06-26T06:32:57.5907459Z Entering 'third_party/cpuinfo' 2024-06-26T06:32:57.5958201Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:32:57.6010935Z Entering 'third_party/cutlass' 2024-06-26T06:32:57.6072722Z Entering 'third_party/eigen' 2024-06-26T06:32:57.6127094Z Entering 'third_party/fbgemm' 2024-06-26T06:32:57.6178580Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:32:57.6230195Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:32:57.6280945Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:32:57.6339887Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:32:57.6391858Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:32:57.6443820Z Entering 'third_party/flatbuffers' 2024-06-26T06:32:57.6498189Z Entering 'third_party/fmt' 2024-06-26T06:32:57.6549375Z Entering 'third_party/foxi' 2024-06-26T06:32:57.6600750Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:57.6651548Z Entering 'third_party/gloo' 2024-06-26T06:32:57.6702906Z Entering 'third_party/googletest' 2024-06-26T06:32:57.6754263Z Entering 'third_party/ideep' 2024-06-26T06:32:57.6803486Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:32:57.6861570Z Entering 'third_party/ittapi' 2024-06-26T06:32:57.6913617Z Entering 'third_party/kineto' 2024-06-26T06:32:57.6965389Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:57.7015999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:57.7069208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:57.7119985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:57.7171574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:57.7221355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:57.7275365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:57.7327305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:57.7379331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:57.7432091Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:57.7485619Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:57.7536272Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:57.7587859Z Entering 'third_party/mimalloc' 2024-06-26T06:32:57.7638862Z Entering 'third_party/nccl/nccl' 2024-06-26T06:32:57.7690251Z Entering 'third_party/nlohmann' 2024-06-26T06:32:57.7742793Z Entering 'third_party/onnx' 2024-06-26T06:32:57.7811206Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:57.7862813Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:57.7916958Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:32:57.7969695Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:57.8020132Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:57.8071383Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:57.8121295Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:57.8173870Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:57.8224329Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:57.8275617Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:57.8325500Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:57.8379154Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:57.8431988Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:57.8502897Z Entering 'third_party/pocketfft' 2024-06-26T06:32:57.8554688Z Entering 'third_party/protobuf' 2024-06-26T06:32:57.8609443Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:57.8660275Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:57.8712999Z Entering 'third_party/psimd' 2024-06-26T06:32:57.8764156Z Entering 'third_party/pthreadpool' 2024-06-26T06:32:57.8816849Z Entering 'third_party/pybind11' 2024-06-26T06:32:57.8868755Z Entering 'third_party/python-peachpy' 2024-06-26T06:32:57.8919733Z Entering 'third_party/sleef' 2024-06-26T06:32:57.8970591Z Entering 'third_party/tensorpipe' 2024-06-26T06:32:57.9022382Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:57.9073770Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:57.9124322Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:57.9174560Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:57.9225452Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:57.9294207Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T06:32:57.9604169Z Entering 'android/libs/fbjni' 2024-06-26T06:32:57.9651512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T06:32:57.9670562Z Entering 'third_party/FP16' 2024-06-26T06:32:57.9718468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T06:32:57.9736445Z Entering 'third_party/FXdiv' 2024-06-26T06:32:57.9786305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T06:32:57.9804850Z Entering 'third_party/NNPACK' 2024-06-26T06:32:57.9853090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T06:32:57.9872532Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:57.9920682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T06:32:57.9939420Z Entering 'third_party/XNNPACK' 2024-06-26T06:32:57.9988636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T06:32:58.0025290Z Entering 'third_party/benchmark' 2024-06-26T06:32:58.0072871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:32:58.0091645Z Entering 'third_party/cpp-httplib' 2024-06-26T06:32:58.0140817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T06:32:58.0159787Z Entering 'third_party/cpuinfo' 2024-06-26T06:32:58.0207927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T06:32:58.0228010Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:32:58.0275596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T06:32:58.0293693Z Entering 'third_party/cutlass' 2024-06-26T06:32:58.0341339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T06:32:58.0368733Z Entering 'third_party/eigen' 2024-06-26T06:32:58.0417760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T06:32:58.0439214Z Entering 'third_party/fbgemm' 2024-06-26T06:32:58.0486811Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T06:32:58.0506015Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:32:58.0553825Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T06:32:58.0571811Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:32:58.0618984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T06:32:58.0637439Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:32:58.0684701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T06:32:58.0710911Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:32:58.0758383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.0776409Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:32:58.0823814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T06:32:58.0843373Z Entering 'third_party/flatbuffers' 2024-06-26T06:32:58.0891496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T06:32:58.0913624Z Entering 'third_party/fmt' 2024-06-26T06:32:58.0960732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T06:32:58.0979417Z Entering 'third_party/foxi' 2024-06-26T06:32:58.1027960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T06:32:58.1046451Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:58.1093767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T06:32:58.1112927Z Entering 'third_party/gloo' 2024-06-26T06:32:58.1161071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T06:32:58.1179830Z Entering 'third_party/googletest' 2024-06-26T06:32:58.1227807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.1246644Z Entering 'third_party/ideep' 2024-06-26T06:32:58.1295382Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T06:32:58.1314756Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:32:58.1361617Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T06:32:58.1388033Z Entering 'third_party/ittapi' 2024-06-26T06:32:58.1434949Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T06:32:58.1453146Z Entering 'third_party/kineto' 2024-06-26T06:32:58.1501189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T06:32:58.1520094Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:58.1567995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T06:32:58.1586153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:58.1634503Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T06:32:58.1653759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:58.1701844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T06:32:58.1720285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:58.1768199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T06:32:58.1787136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:58.1835363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T06:32:58.1852013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:58.1899982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T06:32:58.1919643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:58.1967862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T06:32:58.1987048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:58.2035636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.2053538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:58.2101736Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T06:32:58.2121588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:58.2169814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T06:32:58.2189519Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:58.2237176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T06:32:58.2255919Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:58.2304417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.2324755Z Entering 'third_party/mimalloc' 2024-06-26T06:32:58.2372487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T06:32:58.2392145Z Entering 'third_party/nccl/nccl' 2024-06-26T06:32:58.2439945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T06:32:58.2458660Z Entering 'third_party/nlohmann' 2024-06-26T06:32:58.2506570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T06:32:58.2526230Z Entering 'third_party/onnx' 2024-06-26T06:32:58.2574710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T06:32:58.2610824Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:58.2659717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:32:58.2678210Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:58.2726560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:32:58.2748043Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:32:58.2796356Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T06:32:58.2816403Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:58.2864284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:32:58.2884185Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:58.2930923Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.2949966Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:58.2997094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T06:32:58.3015201Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:58.3063012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T06:32:58.3083524Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:58.3130638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T06:32:58.3149172Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:58.3196001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T06:32:58.3213767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:58.3261501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T06:32:58.3279035Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:58.3326911Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T06:32:58.3347826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:58.3396071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T06:32:58.3415458Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:58.3463012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T06:32:58.3502263Z Entering 'third_party/pocketfft' 2024-06-26T06:32:58.3550778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T06:32:58.3569151Z Entering 'third_party/protobuf' 2024-06-26T06:32:58.3617406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T06:32:58.3639569Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:58.3687186Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:32:58.3706150Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:58.3753341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.3772962Z Entering 'third_party/psimd' 2024-06-26T06:32:58.3820999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T06:32:58.3839895Z Entering 'third_party/pthreadpool' 2024-06-26T06:32:58.3887859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T06:32:58.3906821Z Entering 'third_party/pybind11' 2024-06-26T06:32:58.3954319Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:32:58.3972393Z Entering 'third_party/python-peachpy' 2024-06-26T06:32:58.4019770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T06:32:58.4038473Z Entering 'third_party/sleef' 2024-06-26T06:32:58.4085939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T06:32:58.4104876Z Entering 'third_party/tensorpipe' 2024-06-26T06:32:58.4152536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T06:32:58.4170642Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:58.4217941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:32:58.4236154Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:58.4284228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T06:32:58.4301255Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:58.4348279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T06:32:58.4366270Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:58.4413856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:32:58.4431121Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:58.4479035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T06:32:58.5590646Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T06:32:58.5904107Z Entering 'android/libs/fbjni' 2024-06-26T06:32:58.5946547Z Entering 'third_party/FP16' 2024-06-26T06:32:58.5988438Z Entering 'third_party/FXdiv' 2024-06-26T06:32:58.6030104Z Entering 'third_party/NNPACK' 2024-06-26T06:32:58.6072915Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:58.6114861Z Entering 'third_party/XNNPACK' 2024-06-26T06:32:58.6174053Z Entering 'third_party/benchmark' 2024-06-26T06:32:58.6215642Z Entering 'third_party/cpp-httplib' 2024-06-26T06:32:58.6259389Z Entering 'third_party/cpuinfo' 2024-06-26T06:32:58.6301658Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:32:58.6344415Z Entering 'third_party/cutlass' 2024-06-26T06:32:58.6394590Z Entering 'third_party/eigen' 2024-06-26T06:32:58.6438788Z Entering 'third_party/fbgemm' 2024-06-26T06:32:58.6480154Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:32:58.6521324Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:32:58.6562526Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:32:58.6610278Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:32:58.6650790Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:32:58.6692448Z Entering 'third_party/flatbuffers' 2024-06-26T06:32:58.6737656Z Entering 'third_party/fmt' 2024-06-26T06:32:58.6780632Z Entering 'third_party/foxi' 2024-06-26T06:32:58.6822810Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:58.6864270Z Entering 'third_party/gloo' 2024-06-26T06:32:58.6906561Z Entering 'third_party/googletest' 2024-06-26T06:32:58.6948799Z Entering 'third_party/ideep' 2024-06-26T06:32:58.6990678Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:32:58.7038436Z Entering 'third_party/ittapi' 2024-06-26T06:32:58.7079756Z Entering 'third_party/kineto' 2024-06-26T06:32:58.7121859Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:58.7163757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:58.7206939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:58.7248825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:58.7289702Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:58.7330479Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:58.7373218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:58.7415207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:58.7456709Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:58.7499620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:58.7543431Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:58.7583816Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:58.7626395Z Entering 'third_party/mimalloc' 2024-06-26T06:32:58.7669714Z Entering 'third_party/nccl/nccl' 2024-06-26T06:32:58.7711922Z Entering 'third_party/nlohmann' 2024-06-26T06:32:58.7755016Z Entering 'third_party/onnx' 2024-06-26T06:32:58.7812865Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:58.7855633Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:58.7899197Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:32:58.7944116Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:58.7984676Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:58.8025177Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:58.8065333Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:58.8106947Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:58.8147619Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:58.8188129Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:58.8230989Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:58.8273807Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:58.8316374Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:58.8378607Z Entering 'third_party/pocketfft' 2024-06-26T06:32:58.8420696Z Entering 'third_party/protobuf' 2024-06-26T06:32:58.8465964Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:58.8507439Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:58.8550607Z Entering 'third_party/psimd' 2024-06-26T06:32:58.8592062Z Entering 'third_party/pthreadpool' 2024-06-26T06:32:58.8633582Z Entering 'third_party/pybind11' 2024-06-26T06:32:58.8675256Z Entering 'third_party/python-peachpy' 2024-06-26T06:32:58.8716443Z Entering 'third_party/sleef' 2024-06-26T06:32:58.8759207Z Entering 'third_party/tensorpipe' 2024-06-26T06:32:58.8800829Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:58.8842110Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:58.8883201Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:58.8923632Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:58.8963993Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:58.9023562Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T06:32:58.9334682Z Entering 'android/libs/fbjni' 2024-06-26T06:32:58.9378438Z Entering 'third_party/FP16' 2024-06-26T06:32:58.9420362Z Entering 'third_party/FXdiv' 2024-06-26T06:32:58.9462681Z Entering 'third_party/NNPACK' 2024-06-26T06:32:58.9506511Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:58.9548757Z Entering 'third_party/XNNPACK' 2024-06-26T06:32:58.9608676Z Entering 'third_party/benchmark' 2024-06-26T06:32:58.9651574Z Entering 'third_party/cpp-httplib' 2024-06-26T06:32:58.9692509Z Entering 'third_party/cpuinfo' 2024-06-26T06:32:58.9734897Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:32:58.9777868Z Entering 'third_party/cutlass' 2024-06-26T06:32:58.9829004Z Entering 'third_party/eigen' 2024-06-26T06:32:58.9872932Z Entering 'third_party/fbgemm' 2024-06-26T06:32:58.9915330Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:32:58.9957175Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:32:58.9999011Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:32:59.0048435Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:32:59.0089553Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:32:59.0132953Z Entering 'third_party/flatbuffers' 2024-06-26T06:32:59.0177197Z Entering 'third_party/fmt' 2024-06-26T06:32:59.0219241Z Entering 'third_party/foxi' 2024-06-26T06:32:59.0260633Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:59.0301650Z Entering 'third_party/gloo' 2024-06-26T06:32:59.0343868Z Entering 'third_party/googletest' 2024-06-26T06:32:59.0384677Z Entering 'third_party/ideep' 2024-06-26T06:32:59.0425729Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:32:59.0475358Z Entering 'third_party/ittapi' 2024-06-26T06:32:59.0516779Z Entering 'third_party/kineto' 2024-06-26T06:32:59.0557880Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:32:59.0599814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:32:59.0642819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:32:59.0691131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:32:59.0733955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:32:59.0775806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:32:59.0820484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:32:59.0862808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:32:59.0906001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:32:59.0949456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:32:59.0993458Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:32:59.1035560Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:32:59.1079003Z Entering 'third_party/mimalloc' 2024-06-26T06:32:59.1123854Z Entering 'third_party/nccl/nccl' 2024-06-26T06:32:59.1168226Z Entering 'third_party/nlohmann' 2024-06-26T06:32:59.1211699Z Entering 'third_party/onnx' 2024-06-26T06:32:59.1273187Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:32:59.1315480Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:32:59.1360078Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:32:59.1404483Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:32:59.1446782Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:32:59.1490135Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:32:59.1533035Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:32:59.1576595Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:32:59.1617858Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:32:59.1658945Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:32:59.1700621Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:32:59.1743898Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:32:59.1788008Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:32:59.1853252Z Entering 'third_party/pocketfft' 2024-06-26T06:32:59.1896639Z Entering 'third_party/protobuf' 2024-06-26T06:32:59.1943769Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:32:59.2048734Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:32:59.2178751Z Entering 'third_party/psimd' 2024-06-26T06:32:59.2233504Z Entering 'third_party/pthreadpool' 2024-06-26T06:32:59.2276324Z Entering 'third_party/pybind11' 2024-06-26T06:32:59.2320215Z Entering 'third_party/python-peachpy' 2024-06-26T06:32:59.2364089Z Entering 'third_party/sleef' 2024-06-26T06:32:59.2408434Z Entering 'third_party/tensorpipe' 2024-06-26T06:32:59.2451408Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:32:59.2495200Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:32:59.2536627Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:32:59.2578693Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:32:59.2619189Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:32:59.2676742Z ##[endgroup] 2024-06-26T06:32:59.2720759Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T06:32:59.2748770Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T06:32:59.2979021Z Prepare all required actions 2024-06-26T06:32:59.2979614Z Getting action download info 2024-06-26T06:32:59.4546501Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T06:32:59.4547035Z env: 2024-06-26T06:32:59.4547370Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:59.4547775Z ##[endgroup] 2024-06-26T06:32:59.4758085Z ##[group]Run set -euo pipefail 2024-06-26T06:32:59.4758634Z set -euo pipefail 2024-06-26T06:32:59.4759101Z function get_ec2_metadata() { 2024-06-26T06:32:59.4759760Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T06:32:59.4761102Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T06:32:59.4762063Z  category=$1 2024-06-26T06:32:59.4762719Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T06:32:59.4763505Z  runner_name_str=i-02a7758459db2295e 2024-06-26T06:32:59.4764121Z  if [[ -f /.inarc ]]; then 2024-06-26T06:32:59.4764736Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T06:32:59.4765417Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T06:32:59.4766274Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T06:32:59.4767030Z  else 2024-06-26T06:32:59.4767617Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T06:32:59.4768330Z  fi 2024-06-26T06:32:59.4768674Z } 2024-06-26T06:32:59.4769087Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T06:32:59.4769818Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T06:32:59.4770623Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T06:32:59.4771303Z echo "system info $(uname -a)" 2024-06-26T06:32:59.4779883Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:32:59.4780497Z env: 2024-06-26T06:32:59.4780893Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:59.4781325Z ##[endgroup] 2024-06-26T06:32:59.4888682Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T06:32:59.4956987Z instance-id: i-02a7758459db2295e 2024-06-26T06:32:59.5023661Z instance-type: g3.8xlarge 2024-06-26T06:32:59.5032508Z system info Linux ip-10-0-44-37.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T06:32:59.5082432Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T06:32:59.5083612Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T06:32:59.5092302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:32:59.5092921Z env: 2024-06-26T06:32:59.5093235Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:59.5093661Z ##[endgroup] 2024-06-26T06:32:59.5396549Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T06:32:59.5397311Z if systemctl is-active --quiet docker; then 2024-06-26T06:32:59.5397980Z  echo "Docker daemon is running..."; 2024-06-26T06:32:59.5398532Z else 2024-06-26T06:32:59.5399119Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T06:32:59.5399839Z fi 2024-06-26T06:32:59.5407784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:32:59.5408385Z env: 2024-06-26T06:32:59.5408723Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:59.5409150Z ##[endgroup] 2024-06-26T06:32:59.5458599Z Docker daemon is running... 2024-06-26T06:32:59.5649524Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:32:59.5650237Z with: 2024-06-26T06:32:59.5650562Z shell: bash 2024-06-26T06:32:59.5650902Z timeout_minutes: 5 2024-06-26T06:32:59.5651295Z max_attempts: 3 2024-06-26T06:32:59.5651674Z retry_wait_seconds: 30 2024-06-26T06:32:59.5653612Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T06:32:59.5655472Z polling_interval_seconds: 1 2024-06-26T06:32:59.5655934Z warning_on_retry: true 2024-06-26T06:32:59.5656365Z continue_on_error: false 2024-06-26T06:32:59.5656758Z env: 2024-06-26T06:32:59.5657089Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:59.5657519Z AWS_RETRY_MODE: standard 2024-06-26T06:32:59.5657929Z AWS_MAX_ATTEMPTS: 5 2024-06-26T06:32:59.5658337Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T06:32:59.5658993Z ##[endgroup] 2024-06-26T06:33:00.5845653Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:33:00.5846708Z Configure a credential helper to remove this warning. See 2024-06-26T06:33:00.5847782Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:33:00.5848459Z 2024-06-26T06:33:00.5848638Z Login Succeeded 2024-06-26T06:33:00.6391528Z Command completed after 1 attempt(s). 2024-06-26T06:33:00.6450056Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:33:00.6450965Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:33:00.6451789Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:33:00.6460632Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:00.6461265Z env: 2024-06-26T06:33:00.6461613Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.6462025Z ##[endgroup] 2024-06-26T06:33:00.6545115Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:33:00.6546079Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:33:00.6546816Z # shellcheck disable=SC2046 2024-06-26T06:33:00.6547384Z docker stop $(docker ps -q) || true 2024-06-26T06:33:00.6547964Z # Prune all of the docker images 2024-06-26T06:33:00.6548523Z docker system prune -af 2024-06-26T06:33:00.6556528Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:00.6557132Z env: 2024-06-26T06:33:00.6557477Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.6557905Z ##[endgroup] 2024-06-26T06:33:00.6897651Z "docker stop" requires at least 1 argument. 2024-06-26T06:33:00.6898359Z See 'docker stop --help'. 2024-06-26T06:33:00.6898675Z 2024-06-26T06:33:00.6898957Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T06:33:00.6901735Z 2024-06-26T06:33:00.6902701Z Stop one or more running containers 2024-06-26T06:33:00.7085090Z Total reclaimed space: 0B 2024-06-26T06:33:00.7121223Z ##[group]Run set +e 2024-06-26T06:33:00.7121669Z set +e 2024-06-26T06:33:00.7122023Z set -x 2024-06-26T06:33:00.7122376Z  2024-06-26T06:33:00.7122765Z PT_DOMAIN=download.pytorch.org 2024-06-26T06:33:00.7123745Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T06:33:00.7125110Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T06:33:00.7126055Z # one is returned at random 2024-06-26T06:33:00.7126703Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T06:33:00.7127342Z  2024-06-26T06:33:00.7127716Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T06:33:00.7128462Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T06:33:00.7129413Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T06:33:00.7130114Z  2024-06-26T06:33:00.7130504Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T06:33:00.7131166Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T06:33:00.7131798Z  exit 1 2024-06-26T06:33:00.7132182Z  fi 2024-06-26T06:33:00.7132516Z fi 2024-06-26T06:33:00.7133025Z  2024-06-26T06:33:00.7133486Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T06:33:00.7134103Z  # Clean up any old records first 2024-06-26T06:33:00.7134727Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T06:33:00.7135287Z fi 2024-06-26T06:33:00.7135607Z  2024-06-26T06:33:00.7136134Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T06:33:00.7136815Z cat /etc/hosts 2024-06-26T06:33:00.7145012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:00.7145840Z env: 2024-06-26T06:33:00.7146177Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.7146584Z ##[endgroup] 2024-06-26T06:33:00.7170673Z + PT_DOMAIN=download.pytorch.org 2024-06-26T06:33:00.7176401Z ++ dig -4 +short download.pytorch.org 2024-06-26T06:33:00.7206739Z ++ tail -n1 2024-06-26T06:33:00.7535407Z + RESOLVED_IP=18.160.18.20 2024-06-26T06:33:00.7535938Z + '[' -z 18.160.18.20 ']' 2024-06-26T06:33:00.7536490Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T06:33:00.7545212Z 18.160.10.28 download.pytorch.org 2024-06-26T06:33:00.7546283Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T06:33:00.7697302Z + echo '18.160.18.20 download.pytorch.org' 2024-06-26T06:33:00.7697927Z + sudo tee -a /etc/hosts 2024-06-26T06:33:00.7959048Z 18.160.18.20 download.pytorch.org 2024-06-26T06:33:00.7974923Z + cat /etc/hosts 2024-06-26T06:33:00.7981559Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T06:33:00.7996326Z ::1 localhost6 localhost6.localdomain6 2024-06-26T06:33:00.7997037Z 18.160.18.20 download.pytorch.org 2024-06-26T06:33:00.8201445Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T06:33:00.8202236Z with: 2024-06-26T06:33:00.8203432Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8204821Z docker-build-dir: .ci/docker 2024-06-26T06:33:00.8205287Z working-directory: . 2024-06-26T06:33:00.8205844Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:00.8206504Z force-push: false 2024-06-26T06:33:00.8206871Z env: 2024-06-26T06:33:00.8207181Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.8207601Z ##[endgroup] 2024-06-26T06:33:00.8233579Z ##[group]Run set -ex 2024-06-26T06:33:00.8234047Z set -ex 2024-06-26T06:33:00.8234407Z  2024-06-26T06:33:00.8235077Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T06:33:00.8236366Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T06:33:00.8237362Z # job could then download the pre-built image as usual 2024-06-26T06:33:00.8238284Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T06:33:00.8239126Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8239921Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8240601Z  2024-06-26T06:33:00.8241231Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T06:33:00.8242010Z  exit 0 2024-06-26T06:33:00.8242359Z else 2024-06-26T06:33:00.8242809Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8243370Z fi 2024-06-26T06:33:00.8243685Z  2024-06-26T06:33:00.8244251Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T06:33:00.8245334Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T06:33:00.8246291Z  # use it as it is, but first let's extract the tag 2024-06-26T06:33:00.8247167Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T06:33:00.8248074Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8248946Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8249624Z else 2024-06-26T06:33:00.8250154Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T06:33:00.8250977Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8252078Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8253235Z fi 2024-06-26T06:33:00.8261034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:00.8261638Z env: 2024-06-26T06:33:00.8261972Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.8262695Z REPO_NAME: pytorch 2024-06-26T06:33:00.8263935Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8265298Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T06:33:00.8265907Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:00.8266550Z ##[endgroup] 2024-06-26T06:33:00.8292237Z + [[ ! -d .ci/docker ]] 2024-06-26T06:33:00.8292734Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T06:33:00.8293202Z + echo skip=false 2024-06-26T06:33:00.8295327Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T06:33:00.8299838Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8301290Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T06:33:00.8310009Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8310855Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8312543Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8348674Z ##[group]Run set +e 2024-06-26T06:33:00.8349117Z set +e 2024-06-26T06:33:00.8349484Z set -x 2024-06-26T06:33:00.8349819Z  2024-06-26T06:33:00.8350152Z login() { 2024-06-26T06:33:00.8350971Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T06:33:00.8351890Z } 2024-06-26T06:33:00.8352209Z  2024-06-26T06:33:00.8352551Z retry () { 2024-06-26T06:33:00.8353012Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T06:33:00.8353562Z } 2024-06-26T06:33:00.8353891Z  2024-06-26T06:33:00.8354253Z retry login "${DOCKER_REGISTRY}" 2024-06-26T06:33:00.8354765Z  2024-06-26T06:33:00.8355351Z # Check if image already exists, if it does then skip building it 2024-06-26T06:33:00.8356232Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T06:33:00.8356847Z  exit 0 2024-06-26T06:33:00.8357215Z fi 2024-06-26T06:33:00.8357550Z  2024-06-26T06:33:00.8358150Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T06:33:00.8359186Z # be empty. The default action would be to continue rebuild the image 2024-06-26T06:33:00.8360108Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T06:33:00.8360945Z  # if we're on the base branch then use the parent commit 2024-06-26T06:33:00.8361687Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T06:33:00.8362219Z else 2024-06-26T06:33:00.8362784Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T06:33:00.8363645Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T06:33:00.8364273Z fi 2024-06-26T06:33:00.8364595Z  2024-06-26T06:33:00.8364982Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T06:33:00.8365609Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8366171Z  2024-06-26T06:33:00.8366966Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T06:33:00.8367929Z  exit 0 2024-06-26T06:33:00.8368295Z fi 2024-06-26T06:33:00.8368613Z  2024-06-26T06:33:00.8369321Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T06:33:00.8370551Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T06:33:00.8371555Z  exit 1 2024-06-26T06:33:00.8371923Z fi 2024-06-26T06:33:00.8372260Z  2024-06-26T06:33:00.8372845Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T06:33:00.8374044Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T06:33:00.8375140Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T06:33:00.8376352Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T06:33:00.8377724Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T06:33:00.8378552Z fi 2024-06-26T06:33:00.8378893Z  2024-06-26T06:33:00.8379434Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:33:00.8387740Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:00.8388366Z env: 2024-06-26T06:33:00.8388685Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:00.8389131Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T06:33:00.8389697Z BASE_REVISION: 4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:33:00.8391105Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8392516Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:00.8393260Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:00.8393907Z ##[endgroup] 2024-06-26T06:33:00.8418459Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:00.8419268Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:00.8420748Z + aws ecr get-login-password --region us-east-1 2024-06-26T06:33:00.8422152Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:01.3156999Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:33:01.3158062Z Configure a credential helper to remove this warning. See 2024-06-26T06:33:01.3159123Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:33:01.3159793Z 2024-06-26T06:33:01.3159944Z Login Succeeded 2024-06-26T06:33:01.3170878Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:01.5373157Z { 2024-06-26T06:33:01.5373616Z "schemaVersion": 2, 2024-06-26T06:33:01.5374337Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T06:33:01.5375073Z "config": { 2024-06-26T06:33:01.5375665Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T06:33:01.5376361Z "size": 47470, 2024-06-26T06:33:01.5377039Z "digest": "sha256:1fde65d68e1ba467b7c461d097349da1965328db5e6e5252ccb25702fcc3b83e" 2024-06-26T06:33:01.5377803Z }, 2024-06-26T06:33:01.5378119Z "layers": [ 2024-06-26T06:33:01.5378828Z { 2024-06-26T06:33:01.5379369Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5380066Z "size": 28580681, 2024-06-26T06:33:01.5380813Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T06:33:01.5381581Z }, 2024-06-26T06:33:01.5381891Z { 2024-06-26T06:33:01.5382853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5383557Z "size": 7943353, 2024-06-26T06:33:01.5384246Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-06-26T06:33:01.5385030Z }, 2024-06-26T06:33:01.5385310Z { 2024-06-26T06:33:01.5386263Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5386968Z "size": 56226389, 2024-06-26T06:33:01.5387636Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-06-26T06:33:01.5388416Z }, 2024-06-26T06:33:01.5388720Z { 2024-06-26T06:33:01.5389235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5389935Z "size": 187, 2024-06-26T06:33:01.5390591Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-06-26T06:33:01.5391369Z }, 2024-06-26T06:33:01.5391667Z { 2024-06-26T06:33:01.5392206Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5393089Z + exit 0 2024-06-26T06:33:01.5393416Z "size": 6885, 2024-06-26T06:33:01.5394101Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-06-26T06:33:01.5394884Z }, 2024-06-26T06:33:01.5395167Z { 2024-06-26T06:33:01.5395888Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5396612Z "size": 1383662475, 2024-06-26T06:33:01.5397265Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-06-26T06:33:01.5398027Z }, 2024-06-26T06:33:01.5398361Z { 2024-06-26T06:33:01.5398876Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5399572Z "size": 62282, 2024-06-26T06:33:01.5400237Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-06-26T06:33:01.5400977Z }, 2024-06-26T06:33:01.5401275Z { 2024-06-26T06:33:01.5401808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5402477Z "size": 1684, 2024-06-26T06:33:01.5403162Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-06-26T06:33:01.5403939Z }, 2024-06-26T06:33:01.5404223Z { 2024-06-26T06:33:01.5404755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5405466Z "size": 1522, 2024-06-26T06:33:01.5406121Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-06-26T06:33:01.5406886Z }, 2024-06-26T06:33:01.5407180Z { 2024-06-26T06:33:01.5407693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5408386Z "size": 2445884909, 2024-06-26T06:33:01.5409070Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-06-26T06:33:01.5409799Z }, 2024-06-26T06:33:01.5410100Z { 2024-06-26T06:33:01.5410633Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5411306Z "size": 85882, 2024-06-26T06:33:01.5411978Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-06-26T06:33:01.5412747Z }, 2024-06-26T06:33:01.5413024Z { 2024-06-26T06:33:01.5413558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5414262Z "size": 1822, 2024-06-26T06:33:01.5414932Z "digest": "sha256:bb93d41d4cb4616eabdd81918bbf6c8673164e995dd0f0cc6d05989fdce3df21" 2024-06-26T06:33:01.5415711Z }, 2024-06-26T06:33:01.5416009Z { 2024-06-26T06:33:01.5416519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5417213Z "size": 654858224, 2024-06-26T06:33:01.5417918Z "digest": "sha256:4b89839dd06631d80ca192bae8be4ea835acfde539e7e775339edcce9714c418" 2024-06-26T06:33:01.5418676Z }, 2024-06-26T06:33:01.5418975Z { 2024-06-26T06:33:01.5419508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5420181Z "size": 546, 2024-06-26T06:33:01.5420861Z "digest": "sha256:caa9986c765baa228b63ef2e4e24be796d1aa0f41e20f1a9ccc44423d8a9a3f4" 2024-06-26T06:33:01.5421647Z }, 2024-06-26T06:33:01.5421925Z { 2024-06-26T06:33:01.5422871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5423801Z "size": 1284, 2024-06-26T06:33:01.5424507Z "digest": "sha256:e17c09bdf243963c1185f01eca2a5ed4ae8e627181e77073452ea8a2811b661d" 2024-06-26T06:33:01.5425319Z }, 2024-06-26T06:33:01.5425635Z { 2024-06-26T06:33:01.5426179Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5473678Z "size": 484, 2024-06-26T06:33:01.5474402Z "digest": "sha256:bd2e32de75c90c1ed985c413bb08813271eeb40a7cb9c926e261d381c6fb4d82" 2024-06-26T06:33:01.5475246Z }, 2024-06-26T06:33:01.5475557Z { 2024-06-26T06:33:01.5476255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5476948Z "size": 91710828, 2024-06-26T06:33:01.5477662Z "digest": "sha256:559a1daa0f9885fe07609ca2641d3e9e3d4d6aedca6c4d36ad90c1c765582bf0" 2024-06-26T06:33:01.5478582Z }, 2024-06-26T06:33:01.5478886Z { 2024-06-26T06:33:01.5479410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5480113Z "size": 3235, 2024-06-26T06:33:01.5481091Z "digest": "sha256:d96cec7a425ccddeae1efd3e0f73355515e86e146cbaab2da9739cd651741068" 2024-06-26T06:33:01.5481919Z }, 2024-06-26T06:33:01.5482215Z { 2024-06-26T06:33:01.5482747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5483424Z "size": 1905, 2024-06-26T06:33:01.5484106Z "digest": "sha256:1db6fc6c25902dd64b1dad171454f5b492ecb8895b4f8b6270de2a65750cfe12" 2024-06-26T06:33:01.5484886Z }, 2024-06-26T06:33:01.5485165Z { 2024-06-26T06:33:01.5485698Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5486397Z "size": 700, 2024-06-26T06:33:01.5487064Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:33:01.5487851Z }, 2024-06-26T06:33:01.5488151Z { 2024-06-26T06:33:01.5488659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5489358Z "size": 2812517131, 2024-06-26T06:33:01.5490065Z "digest": "sha256:a05e0234ee85b8358a1b4c4bf7d9d7c4bc38e991a22895c0f9ffcbc709645fd9" 2024-06-26T06:33:01.5490840Z }, 2024-06-26T06:33:01.5491138Z { 2024-06-26T06:33:01.5491673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5492343Z "size": 381, 2024-06-26T06:33:01.5493011Z "digest": "sha256:44f27f91d8fa036ce2af809752cba0d79d7da4ffe91d4981827319d759d6a230" 2024-06-26T06:33:01.5493792Z }, 2024-06-26T06:33:01.5494070Z { 2024-06-26T06:33:01.5494596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5495291Z "size": 11891, 2024-06-26T06:33:01.5495947Z "digest": "sha256:0a0ded10616759d17568ed6ae0f22ebfe503413d2d4710223b5852c8eeb131b3" 2024-06-26T06:33:01.5496745Z }, 2024-06-26T06:33:01.5497031Z { 2024-06-26T06:33:01.5497560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5498258Z "size": 803, 2024-06-26T06:33:01.5498917Z "digest": "sha256:0c62bfb80b7cdf32a713040cc54d11d45eec4105a51976da8b88f6e8dd79bb49" 2024-06-26T06:33:01.5499713Z }, 2024-06-26T06:33:01.5500014Z { 2024-06-26T06:33:01.5500533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5501235Z "size": 106, 2024-06-26T06:33:01.5501917Z "digest": "sha256:0c8d1e7f32823458cc5bf0e51b63e2abb70878dea21687edffcf738af41051b6" 2024-06-26T06:33:01.5503112Z }, 2024-06-26T06:33:01.5503420Z { 2024-06-26T06:33:01.5503954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5504634Z "size": 503, 2024-06-26T06:33:01.5505274Z "digest": "sha256:846366e78885c05844e15926128f2d3113c617522f700e485df868fbbba1c876" 2024-06-26T06:33:01.5506017Z }, 2024-06-26T06:33:01.5506301Z { 2024-06-26T06:33:01.5506831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5507533Z "size": 121477247, 2024-06-26T06:33:01.5508219Z "digest": "sha256:efa11c713d18f2a9dcf7901ba63df5b9766293ca7c7824e5f3c089c960f45162" 2024-06-26T06:33:01.5509179Z }, 2024-06-26T06:33:01.5509476Z { 2024-06-26T06:33:01.5509998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5510695Z "size": 109, 2024-06-26T06:33:01.5511377Z "digest": "sha256:32f0221befebf94bb7f0f81bdaef0f566e32695e7a98cb9020b555956fc9b89b" 2024-06-26T06:33:01.5512139Z }, 2024-06-26T06:33:01.5512438Z { 2024-06-26T06:33:01.5512967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5513643Z "size": 490, 2024-06-26T06:33:01.5514315Z "digest": "sha256:a1d57c94ba33612a149ca2fea27363c0f960fd8516723c166d5b2783aabdaf16" 2024-06-26T06:33:01.5515086Z }, 2024-06-26T06:33:01.5515364Z { 2024-06-26T06:33:01.5515877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5516567Z "size": 295, 2024-06-26T06:33:01.5517205Z "digest": "sha256:3dc15441e6d000a89a95092afb061c4c3c36b263a5159e68b4385e453b793faf" 2024-06-26T06:33:01.5517966Z }, 2024-06-26T06:33:01.5518264Z { 2024-06-26T06:33:01.5518940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5519665Z "size": 103, 2024-06-26T06:33:01.5520329Z "digest": "sha256:668eadf990e86d6ac337335a56269ee7d21c3b7ab9d73a08b6234234070fcd70" 2024-06-26T06:33:01.5521079Z }, 2024-06-26T06:33:01.5521382Z { 2024-06-26T06:33:01.5521912Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5522589Z "size": 1473, 2024-06-26T06:33:01.5523266Z "digest": "sha256:fd5a832fe26e3c2c26c20395f500fc681ab28e9f9812e4b6b3e95aa15442103d" 2024-06-26T06:33:01.5524039Z }, 2024-06-26T06:33:01.5524320Z { 2024-06-26T06:33:01.5524851Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5525547Z "size": 446711131, 2024-06-26T06:33:01.5526232Z "digest": "sha256:b581c3782fd627bac2e6cc292144b92a21b499a07e2b598eaa2a48252e57a537" 2024-06-26T06:33:01.5527001Z }, 2024-06-26T06:33:01.5527300Z { 2024-06-26T06:33:01.5527815Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5528524Z "size": 160, 2024-06-26T06:33:01.5529175Z "digest": "sha256:3f42611c56360ba670f886b3c62442f9f556652382512f0153dbd888798fe690" 2024-06-26T06:33:01.5529912Z }, 2024-06-26T06:33:01.5530213Z { 2024-06-26T06:33:01.5530749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5531428Z "size": 564, 2024-06-26T06:33:01.5532109Z "digest": "sha256:fe8dd930eda117742ffbb011f0c821bd08f9d183b46a81a9dc5fc3fca741b8bf" 2024-06-26T06:33:01.5532897Z }, 2024-06-26T06:33:01.5533180Z { 2024-06-26T06:33:01.5533711Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5534409Z "size": 35807552, 2024-06-26T06:33:01.5535079Z "digest": "sha256:6a9711a8a50dc2f9871121744f60deeb06e436ea2282ab2ed3d153ed4f2360d7" 2024-06-26T06:33:01.5535845Z }, 2024-06-26T06:33:01.5536145Z { 2024-06-26T06:33:01.5536655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5537359Z "size": 104, 2024-06-26T06:33:01.5538034Z "digest": "sha256:76cd931663c9f6f95baad24c47a3f0125db05e116cc221dd313fb9f089af72d6" 2024-06-26T06:33:01.5538792Z }, 2024-06-26T06:33:01.5539091Z { 2024-06-26T06:33:01.5539620Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5540293Z "size": 426, 2024-06-26T06:33:01.5540953Z "digest": "sha256:c9ec95615433cc9fb674bbd2820228a5977f981d70ff677927b0caf42d8c56b8" 2024-06-26T06:33:01.5541724Z }, 2024-06-26T06:33:01.5542006Z { 2024-06-26T06:33:01.5542960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5543667Z "size": 20262060, 2024-06-26T06:33:01.5544335Z "digest": "sha256:7da33ab0319397119e259c5ee2c37d7c6cb854983c8a320c59746a27d4826ed2" 2024-06-26T06:33:01.5545129Z }, 2024-06-26T06:33:01.5545447Z { 2024-06-26T06:33:01.5546003Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5546883Z "size": 439, 2024-06-26T06:33:01.5547567Z "digest": "sha256:da956bbe6180c03f03d667edd26b2f9a43df083200e13847da28ad27a9f83489" 2024-06-26T06:33:01.5548348Z }, 2024-06-26T06:33:01.5548631Z { 2024-06-26T06:33:01.5549160Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5549851Z "size": 700, 2024-06-26T06:33:01.5550519Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:33:01.5551308Z }, 2024-06-26T06:33:01.5551610Z { 2024-06-26T06:33:01.5552122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5552820Z "size": 143, 2024-06-26T06:33:01.5553484Z "digest": "sha256:73e6b5984544e73e642e8a11fd1dddf14a05913257132ee47f7782b6fb519ee6" 2024-06-26T06:33:01.5554236Z }, 2024-06-26T06:33:01.5554538Z { 2024-06-26T06:33:01.5555066Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5555745Z "size": 135, 2024-06-26T06:33:01.5556544Z "digest": "sha256:e5a6ec0a41d77fcb1af98a4aa079b9219c47c2e5eb696b52883c390373b1af8e" 2024-06-26T06:33:01.5557432Z }, 2024-06-26T06:33:01.5557715Z { 2024-06-26T06:33:01.5558248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5558941Z "size": 32, 2024-06-26T06:33:01.5559594Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5560372Z }, 2024-06-26T06:33:01.5560671Z { 2024-06-26T06:33:01.5561182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5561884Z "size": 191, 2024-06-26T06:33:01.5562575Z "digest": "sha256:4b25a744b402ec8a8cdfd55f95c346bcf9178c450cd08e8bb2834efa41bb621e" 2024-06-26T06:33:01.5563336Z }, 2024-06-26T06:33:01.5563631Z { 2024-06-26T06:33:01.5564162Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5564834Z "size": 564, 2024-06-26T06:33:01.5565516Z "digest": "sha256:21f74857bd1744f0b55351469eec060cd60d78fac461e695f550de53cec8edc0" 2024-06-26T06:33:01.5566294Z }, 2024-06-26T06:33:01.5566584Z { 2024-06-26T06:33:01.5567118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5567811Z "size": 43163151, 2024-06-26T06:33:01.5568473Z "digest": "sha256:6c12f547d202783d251e0813abaeaad2b30ba464d28f5902171b637f829ab84a" 2024-06-26T06:33:01.5569235Z }, 2024-06-26T06:33:01.5569532Z { 2024-06-26T06:33:01.5570041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5570735Z "size": 106, 2024-06-26T06:33:01.5571412Z "digest": "sha256:ed4e5a4f8526fca89a9272c39328cb84dbec72a12e74d4abde50a706f2983b55" 2024-06-26T06:33:01.5572174Z }, 2024-06-26T06:33:01.5572470Z { 2024-06-26T06:33:01.5572997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5573672Z "size": 1212, 2024-06-26T06:33:01.5574345Z "digest": "sha256:6f09f741ead60ea209087bf9071ef1a6fcc299c42ce4b3a297c6d96a6393f67f" 2024-06-26T06:33:01.5575129Z }, 2024-06-26T06:33:01.5575409Z { 2024-06-26T06:33:01.5575947Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5576640Z "size": 700, 2024-06-26T06:33:01.5577303Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:33:01.5578090Z }, 2024-06-26T06:33:01.5578390Z { 2024-06-26T06:33:01.5578904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5579600Z "size": 138, 2024-06-26T06:33:01.5580281Z "digest": "sha256:411fce2db725b6d9a9d7b7f953cdd354ceef707e1dec7a2ef5115beba5a6788e" 2024-06-26T06:33:01.5581101Z }, 2024-06-26T06:33:01.5581400Z { 2024-06-26T06:33:01.5581931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5582901Z "size": 120, 2024-06-26T06:33:01.5583575Z "digest": "sha256:87f7902d12a4c612e9be6b4cceae10e66baf25332f924496d3b70f1967b83360" 2024-06-26T06:33:01.5584357Z }, 2024-06-26T06:33:01.5584787Z { 2024-06-26T06:33:01.5585324Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5586021Z "size": 1905185822, 2024-06-26T06:33:01.5586687Z "digest": "sha256:9777c07909ada379d44a538999c6492011749a4f0fcbd5dfee534795401df9d3" 2024-06-26T06:33:01.5587442Z }, 2024-06-26T06:33:01.5587741Z { 2024-06-26T06:33:01.5588253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5588949Z "size": 172, 2024-06-26T06:33:01.5589609Z "digest": "sha256:9755a952d3ecbc817b3290445710170b8c63a35f902a9bddec1837eee9a4db57" 2024-06-26T06:33:01.5590354Z }, 2024-06-26T06:33:01.5590653Z { 2024-06-26T06:33:01.5591181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5591857Z "size": 908, 2024-06-26T06:33:01.5592536Z "digest": "sha256:5ca5d2e8bebe5a5641fd35fc6585f4c83ae8ae4528b84682b276a04b9006faa2" 2024-06-26T06:33:01.5593320Z }, 2024-06-26T06:33:01.5593602Z { 2024-06-26T06:33:01.5594144Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5594957Z "size": 700, 2024-06-26T06:33:01.5595645Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:33:01.5596430Z }, 2024-06-26T06:33:01.5596727Z { 2024-06-26T06:33:01.5597239Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5597928Z "size": 135, 2024-06-26T06:33:01.5598601Z "digest": "sha256:d07af843db47bac709de490bf12e9f993080a666ddda8ee54210dd4b6526f97d" 2024-06-26T06:33:01.5599364Z }, 2024-06-26T06:33:01.5599663Z { 2024-06-26T06:33:01.5600190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5600866Z "size": 32, 2024-06-26T06:33:01.5601537Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5602314Z }, 2024-06-26T06:33:01.5602596Z { 2024-06-26T06:33:01.5603122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5603824Z "size": 157, 2024-06-26T06:33:01.5604497Z "digest": "sha256:dbbd7a4ca8469b1fc21ae5f549fc832e79c4dae36515cc0b594a4f8d6531832e" 2024-06-26T06:33:01.5605289Z }, 2024-06-26T06:33:01.5605589Z { 2024-06-26T06:33:01.5606105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5606795Z "size": 1841, 2024-06-26T06:33:01.5607470Z "digest": "sha256:6a18c61a96dd8f2f6cd96d48d62262059c9b024ffad4cf0df6a7d4c02f58d9e2" 2024-06-26T06:33:01.5608249Z }, 2024-06-26T06:33:01.5608528Z { 2024-06-26T06:33:01.5609053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5609729Z "size": 7529784, 2024-06-26T06:33:01.5610411Z "digest": "sha256:bd12222c02373373bcdd92c079d4d6e405dee01eb986422d608659c4bb0cd809" 2024-06-26T06:33:01.5611175Z }, 2024-06-26T06:33:01.5611453Z { 2024-06-26T06:33:01.5611984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5612691Z "size": 163, 2024-06-26T06:33:01.5613381Z "digest": "sha256:a5dfe6ba36fd2223ebbc9e888593b6341d5bd3b56b1674cbd9064f55c26de604" 2024-06-26T06:33:01.5614153Z }, 2024-06-26T06:33:01.5614451Z { 2024-06-26T06:33:01.5614962Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5615653Z "size": 7944, 2024-06-26T06:33:01.5616335Z "digest": "sha256:4b990eb76e8f2de191d0c2e639846adb8df24ce3380ec150ead1baa95b26d7ec" 2024-06-26T06:33:01.5617120Z }, 2024-06-26T06:33:01.5617403Z { 2024-06-26T06:33:01.5617933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5618628Z "size": 8069, 2024-06-26T06:33:01.5619291Z "digest": "sha256:d9df85e8d42263f8089be7b63aaf1e2bfc3207f8d81b1d79f9f86e96c17298ee" 2024-06-26T06:33:01.5620069Z }, 2024-06-26T06:33:01.5620369Z { 2024-06-26T06:33:01.5620884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5621575Z "size": 304, 2024-06-26T06:33:01.5622640Z "digest": "sha256:47d89094b1a7e9cc21b8db8ab9c66fefcc9bf7544c170508f41efb5d26e3ccd2" 2024-06-26T06:33:01.5623423Z }, 2024-06-26T06:33:01.5623721Z { 2024-06-26T06:33:01.5624250Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5624928Z "size": 7628836, 2024-06-26T06:33:01.5625608Z "digest": "sha256:94beee2c356a83536890159dc698da22baca90a4c8289622ed556f5a47947d79" 2024-06-26T06:33:01.5626372Z }, 2024-06-26T06:33:01.5626653Z { 2024-06-26T06:33:01.5627178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5627860Z "size": 108, 2024-06-26T06:33:01.5628497Z "digest": "sha256:322c644984f492ea6b5b3d0b9eaab0bdd8d29e156e25a60d475a7dbe0c08621f" 2024-06-26T06:33:01.5629257Z }, 2024-06-26T06:33:01.5629553Z { 2024-06-26T06:33:01.5630061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5630758Z "size": 54145774, 2024-06-26T06:33:01.5631442Z "digest": "sha256:b43ee04d974d5ef5236d381dd8e9d87677715edc695f235bb1e954b54a8f1143" 2024-06-26T06:33:01.5632322Z }, 2024-06-26T06:33:01.5632642Z { 2024-06-26T06:33:01.5633173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5633847Z "size": 474, 2024-06-26T06:33:01.5634516Z "digest": "sha256:40f95cbeec2c62a99f4548e3f7139c89986c010bce3742c85d203fdd5a120c4e" 2024-06-26T06:33:01.5635288Z }, 2024-06-26T06:33:01.5635570Z { 2024-06-26T06:33:01.5636088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5636778Z "size": 1373229352, 2024-06-26T06:33:01.5637475Z "digest": "sha256:4b2a3c41d6ae93889d0aed7b8b3e87eb334b8fb8aedb20bab9d7fea2759b7055" 2024-06-26T06:33:01.5638258Z }, 2024-06-26T06:33:01.5638544Z { 2024-06-26T06:33:01.5639057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5639748Z "size": 106, 2024-06-26T06:33:01.5640412Z "digest": "sha256:b2689866a0b9bc7b622498695ae245e7f4996dfa2db1311cff0b6fae0ff7e6e8" 2024-06-26T06:33:01.5641178Z }, 2024-06-26T06:33:01.5641455Z { 2024-06-26T06:33:01.5641952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5642612Z "size": 556, 2024-06-26T06:33:01.5643267Z "digest": "sha256:9b36f9d4ace6c70a0fddeca47d17eb19639fd636a6084e02555f7d862bf44c40" 2024-06-26T06:33:01.5644030Z }, 2024-06-26T06:33:01.5644308Z { 2024-06-26T06:33:01.5644835Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5645525Z "size": 36532084, 2024-06-26T06:33:01.5646186Z "digest": "sha256:a863a1c01885d8f3506984d7a7d8179273b3d29ed4b4e2dee41471a6437155f4" 2024-06-26T06:33:01.5646942Z }, 2024-06-26T06:33:01.5647235Z { 2024-06-26T06:33:01.5647746Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5648435Z "size": 111, 2024-06-26T06:33:01.5649109Z "digest": "sha256:b9b8edd214be18381ccf301a0b92a7853f7ecba0b219699cd74c9b2e064183fd" 2024-06-26T06:33:01.5649881Z }, 2024-06-26T06:33:01.5650212Z { 2024-06-26T06:33:01.5650750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5651432Z "size": 32, 2024-06-26T06:33:01.5652102Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5652876Z }, 2024-06-26T06:33:01.5653157Z { 2024-06-26T06:33:01.5653681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5654372Z "size": 32, 2024-06-26T06:33:01.5655022Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5655795Z }, 2024-06-26T06:33:01.5656092Z { 2024-06-26T06:33:01.5656599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5657292Z "size": 32, 2024-06-26T06:33:01.5657957Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5658718Z }, 2024-06-26T06:33:01.5659191Z { 2024-06-26T06:33:01.5659731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:33:01.5660412Z "size": 32, 2024-06-26T06:33:01.5661083Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:33:01.5661858Z } 2024-06-26T06:33:01.5662139Z ] 2024-06-26T06:33:01.5662665Z } 2024-06-26T06:33:01.5799085Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T06:33:01.5799707Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T06:33:01.5800387Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T06:33:01.5808797Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:01.5809422Z env: 2024-06-26T06:33:01.5809762Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:01.5811062Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:01.5812391Z ##[endgroup] 2024-06-26T06:33:01.5840814Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:01.5906002Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T06:33:01.5906742Z with: 2024-06-26T06:33:01.5907916Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:01.5909412Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:01.5910031Z env: 2024-06-26T06:33:01.5910362Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:01.5910780Z ##[endgroup] 2024-06-26T06:33:01.5933437Z ##[group]Run set -x 2024-06-26T06:33:01.5933856Z set -x 2024-06-26T06:33:01.5934224Z set +e 2024-06-26T06:33:01.5934577Z  2024-06-26T06:33:01.5934896Z login() { 2024-06-26T06:33:01.5935724Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T06:33:01.5936646Z } 2024-06-26T06:33:01.5936962Z  2024-06-26T06:33:01.5937356Z retry () { 2024-06-26T06:33:01.5937823Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T06:33:01.5938378Z } 2024-06-26T06:33:01.5938704Z  2024-06-26T06:33:01.5939059Z retry login "${DOCKER_REGISTRY}" 2024-06-26T06:33:01.5939575Z  2024-06-26T06:33:01.5939901Z set -e 2024-06-26T06:33:01.5940467Z # ignore output since only exit code is used for conditional 2024-06-26T06:33:01.5941348Z # only pull docker image if it's not available locally 2024-06-26T06:33:01.5942716Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T06:33:01.5943616Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T06:33:01.5944139Z fi 2024-06-26T06:33:01.5951548Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:33:01.5952182Z env: 2024-06-26T06:33:01.5952504Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:33:01.5953773Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:01.5955262Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:01.5955887Z ##[endgroup] 2024-06-26T06:33:01.5979564Z + set +e 2024-06-26T06:33:01.5980179Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:01.5981031Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:01.5983704Z + aws ecr get-login-password --region us-east-1 2024-06-26T06:33:01.5985489Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:33:02.0851109Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:33:02.0852185Z Configure a credential helper to remove this warning. See 2024-06-26T06:33:02.0853537Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:33:02.0854223Z 2024-06-26T06:33:02.0854388Z Login Succeeded 2024-06-26T06:33:02.0863809Z + set -e 2024-06-26T06:33:02.0865282Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:02.1041305Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:02.1043686Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:33:02.3218204Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-06-26T06:33:02.3220049Z 7a2c55901189: Pulling fs layer 2024-06-26T06:33:02.3220854Z 09d415c238d7: Pulling fs layer 2024-06-26T06:33:02.3221375Z 9fe6e2e61518: Pulling fs layer 2024-06-26T06:33:02.3222622Z 41f16248e682: Pulling fs layer 2024-06-26T06:33:02.3223428Z 95d7b7817039: Pulling fs layer 2024-06-26T06:33:02.3223942Z 8f6c90485347: Pulling fs layer 2024-06-26T06:33:02.3224398Z ab17245097e4: Pulling fs layer 2024-06-26T06:33:02.3225267Z dfecd7e9912b: Pulling fs layer 2024-06-26T06:33:02.3226174Z 464a8f745445: Pulling fs layer 2024-06-26T06:33:02.3226657Z d67c111fa588: Pulling fs layer 2024-06-26T06:33:02.3227127Z 0723abde19c0: Pulling fs layer 2024-06-26T06:33:02.3227595Z bb93d41d4cb4: Pulling fs layer 2024-06-26T06:33:02.3228108Z 4b89839dd066: Pulling fs layer 2024-06-26T06:33:02.3229017Z caa9986c765b: Pulling fs layer 2024-06-26T06:33:02.3229814Z 41f16248e682: Waiting 2024-06-26T06:33:02.3230219Z e17c09bdf243: Pulling fs layer 2024-06-26T06:33:02.3230697Z bd2e32de75c9: Pulling fs layer 2024-06-26T06:33:02.3231146Z 95d7b7817039: Waiting 2024-06-26T06:33:02.3231509Z ab17245097e4: Waiting 2024-06-26T06:33:02.3231920Z 559a1daa0f98: Pulling fs layer 2024-06-26T06:33:02.3232394Z 8f6c90485347: Waiting 2024-06-26T06:33:02.3232784Z d96cec7a425c: Pulling fs layer 2024-06-26T06:33:02.3233262Z 1db6fc6c2590: Pulling fs layer 2024-06-26T06:33:02.3233742Z ac522bc70345: Pulling fs layer 2024-06-26T06:33:02.3234623Z a05e0234ee85: Pulling fs layer 2024-06-26T06:33:02.3235576Z 44f27f91d8fa: Pulling fs layer 2024-06-26T06:33:02.3236500Z 0a0ded106167: Pulling fs layer 2024-06-26T06:33:02.3237353Z d67c111fa588: Waiting 2024-06-26T06:33:02.3238169Z 0c62bfb80b7c: Pulling fs layer 2024-06-26T06:33:02.3238723Z d96cec7a425c: Waiting 2024-06-26T06:33:02.3239106Z 1db6fc6c2590: Waiting 2024-06-26T06:33:02.3239499Z 464a8f745445: Waiting 2024-06-26T06:33:02.3239883Z 0c8d1e7f3282: Pulling fs layer 2024-06-26T06:33:02.3240330Z 559a1daa0f98: Waiting 2024-06-26T06:33:02.3240739Z 846366e78885: Pulling fs layer 2024-06-26T06:33:02.3241249Z 0723abde19c0: Waiting 2024-06-26T06:33:02.3241617Z caa9986c765b: Waiting 2024-06-26T06:33:02.3242009Z ac522bc70345: Waiting 2024-06-26T06:33:02.3242429Z efa11c713d18: Pulling fs layer 2024-06-26T06:33:02.3242864Z e17c09bdf243: Waiting 2024-06-26T06:33:02.3243267Z bd2e32de75c9: Waiting 2024-06-26T06:33:02.3243680Z 32f0221befeb: Pulling fs layer 2024-06-26T06:33:02.3244113Z a05e0234ee85: Waiting 2024-06-26T06:33:02.3244521Z a1d57c94ba33: Pulling fs layer 2024-06-26T06:33:02.3244978Z 4b89839dd066: Waiting 2024-06-26T06:33:02.3245345Z dfecd7e9912b: Waiting 2024-06-26T06:33:02.3245737Z 44f27f91d8fa: Waiting 2024-06-26T06:33:02.3246131Z 0c62bfb80b7c: Waiting 2024-06-26T06:33:02.3246523Z 3dc15441e6d0: Pulling fs layer 2024-06-26T06:33:02.3246989Z 668eadf990e8: Pulling fs layer 2024-06-26T06:33:02.3247462Z fd5a832fe26e: Pulling fs layer 2024-06-26T06:33:02.3247915Z b581c3782fd6: Pulling fs layer 2024-06-26T06:33:02.3248360Z 846366e78885: Waiting 2024-06-26T06:33:02.3248750Z 0c8d1e7f3282: Waiting 2024-06-26T06:33:02.3249119Z 0a0ded106167: Waiting 2024-06-26T06:33:02.3249525Z 3f42611c5636: Pulling fs layer 2024-06-26T06:33:02.3249996Z fe8dd930eda1: Pulling fs layer 2024-06-26T06:33:02.3250708Z 6a9711a8a50d: Pulling fs layer 2024-06-26T06:33:02.3251157Z fd5a832fe26e: Waiting 2024-06-26T06:33:02.3251578Z 76cd931663c9: Pulling fs layer 2024-06-26T06:33:02.3252009Z 3f42611c5636: Waiting 2024-06-26T06:33:02.3252397Z 3dc15441e6d0: Waiting 2024-06-26T06:33:02.3252813Z c9ec95615433: Pulling fs layer 2024-06-26T06:33:02.3253264Z 7da33ab03193: Pulling fs layer 2024-06-26T06:33:02.3253733Z da956bbe6180: Pulling fs layer 2024-06-26T06:33:02.3254186Z a1d57c94ba33: Waiting 2024-06-26T06:33:02.3254557Z bb93d41d4cb4: Waiting 2024-06-26T06:33:02.3254964Z 73e6b5984544: Pulling fs layer 2024-06-26T06:33:02.3255409Z da956bbe6180: Waiting 2024-06-26T06:33:02.3255783Z 32f0221befeb: Waiting 2024-06-26T06:33:02.3256172Z efa11c713d18: Waiting 2024-06-26T06:33:02.3256584Z e5a6ec0a41d7: Pulling fs layer 2024-06-26T06:33:02.3257035Z 4f4fb700ef54: Pulling fs layer 2024-06-26T06:33:02.3257502Z 4b25a744b402: Pulling fs layer 2024-06-26T06:33:02.3257968Z 21f74857bd17: Pulling fs layer 2024-06-26T06:33:02.3258407Z 4f4fb700ef54: Waiting 2024-06-26T06:33:02.3258818Z 6c12f547d202: Pulling fs layer 2024-06-26T06:33:02.3259465Z ed4e5a4f8526: Pulling fs layer 2024-06-26T06:33:02.3259943Z 6f09f741ead6: Pulling fs layer 2024-06-26T06:33:02.3260394Z b581c3782fd6: Waiting 2024-06-26T06:33:02.3260785Z 21f74857bd17: Waiting 2024-06-26T06:33:02.3261159Z 6c12f547d202: Waiting 2024-06-26T06:33:02.3261573Z 411fce2db725: Pulling fs layer 2024-06-26T06:33:02.3262021Z 4b25a744b402: Waiting 2024-06-26T06:33:02.3262686Z 87f7902d12a4: Pulling fs layer 2024-06-26T06:33:02.3263145Z 6f09f741ead6: Waiting 2024-06-26T06:33:02.3263518Z e5a6ec0a41d7: Waiting 2024-06-26T06:33:02.3263930Z 9777c07909ad: Pulling fs layer 2024-06-26T06:33:02.3264397Z 9755a952d3ec: Pulling fs layer 2024-06-26T06:33:02.3264846Z 5ca5d2e8bebe: Pulling fs layer 2024-06-26T06:33:02.3265298Z 9755a952d3ec: Waiting 2024-06-26T06:33:02.3265709Z d07af843db47: Pulling fs layer 2024-06-26T06:33:02.3266142Z ed4e5a4f8526: Waiting 2024-06-26T06:33:02.3266555Z d07af843db47: Waiting 2024-06-26T06:33:02.3266970Z dbbd7a4ca846: Pulling fs layer 2024-06-26T06:33:02.3267428Z 6a18c61a96dd: Pulling fs layer 2024-06-26T06:33:02.3267879Z 87f7902d12a4: Waiting 2024-06-26T06:33:02.3268289Z bd12222c0237: Pulling fs layer 2024-06-26T06:33:02.3268738Z a5dfe6ba36fd: Pulling fs layer 2024-06-26T06:33:02.3269214Z 4b990eb76e8f: Pulling fs layer 2024-06-26T06:33:02.3269670Z dbbd7a4ca846: Waiting 2024-06-26T06:33:02.3270069Z d9df85e8d422: Pulling fs layer 2024-06-26T06:33:02.3270537Z 47d89094b1a7: Pulling fs layer 2024-06-26T06:33:02.3271001Z 94beee2c356a: Pulling fs layer 2024-06-26T06:33:02.3271427Z bd12222c0237: Waiting 2024-06-26T06:33:02.3271833Z 322c644984f4: Pulling fs layer 2024-06-26T06:33:02.3272296Z b43ee04d974d: Pulling fs layer 2024-06-26T06:33:02.3272722Z 6a18c61a96dd: Waiting 2024-06-26T06:33:02.3273131Z 40f95cbeec2c: Pulling fs layer 2024-06-26T06:33:02.3273577Z 322c644984f4: Waiting 2024-06-26T06:33:02.3273969Z 4b2a3c41d6ae: Pulling fs layer 2024-06-26T06:33:02.3274429Z 94beee2c356a: Waiting 2024-06-26T06:33:02.3274825Z 4b990eb76e8f: Waiting 2024-06-26T06:33:02.3275222Z b2689866a0b9: Pulling fs layer 2024-06-26T06:33:02.3275695Z 9b36f9d4ace6: Pulling fs layer 2024-06-26T06:33:02.3276263Z a863a1c01885: Pulling fs layer 2024-06-26T06:33:02.3277144Z b9b8edd214be: Pulling fs layer 2024-06-26T06:33:02.3278070Z 9777c07909ad: Waiting 2024-06-26T06:33:02.3278813Z 4b2a3c41d6ae: Waiting 2024-06-26T06:33:02.3279538Z a863a1c01885: Waiting 2024-06-26T06:33:02.3280299Z 47d89094b1a7: Waiting 2024-06-26T06:33:02.3281168Z b9b8edd214be: Waiting 2024-06-26T06:33:02.3281913Z b2689866a0b9: Waiting 2024-06-26T06:33:02.3282661Z b43ee04d974d: Waiting 2024-06-26T06:33:02.3283412Z d9df85e8d422: Waiting 2024-06-26T06:33:02.4849575Z 09d415c238d7: Verifying Checksum 2024-06-26T06:33:02.4850092Z 09d415c238d7: Download complete 2024-06-26T06:33:02.5672307Z 41f16248e682: Verifying Checksum 2024-06-26T06:33:02.5672833Z 41f16248e682: Download complete 2024-06-26T06:33:02.6365569Z 95d7b7817039: Download complete 2024-06-26T06:33:02.6690321Z 7a2c55901189: Verifying Checksum 2024-06-26T06:33:02.6690867Z 7a2c55901189: Download complete 2024-06-26T06:33:02.7519552Z ab17245097e4: Verifying Checksum 2024-06-26T06:33:02.7520553Z ab17245097e4: Download complete 2024-06-26T06:33:02.8169201Z dfecd7e9912b: Verifying Checksum 2024-06-26T06:33:02.8169804Z dfecd7e9912b: Download complete 2024-06-26T06:33:02.8948605Z 464a8f745445: Verifying Checksum 2024-06-26T06:33:02.8949146Z 464a8f745445: Download complete 2024-06-26T06:33:02.9771117Z 9fe6e2e61518: Verifying Checksum 2024-06-26T06:33:02.9772098Z 9fe6e2e61518: Download complete 2024-06-26T06:33:03.0558094Z 0723abde19c0: Download complete 2024-06-26T06:33:03.1299970Z bb93d41d4cb4: Verifying Checksum 2024-06-26T06:33:03.1300606Z bb93d41d4cb4: Download complete 2024-06-26T06:33:04.0147276Z 7a2c55901189: Pull complete 2024-06-26T06:33:04.3832059Z 09d415c238d7: Pull complete 2024-06-26T06:33:05.5492837Z 9fe6e2e61518: Pull complete 2024-06-26T06:33:05.6127241Z 41f16248e682: Pull complete 2024-06-26T06:33:05.6694123Z 95d7b7817039: Pull complete 2024-06-26T06:33:10.0936676Z 4b89839dd066: Verifying Checksum 2024-06-26T06:33:10.0937720Z 4b89839dd066: Download complete 2024-06-26T06:33:10.1819406Z caa9986c765b: Verifying Checksum 2024-06-26T06:33:10.1820488Z caa9986c765b: Download complete 2024-06-26T06:33:10.2640149Z e17c09bdf243: Verifying Checksum 2024-06-26T06:33:10.2641172Z e17c09bdf243: Download complete 2024-06-26T06:33:10.3403585Z bd2e32de75c9: Download complete 2024-06-26T06:33:11.3345425Z 559a1daa0f98: Verifying Checksum 2024-06-26T06:33:11.3346011Z 559a1daa0f98: Download complete 2024-06-26T06:33:11.4055458Z d96cec7a425c: Verifying Checksum 2024-06-26T06:33:11.4056003Z d96cec7a425c: Download complete 2024-06-26T06:33:11.4774964Z 1db6fc6c2590: Verifying Checksum 2024-06-26T06:33:11.4775481Z 1db6fc6c2590: Download complete 2024-06-26T06:33:11.5670640Z ac522bc70345: Verifying Checksum 2024-06-26T06:33:11.5671214Z ac522bc70345: Download complete 2024-06-26T06:33:16.5321896Z 8f6c90485347: Download complete 2024-06-26T06:33:16.6169651Z 44f27f91d8fa: Download complete 2024-06-26T06:33:16.7029099Z 0a0ded106167: Download complete 2024-06-26T06:33:16.7864022Z 0c62bfb80b7c: Verifying Checksum 2024-06-26T06:33:16.7864603Z 0c62bfb80b7c: Download complete 2024-06-26T06:33:16.8604324Z 0c8d1e7f3282: Verifying Checksum 2024-06-26T06:33:16.8604892Z 0c8d1e7f3282: Download complete 2024-06-26T06:33:16.9469581Z 846366e78885: Verifying Checksum 2024-06-26T06:33:16.9470146Z 846366e78885: Download complete 2024-06-26T06:33:18.2771660Z efa11c713d18: Verifying Checksum 2024-06-26T06:33:18.2772287Z efa11c713d18: Download complete 2024-06-26T06:33:18.3560553Z 32f0221befeb: Verifying Checksum 2024-06-26T06:33:18.3561133Z 32f0221befeb: Download complete 2024-06-26T06:33:18.4619625Z a1d57c94ba33: Verifying Checksum 2024-06-26T06:33:18.4620262Z a1d57c94ba33: Download complete 2024-06-26T06:33:18.5312493Z 3dc15441e6d0: Download complete 2024-06-26T06:33:18.6152989Z 668eadf990e8: Download complete 2024-06-26T06:33:18.6947624Z fd5a832fe26e: Verifying Checksum 2024-06-26T06:33:18.6948721Z fd5a832fe26e: Download complete 2024-06-26T06:33:23.2871153Z b581c3782fd6: Verifying Checksum 2024-06-26T06:33:23.2872079Z b581c3782fd6: Download complete 2024-06-26T06:33:23.3549022Z 3f42611c5636: Verifying Checksum 2024-06-26T06:33:23.3549609Z 3f42611c5636: Download complete 2024-06-26T06:33:23.4281410Z fe8dd930eda1: Download complete 2024-06-26T06:33:23.8777749Z 6a9711a8a50d: Verifying Checksum 2024-06-26T06:33:23.8778347Z 6a9711a8a50d: Download complete 2024-06-26T06:33:23.9415897Z 76cd931663c9: Verifying Checksum 2024-06-26T06:33:23.9416448Z 76cd931663c9: Download complete 2024-06-26T06:33:24.0401754Z c9ec95615433: Verifying Checksum 2024-06-26T06:33:24.0402357Z c9ec95615433: Download complete 2024-06-26T06:33:24.3241929Z 7da33ab03193: Verifying Checksum 2024-06-26T06:33:24.3242646Z 7da33ab03193: Download complete 2024-06-26T06:33:24.3937863Z da956bbe6180: Verifying Checksum 2024-06-26T06:33:24.3938433Z da956bbe6180: Download complete 2024-06-26T06:33:24.4813077Z 73e6b5984544: Verifying Checksum 2024-06-26T06:33:24.5596472Z e5a6ec0a41d7: Verifying Checksum 2024-06-26T06:33:24.5597246Z e5a6ec0a41d7: Download complete 2024-06-26T06:33:24.5692400Z 4f4fb700ef54: Download complete 2024-06-26T06:33:24.6485949Z 4b25a744b402: Verifying Checksum 2024-06-26T06:33:24.6486724Z 4b25a744b402: Download complete 2024-06-26T06:33:24.7300761Z 21f74857bd17: Download complete 2024-06-26T06:33:25.2227780Z 6c12f547d202: Verifying Checksum 2024-06-26T06:33:25.2228364Z 6c12f547d202: Download complete 2024-06-26T06:33:25.3868378Z 6f09f741ead6: Verifying Checksum 2024-06-26T06:33:25.3869480Z 6f09f741ead6: Download complete 2024-06-26T06:33:25.4739319Z 411fce2db725: Download complete 2024-06-26T06:33:25.5428718Z 87f7902d12a4: Download complete 2024-06-26T06:33:27.4164705Z d67c111fa588: Verifying Checksum 2024-06-26T06:33:27.4165452Z d67c111fa588: Download complete 2024-06-26T06:33:27.5160400Z 9755a952d3ec: Verifying Checksum 2024-06-26T06:33:27.5161005Z 9755a952d3ec: Download complete 2024-06-26T06:33:27.6011356Z 5ca5d2e8bebe: Verifying Checksum 2024-06-26T06:33:27.6012243Z 5ca5d2e8bebe: Download complete 2024-06-26T06:33:27.6797300Z d07af843db47: Verifying Checksum 2024-06-26T06:33:27.6797880Z d07af843db47: Download complete 2024-06-26T06:33:27.7830696Z dbbd7a4ca846: Verifying Checksum 2024-06-26T06:33:27.7831392Z dbbd7a4ca846: Download complete 2024-06-26T06:33:27.8738278Z 6a18c61a96dd: Verifying Checksum 2024-06-26T06:33:27.8739263Z 6a18c61a96dd: Download complete 2024-06-26T06:33:28.0533266Z bd12222c0237: Verifying Checksum 2024-06-26T06:33:28.0533861Z bd12222c0237: Download complete 2024-06-26T06:33:28.1377696Z a5dfe6ba36fd: Verifying Checksum 2024-06-26T06:33:28.1378292Z a5dfe6ba36fd: Download complete 2024-06-26T06:33:28.2491471Z 4b990eb76e8f: Verifying Checksum 2024-06-26T06:33:28.2492041Z 4b990eb76e8f: Download complete 2024-06-26T06:33:28.3318082Z d9df85e8d422: Download complete 2024-06-26T06:33:28.4334063Z 47d89094b1a7: Download complete 2024-06-26T06:33:28.5981413Z 94beee2c356a: Verifying Checksum 2024-06-26T06:33:28.5982069Z 94beee2c356a: Download complete 2024-06-26T06:33:28.6729811Z 322c644984f4: Verifying Checksum 2024-06-26T06:33:28.6730875Z 322c644984f4: Download complete 2024-06-26T06:33:29.2868860Z b43ee04d974d: Verifying Checksum 2024-06-26T06:33:29.2869450Z b43ee04d974d: Download complete 2024-06-26T06:33:29.3931006Z 40f95cbeec2c: Verifying Checksum 2024-06-26T06:33:29.3932112Z 40f95cbeec2c: Download complete 2024-06-26T06:33:35.2676395Z 8f6c90485347: Pull complete 2024-06-26T06:33:35.3071004Z ab17245097e4: Pull complete 2024-06-26T06:33:35.3707500Z dfecd7e9912b: Pull complete 2024-06-26T06:33:35.4225236Z 464a8f745445: Pull complete 2024-06-26T06:33:39.7638244Z a05e0234ee85: Verifying Checksum 2024-06-26T06:33:39.7638955Z a05e0234ee85: Download complete 2024-06-26T06:33:39.8554406Z b2689866a0b9: Verifying Checksum 2024-06-26T06:33:39.8555255Z b2689866a0b9: Download complete 2024-06-26T06:33:39.9370921Z 9b36f9d4ace6: Verifying Checksum 2024-06-26T06:33:39.9371734Z 9b36f9d4ace6: Download complete 2024-06-26T06:33:40.4503901Z a863a1c01885: Verifying Checksum 2024-06-26T06:33:40.4504565Z a863a1c01885: Download complete 2024-06-26T06:33:40.5228568Z b9b8edd214be: Verifying Checksum 2024-06-26T06:33:40.5229166Z b9b8edd214be: Download complete 2024-06-26T06:33:43.1838072Z 4b2a3c41d6ae: Verifying Checksum 2024-06-26T06:33:43.1838666Z 4b2a3c41d6ae: Download complete 2024-06-26T06:33:44.6695858Z 9777c07909ad: Verifying Checksum 2024-06-26T06:33:44.6696473Z 9777c07909ad: Download complete 2024-06-26T06:34:14.3824395Z d67c111fa588: Pull complete 2024-06-26T06:34:16.8216183Z 0723abde19c0: Pull complete 2024-06-26T06:34:19.5424927Z bb93d41d4cb4: Pull complete 2024-06-26T06:34:35.9486758Z 4b89839dd066: Pull complete 2024-06-26T06:34:38.3584593Z caa9986c765b: Pull complete 2024-06-26T06:34:40.7997558Z e17c09bdf243: Pull complete 2024-06-26T06:34:43.2427003Z bd2e32de75c9: Pull complete 2024-06-26T06:34:49.0750970Z 559a1daa0f98: Pull complete 2024-06-26T06:34:51.5188045Z d96cec7a425c: Pull complete 2024-06-26T06:34:53.9437797Z 1db6fc6c2590: Pull complete 2024-06-26T06:34:56.6310373Z ac522bc70345: Pull complete 2024-06-26T06:36:04.1011973Z a05e0234ee85: Pull complete 2024-06-26T06:36:07.0031209Z 44f27f91d8fa: Pull complete 2024-06-26T06:36:11.8975911Z 0a0ded106167: Pull complete 2024-06-26T06:36:13.0781788Z 0c62bfb80b7c: Pull complete 2024-06-26T06:36:13.1310147Z 0c8d1e7f3282: Pull complete 2024-06-26T06:36:13.1742017Z 846366e78885: Pull complete 2024-06-26T06:36:17.4068396Z efa11c713d18: Pull complete 2024-06-26T06:36:17.4489781Z 32f0221befeb: Pull complete 2024-06-26T06:36:17.5173389Z a1d57c94ba33: Pull complete 2024-06-26T06:36:17.5758553Z 3dc15441e6d0: Pull complete 2024-06-26T06:36:17.6135534Z 668eadf990e8: Pull complete 2024-06-26T06:36:17.6620066Z fd5a832fe26e: Pull complete 2024-06-26T06:36:29.7246605Z b581c3782fd6: Pull complete 2024-06-26T06:36:32.7370929Z 3f42611c5636: Pull complete 2024-06-26T06:36:35.7645080Z fe8dd930eda1: Pull complete 2024-06-26T06:36:36.8983593Z 6a9711a8a50d: Pull complete 2024-06-26T06:36:36.9544060Z 76cd931663c9: Pull complete 2024-06-26T06:36:36.9988090Z c9ec95615433: Pull complete 2024-06-26T06:36:37.3890843Z 7da33ab03193: Pull complete 2024-06-26T06:36:37.4409113Z da956bbe6180: Pull complete 2024-06-26T06:36:37.5602700Z 73e6b5984544: Pull complete 2024-06-26T06:36:37.6124617Z e5a6ec0a41d7: Pull complete 2024-06-26T06:36:37.6506563Z 4f4fb700ef54: Pull complete 2024-06-26T06:36:37.6908215Z 4b25a744b402: Pull complete 2024-06-26T06:36:37.7444098Z 21f74857bd17: Pull complete 2024-06-26T06:36:40.3106454Z 6c12f547d202: Pull complete 2024-06-26T06:36:40.3651198Z ed4e5a4f8526: Pull complete 2024-06-26T06:36:40.4123692Z 6f09f741ead6: Pull complete 2024-06-26T06:36:40.5246020Z 411fce2db725: Pull complete 2024-06-26T06:36:40.5649630Z 87f7902d12a4: Pull complete 2024-06-26T06:37:23.6584148Z 9777c07909ad: Pull complete 2024-06-26T06:37:23.6960812Z 9755a952d3ec: Pull complete 2024-06-26T06:37:23.7280680Z 5ca5d2e8bebe: Pull complete 2024-06-26T06:37:26.4914366Z d07af843db47: Pull complete 2024-06-26T06:37:31.3675554Z dbbd7a4ca846: Pull complete 2024-06-26T06:37:33.8079582Z 6a18c61a96dd: Pull complete 2024-06-26T06:37:36.4922316Z bd12222c0237: Pull complete 2024-06-26T06:37:38.6840964Z a5dfe6ba36fd: Pull complete 2024-06-26T06:37:41.1887221Z 4b990eb76e8f: Pull complete 2024-06-26T06:37:43.6303632Z d9df85e8d422: Pull complete 2024-06-26T06:37:46.4982906Z 47d89094b1a7: Pull complete 2024-06-26T06:37:50.9612157Z 94beee2c356a: Pull complete 2024-06-26T06:37:53.5759163Z 322c644984f4: Pull complete 2024-06-26T06:37:58.0601908Z b43ee04d974d: Pull complete 2024-06-26T06:37:58.1117655Z 40f95cbeec2c: Pull complete 2024-06-26T06:38:15.0366428Z 4b2a3c41d6ae: Pull complete 2024-06-26T06:38:15.0973329Z b2689866a0b9: Pull complete 2024-06-26T06:38:15.1485184Z 9b36f9d4ace6: Pull complete 2024-06-26T06:38:15.7989627Z a863a1c01885: Pull complete 2024-06-26T06:38:15.8559300Z b9b8edd214be: Pull complete 2024-06-26T06:38:16.1312418Z Digest: sha256:493391c3b293fbc39c8ca4208a5b70ab17031329cb9c855a641ab5a208c52fe6 2024-06-26T06:38:16.1342042Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:38:16.1372460Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:38:16.1428040Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T06:38:16.1429218Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T06:38:16.1438308Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:38:16.1438928Z env: 2024-06-26T06:38:16.1439244Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:16.1439674Z ##[endgroup] 2024-06-26T06:38:16.1643191Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T06:38:16.1643869Z with: 2024-06-26T06:38:16.1644429Z driver-version: 550.54.15 2024-06-26T06:38:16.1644848Z env: 2024-06-26T06:38:16.1645156Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:16.1645574Z ##[endgroup] 2024-06-26T06:38:16.1699935Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:38:16.1700642Z with: 2024-06-26T06:38:16.1700957Z timeout_minutes: 10 2024-06-26T06:38:16.1701359Z max_attempts: 3 2024-06-26T06:38:16.1743145Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T06:38:16.1785250Z retry_wait_seconds: 10 2024-06-26T06:38:16.1785700Z polling_interval_seconds: 1 2024-06-26T06:38:16.1786144Z warning_on_retry: true 2024-06-26T06:38:16.1786573Z continue_on_error: false 2024-06-26T06:38:16.1786987Z env: 2024-06-26T06:38:16.1787298Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:16.1787743Z DRIVER_VERSION: 550.54.15 2024-06-26T06:38:16.1788172Z ##[endgroup] 2024-06-26T06:38:16.2554647Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T06:38:16.2556709Z + pre_install_nvidia_driver_amzn2 2024-06-26T06:38:16.2557371Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T06:38:16.7730720Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:38:16.8319506Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T06:38:16.8709069Z No Packages marked for removal 2024-06-26T06:38:16.8872655Z + install_nvidia_driver_common 2024-06-26T06:38:16.8876750Z + echo 'Before installing NVIDIA driver' 2024-06-26T06:38:16.8877773Z + lspci 2024-06-26T06:38:16.8878484Z Before installing NVIDIA driver 2024-06-26T06:38:16.9063725Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-06-26T06:38:16.9064778Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T06:38:16.9065783Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-06-26T06:38:16.9066741Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-06-26T06:38:16.9067574Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-06-26T06:38:16.9068420Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T06:38:16.9069468Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:38:16.9070540Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:38:16.9071581Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-06-26T06:38:16.9072303Z + lsmod 2024-06-26T06:38:16.9087292Z Module Size Used by 2024-06-26T06:38:16.9088169Z ib_core 266240 0 2024-06-26T06:38:16.9088649Z nvidia_modeset 1339392 0 2024-06-26T06:38:16.9089262Z veth 16384 0 2024-06-26T06:38:16.9090289Z nvidia_uvm 4599808 0 2024-06-26T06:38:16.9091403Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T06:38:16.9092889Z drm 421888 1 nvidia 2024-06-26T06:38:16.9093902Z i2c_core 77824 2 nvidia,drm 2024-06-26T06:38:16.9094698Z backlight 16384 1 nvidia_modeset 2024-06-26T06:38:16.9095620Z xt_conntrack 16384 1 2024-06-26T06:38:16.9096097Z ipt_MASQUERADE 16384 1 2024-06-26T06:38:16.9096643Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T06:38:16.9097205Z nf_conntrack_netlink 49152 0 2024-06-26T06:38:16.9097762Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T06:38:16.9098341Z xfrm_user 45056 1 2024-06-26T06:38:16.9098803Z xfrm_algo 16384 1 xfrm_user 2024-06-26T06:38:16.9099323Z iptable_nat 16384 1 2024-06-26T06:38:16.9099792Z nf_conntrack_ipv4 16384 3 2024-06-26T06:38:16.9100308Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T06:38:16.9100913Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T06:38:16.9101559Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T06:38:16.9103373Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T06:38:16.9104464Z xt_addrtype 16384 2 2024-06-26T06:38:16.9104932Z iptable_filter 16384 1 2024-06-26T06:38:16.9105379Z br_netfilter 24576 0 2024-06-26T06:38:16.9105886Z bridge 172032 1 br_netfilter 2024-06-26T06:38:16.9106434Z stp 16384 1 bridge 2024-06-26T06:38:16.9106935Z llc 16384 2 bridge,stp 2024-06-26T06:38:16.9107448Z overlay 86016 0 2024-06-26T06:38:16.9107903Z sunrpc 393216 1 2024-06-26T06:38:16.9108684Z dm_mirror 28672 0 2024-06-26T06:38:16.9109564Z dm_region_hash 20480 1 dm_mirror 2024-06-26T06:38:16.9110864Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T06:38:16.9111544Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T06:38:16.9112134Z dax 69632 1 dm_mod 2024-06-26T06:38:16.9112660Z sb_edac 24576 0 2024-06-26T06:38:16.9113537Z crc32_pclmul 16384 0 2024-06-26T06:38:16.9114205Z ghash_clmulni_intel 16384 0 2024-06-26T06:38:16.9115104Z pcbc 16384 0 2024-06-26T06:38:16.9116023Z aesni_intel 188416 0 2024-06-26T06:38:16.9116833Z aes_x86_64 20480 1 aesni_intel 2024-06-26T06:38:16.9117577Z crypto_simd 16384 1 aesni_intel 2024-06-26T06:38:16.9118694Z glue_helper 16384 1 aesni_intel 2024-06-26T06:38:16.9119219Z mousedev 24576 0 2024-06-26T06:38:16.9119950Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T06:38:16.9121318Z evdev 20480 3 2024-06-26T06:38:16.9122225Z psmouse 32768 0 2024-06-26T06:38:16.9123378Z pcc_cpufreq 16384 0 2024-06-26T06:38:16.9124191Z button 16384 0 2024-06-26T06:38:16.9124695Z ena 139264 0 2024-06-26T06:38:16.9125162Z ptp 20480 1 ena 2024-06-26T06:38:16.9125651Z pps_core 20480 1 ptp 2024-06-26T06:38:16.9126119Z ata_piix 36864 0 2024-06-26T06:38:16.9126593Z libata 266240 1 ata_piix 2024-06-26T06:38:16.9127113Z xen_blkfront 49152 2 2024-06-26T06:38:16.9127556Z crc32c_intel 24576 0 2024-06-26T06:38:16.9128024Z scsi_mod 245760 1 libata 2024-06-26T06:38:16.9128517Z autofs4 49152 2 2024-06-26T06:38:16.9128944Z + modinfo nvidia 2024-06-26T06:38:16.9129782Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T06:38:16.9130674Z alias: char-major-195-* 2024-06-26T06:38:16.9131126Z version: 550.54.15 2024-06-26T06:38:16.9131551Z supported: external 2024-06-26T06:38:16.9132157Z license: NVIDIA 2024-06-26T06:38:16.9132636Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T06:38:16.9133235Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T06:38:16.9133914Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T06:38:16.9134474Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T06:38:16.9135087Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T06:38:16.9135690Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T06:38:16.9136290Z depends: i2c-core,drm 2024-06-26T06:38:16.9136733Z retpoline: Y 2024-06-26T06:38:16.9137109Z name: nvidia 2024-06-26T06:38:16.9137786Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T06:38:16.9138609Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T06:38:16.9139414Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T06:38:16.9140138Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T06:38:16.9140692Z parm: NVreg_RmLogonRC:int 2024-06-26T06:38:16.9141231Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T06:38:16.9141787Z parm: NVreg_DeviceFileUID:int 2024-06-26T06:38:16.9142990Z parm: NVreg_DeviceFileGID:int 2024-06-26T06:38:16.9143564Z parm: NVreg_DeviceFileMode:int 2024-06-26T06:38:16.9144197Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T06:38:16.9144902Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T06:38:16.9145502Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T06:38:16.9146017Z parm: NVreg_EnableMSI:int 2024-06-26T06:38:16.9146530Z parm: NVreg_TCEBypassMode:int 2024-06-26T06:38:16.9147095Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T06:38:16.9147738Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T06:38:16.9148459Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T06:38:16.9149167Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T06:38:16.9149915Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T06:38:16.9150669Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T06:38:16.9151438Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T06:38:16.9152164Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T06:38:16.9152779Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T06:38:16.9153448Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T06:38:16.9154133Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T06:38:16.9154735Z parm: NVreg_MemoryPoolSize:int 2024-06-26T06:38:16.9155318Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T06:38:16.9155917Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T06:38:16.9156488Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T06:38:16.9157046Z parm: NVreg_NvLinkDisable:int 2024-06-26T06:38:16.9157676Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T06:38:16.9158314Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T06:38:16.9158908Z parm: NVreg_EnableResizableBar:int 2024-06-26T06:38:16.9159519Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T06:38:16.9160120Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T06:38:16.9160724Z parm: NVreg_RegistryDwords:charp 2024-06-26T06:38:16.9161340Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T06:38:16.9161921Z parm: NVreg_RmMsg:charp 2024-06-26T06:38:16.9162429Z parm: NVreg_GpuBlacklist:charp 2024-06-26T06:38:16.9163007Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T06:38:16.9163575Z parm: NVreg_ExcludedGpus:charp 2024-06-26T06:38:16.9164140Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T06:38:16.9164733Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T06:38:16.9165302Z parm: NVreg_ImexChannelCount:int 2024-06-26T06:38:16.9165865Z parm: rm_firmware_active:charp 2024-06-26T06:38:16.9166519Z + HAS_NVIDIA_DRIVER=0 2024-06-26T06:38:16.9167007Z ++ command -v nvidia-smi 2024-06-26T06:38:16.9167501Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T06:38:16.9168074Z + set +e 2024-06-26T06:38:16.9168689Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T06:38:19.3238108Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T06:38:19.3239063Z + NVIDIA_SMI_STATUS=0 2024-06-26T06:38:19.3240100Z + '[' 0 -ne 0 ']' 2024-06-26T06:38:19.3240538Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T06:38:19.3241009Z + HAS_NVIDIA_DRIVER=1 2024-06-26T06:38:19.3241922Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T06:38:19.3242804Z + set -e 2024-06-26T06:38:19.3243174Z + '[' 1 -eq 0 ']' 2024-06-26T06:38:19.3243887Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T06:38:19.3244762Z + post_install_nvidia_driver_common 2024-06-26T06:38:19.3245273Z + sudo modprobe nvidia 2024-06-26T06:38:19.3389179Z + echo 'After installing NVIDIA driver' 2024-06-26T06:38:19.3389965Z + lspci 2024-06-26T06:38:19.3390324Z After installing NVIDIA driver 2024-06-26T06:38:19.3566256Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-06-26T06:38:19.3568017Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T06:38:19.3568984Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-06-26T06:38:19.3569925Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-06-26T06:38:19.3570733Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-06-26T06:38:19.3571574Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T06:38:19.3572611Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:38:19.3573669Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:38:19.3574719Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-06-26T06:38:19.3575435Z + lsmod 2024-06-26T06:38:19.3589849Z Module Size Used by 2024-06-26T06:38:19.3590777Z ib_core 266240 0 2024-06-26T06:38:19.3591661Z nvidia_modeset 1339392 0 2024-06-26T06:38:19.3592590Z veth 16384 0 2024-06-26T06:38:19.3593523Z nvidia_uvm 4599808 0 2024-06-26T06:38:19.3594328Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T06:38:19.3595550Z drm 421888 1 nvidia 2024-06-26T06:38:19.3596445Z i2c_core 77824 2 nvidia,drm 2024-06-26T06:38:19.3597007Z backlight 16384 1 nvidia_modeset 2024-06-26T06:38:19.3597566Z xt_conntrack 16384 1 2024-06-26T06:38:19.3598035Z ipt_MASQUERADE 16384 1 2024-06-26T06:38:19.3598558Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T06:38:19.3599138Z nf_conntrack_netlink 49152 0 2024-06-26T06:38:19.3599735Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T06:38:19.3600310Z xfrm_user 45056 1 2024-06-26T06:38:19.3600782Z xfrm_algo 16384 1 xfrm_user 2024-06-26T06:38:19.3601303Z iptable_nat 16384 1 2024-06-26T06:38:19.3602027Z nf_conntrack_ipv4 16384 3 2024-06-26T06:38:19.3603081Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T06:38:19.3604348Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T06:38:19.3605621Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T06:38:19.3607557Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T06:38:19.3609485Z xt_addrtype 16384 2 2024-06-26T06:38:19.3610474Z iptable_filter 16384 1 2024-06-26T06:38:19.3610940Z br_netfilter 24576 0 2024-06-26T06:38:19.3611447Z bridge 172032 1 br_netfilter 2024-06-26T06:38:19.3612258Z stp 16384 1 bridge 2024-06-26T06:38:19.3612797Z llc 16384 2 bridge,stp 2024-06-26T06:38:19.3613316Z overlay 86016 0 2024-06-26T06:38:19.3613942Z sunrpc 393216 1 2024-06-26T06:38:19.3614380Z dm_mirror 28672 0 2024-06-26T06:38:19.3614866Z dm_region_hash 20480 1 dm_mirror 2024-06-26T06:38:19.3615466Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T06:38:19.3616160Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T06:38:19.3616789Z dax 69632 1 dm_mod 2024-06-26T06:38:19.3617279Z sb_edac 24576 0 2024-06-26T06:38:19.3617715Z crc32_pclmul 16384 0 2024-06-26T06:38:19.3618295Z ghash_clmulni_intel 16384 0 2024-06-26T06:38:19.3618765Z pcbc 16384 0 2024-06-26T06:38:19.3619209Z aesni_intel 188416 0 2024-06-26T06:38:19.3619697Z aes_x86_64 20480 1 aesni_intel 2024-06-26T06:38:19.3620264Z crypto_simd 16384 1 aesni_intel 2024-06-26T06:38:19.3620803Z glue_helper 16384 1 aesni_intel 2024-06-26T06:38:19.3621324Z mousedev 24576 0 2024-06-26T06:38:19.3621951Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T06:38:19.3622943Z evdev 20480 3 2024-06-26T06:38:19.3623407Z psmouse 32768 0 2024-06-26T06:38:19.3623875Z pcc_cpufreq 16384 0 2024-06-26T06:38:19.3624313Z button 16384 0 2024-06-26T06:38:19.3624764Z ena 139264 0 2024-06-26T06:38:19.3625220Z ptp 20480 1 ena 2024-06-26T06:38:19.3625692Z pps_core 20480 1 ptp 2024-06-26T06:38:19.3626174Z ata_piix 36864 0 2024-06-26T06:38:19.3626649Z libata 266240 1 ata_piix 2024-06-26T06:38:19.3627142Z xen_blkfront 49152 2 2024-06-26T06:38:19.3627602Z crc32c_intel 24576 0 2024-06-26T06:38:19.3628074Z scsi_mod 245760 1 libata 2024-06-26T06:38:19.3628556Z autofs4 49152 2 2024-06-26T06:38:19.3629009Z + modinfo nvidia 2024-06-26T06:38:19.3629884Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T06:38:19.3630780Z alias: char-major-195-* 2024-06-26T06:38:19.3631230Z version: 550.54.15 2024-06-26T06:38:19.3631660Z supported: external 2024-06-26T06:38:19.3632080Z license: NVIDIA 2024-06-26T06:38:19.3632527Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T06:38:19.3633133Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T06:38:19.3633696Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T06:38:19.3634253Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T06:38:19.3634873Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T06:38:19.3635480Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T06:38:19.3636068Z depends: i2c-core,drm 2024-06-26T06:38:19.3636514Z retpoline: Y 2024-06-26T06:38:19.3636898Z name: nvidia 2024-06-26T06:38:19.3637570Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T06:38:19.3638400Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T06:38:19.3639215Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T06:38:19.3639946Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T06:38:19.3640497Z parm: NVreg_RmLogonRC:int 2024-06-26T06:38:19.3641032Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T06:38:19.3641579Z parm: NVreg_DeviceFileUID:int 2024-06-26T06:38:19.3642120Z parm: NVreg_DeviceFileGID:int 2024-06-26T06:38:19.3642657Z parm: NVreg_DeviceFileMode:int 2024-06-26T06:38:19.3643292Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T06:38:19.3643994Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T06:38:19.3644586Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T06:38:19.3645263Z parm: NVreg_EnableMSI:int 2024-06-26T06:38:19.3645803Z parm: NVreg_TCEBypassMode:int 2024-06-26T06:38:19.3646368Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T06:38:19.3647135Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T06:38:19.3647861Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T06:38:19.3648562Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T06:38:19.3649325Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T06:38:19.3650054Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T06:38:19.3650814Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T06:38:19.3651550Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T06:38:19.3652140Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T06:38:19.3652801Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T06:38:19.3653483Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T06:38:19.3654092Z parm: NVreg_MemoryPoolSize:int 2024-06-26T06:38:19.3654668Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T06:38:19.3655271Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T06:38:19.3655837Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T06:38:19.3656388Z parm: NVreg_NvLinkDisable:int 2024-06-26T06:38:19.3657009Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T06:38:19.3657645Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T06:38:19.3658240Z parm: NVreg_EnableResizableBar:int 2024-06-26T06:38:19.3658845Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T06:38:19.3659447Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T06:38:19.3660050Z parm: NVreg_RegistryDwords:charp 2024-06-26T06:38:19.3660666Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T06:38:19.3661242Z parm: NVreg_RmMsg:charp 2024-06-26T06:38:19.3661752Z parm: NVreg_GpuBlacklist:charp 2024-06-26T06:38:19.3662604Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T06:38:19.3663182Z parm: NVreg_ExcludedGpus:charp 2024-06-26T06:38:19.3663754Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T06:38:19.3664347Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T06:38:19.3664918Z parm: NVreg_ImexChannelCount:int 2024-06-26T06:38:19.3665475Z parm: rm_firmware_active:charp 2024-06-26T06:38:19.3665974Z + set +e 2024-06-26T06:38:19.3666333Z + nvidia-smi 2024-06-26T06:38:20.9270227Z Wed Jun 26 06:38:20 2024 2024-06-26T06:38:20.9271131Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:38:20.9272186Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T06:38:20.9273188Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:38:20.9274215Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T06:38:20.9275344Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T06:38:20.9276236Z | | | MIG M. | 2024-06-26T06:38:20.9276895Z |=========================================+========================+======================| 2024-06-26T06:38:20.9317835Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-06-26T06:38:20.9318748Z | N/A 36C P0 40W / 150W | 0MiB / 7680MiB | 100% Default | 2024-06-26T06:38:20.9319542Z | | | N/A | 2024-06-26T06:38:20.9320375Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:38:20.9365661Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T06:38:20.9366896Z | N/A 40C P0 39W / 150W | 0MiB / 7680MiB | 6% Default | 2024-06-26T06:38:20.9367733Z | | | N/A | 2024-06-26T06:38:20.9368719Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:38:20.9369428Z 2024-06-26T06:38:20.9370248Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:38:20.9371023Z | Processes: | 2024-06-26T06:38:20.9371906Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T06:38:20.9372772Z | ID ID Usage | 2024-06-26T06:38:20.9373468Z |=========================================================================================| 2024-06-26T06:38:20.9374294Z | No running processes found | 2024-06-26T06:38:20.9375261Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:38:21.8948050Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T06:38:23.4452575Z Tesla M60 2024-06-26T06:38:24.0889102Z + NVIDIA_SMI_STATUS=0 2024-06-26T06:38:24.0889734Z + '[' 0 -eq 0 ']' 2024-06-26T06:38:24.0890216Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T06:38:24.0890750Z + set -e 2024-06-26T06:38:24.0891113Z INFO: Ignoring allowed status 0 2024-06-26T06:38:24.0896289Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T06:38:24.0900109Z + sudo yum install -y yum-utils 2024-06-26T06:38:24.6041365Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:38:26.2937006Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T06:38:26.2937914Z Nothing to do 2024-06-26T06:38:26.4391822Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T06:38:26.4392888Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:38:26.4394307Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:38:26.9669622Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:38:26.9988979Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:38:26.9990541Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T06:38:26.9991770Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T06:38:27.0132358Z + sudo yum install -y nvidia-docker2 2024-06-26T06:38:27.5272552Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:38:29.2211943Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T06:38:29.2212776Z Nothing to do 2024-06-26T06:38:29.3651635Z + sudo systemctl restart docker 2024-06-26T06:38:47.2867121Z Command completed after 1 attempt(s). 2024-06-26T06:38:47.2944659Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T06:38:47.2945660Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T06:38:47.2946525Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T06:38:47.2947329Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:38:47.2956028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:38:47.2956651Z env: 2024-06-26T06:38:47.2956987Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:47.2957521Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:38:47.2958125Z ##[endgroup] 2024-06-26T06:38:47.5928628Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T06:38:47.6140748Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T06:38:47.6268448Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T06:38:47.7446824Z Prepare all required actions 2024-06-26T06:38:47.7447494Z Getting action download info 2024-06-26T06:38:47.8904881Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T06:38:48.0801237Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T06:38:48.2108756Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T06:38:48.2109363Z with: 2024-06-26T06:38:48.2109728Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:38:48.2110268Z s3-bucket: gha-artifacts 2024-06-26T06:38:48.2110682Z env: 2024-06-26T06:38:48.2110996Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:48.2111553Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:38:48.2112152Z ##[endgroup] 2024-06-26T06:38:48.2156545Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T06:38:48.2157104Z with: 2024-06-26T06:38:48.2157549Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:38:48.2158071Z s3-bucket: gha-artifacts 2024-06-26T06:38:48.2158503Z region: us-east-1 2024-06-26T06:38:48.2158866Z env: 2024-06-26T06:38:48.2159180Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:38:48.2159731Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:38:48.2160331Z ##[endgroup] 2024-06-26T06:38:48.8337956Z (node:102405) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T06:38:48.8338875Z 2024-06-26T06:38:48.8339216Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T06:38:48.8340116Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T06:38:48.8341201Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T06:38:48.9448519Z Found 1 objects with prefix pytorch/pytorch/9673645538/linux-focal-cuda11.8-py3.10-gcc9/ 2024-06-26T06:38:48.9450500Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T06:39:09.4031047Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T06:39:09.4042054Z Artifact download has finished successfully 2024-06-26T06:39:09.4263227Z ##[group]Run unzip -o artifacts.zip 2024-06-26T06:39:09.4263778Z unzip -o artifacts.zip 2024-06-26T06:39:09.4272268Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:09.4272891Z env: 2024-06-26T06:39:09.4273225Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:09.4273761Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:09.4274359Z ##[endgroup] 2024-06-26T06:39:09.4312965Z Archive: artifacts.zip 2024-06-26T06:39:09.4314385Z creating: dist/ 2024-06-26T06:39:11.8785822Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T06:39:11.8786701Z creating: build/custom_test_artifacts/ 2024-06-26T06:39:11.8787467Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T06:39:11.8788386Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T06:39:11.8789449Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:39:11.8796573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:39:11.8797795Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T06:39:11.8799008Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:39:11.8800285Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:39:11.8801515Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:39:11.8802933Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:39:11.8804899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:39:11.8806205Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:39:11.8807486Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:39:11.8808987Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:39:11.8810494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:39:11.8811948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:39:11.8813423Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:39:11.8814938Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:39:11.8816468Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:39:11.8817786Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:39:11.8819082Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:39:11.8874127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:39:11.8875994Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:39:11.8877846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:39:11.8879772Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:39:11.8881698Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:39:11.8883499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:39:11.8885300Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:39:11.8887121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:39:11.8888922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:39:11.8930573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:39:11.8980730Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:39:11.8982968Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:39:11.8984676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:39:11.8986295Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:39:11.8987865Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:39:11.8989473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:39:11.8991031Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:39:11.8992591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:39:11.9065210Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:39:11.9140690Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:39:11.9142542Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:39:11.9143833Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:39:11.9144945Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:39:11.9146120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T06:39:11.9147315Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T06:39:11.9148625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T06:39:11.9150114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T06:39:11.9151568Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T06:39:11.9152924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T06:39:11.9154292Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T06:39:11.9155691Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T06:39:11.9157092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T06:39:11.9158462Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T06:39:11.9159828Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T06:39:11.9177679Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T06:39:11.9344309Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T06:39:11.9345600Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T06:39:11.9347018Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T06:39:11.9348591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T06:39:11.9350115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T06:39:11.9351512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T06:39:11.9352977Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T06:39:11.9354464Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T06:39:11.9355924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T06:39:11.9357391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T06:39:11.9358832Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T06:39:11.9377060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T06:39:11.9481060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T06:39:11.9482620Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:39:11.9484019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:39:11.9485631Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T06:39:11.9486815Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T06:39:11.9487966Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T06:39:11.9489121Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T06:39:11.9490190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T06:39:11.9491190Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T06:39:11.9492228Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T06:39:11.9631193Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T06:39:11.9709839Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T06:39:11.9710790Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T06:39:11.9711698Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T06:39:11.9712725Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:39:11.9719701Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:39:11.9720895Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T06:39:11.9722082Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:39:11.9723319Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:39:11.9724550Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:39:11.9725955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:39:11.9727368Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:39:11.9728642Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:39:11.9729909Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:39:11.9731379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:39:11.9732843Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:39:11.9734308Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:39:11.9735771Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:39:11.9737261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:39:11.9738730Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:39:11.9740070Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:39:11.9741352Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:39:11.9797180Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:39:11.9799003Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:39:11.9800867Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:39:11.9802785Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:39:11.9804666Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:39:11.9806761Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:39:11.9808585Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:39:11.9810402Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:39:11.9812203Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:39:11.9853424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:39:11.9903561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:39:11.9905395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:39:11.9907062Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:39:11.9908650Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:39:11.9910225Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:39:11.9911807Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:39:11.9913344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:39:11.9914899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:39:11.9987103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:39:12.0063319Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:39:12.0064881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:39:12.0066135Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:39:12.0067233Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:39:12.0068395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T06:39:12.0069610Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T06:39:12.0070959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T06:39:12.0072491Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T06:39:12.0073982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T06:39:12.0075349Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T06:39:12.0076781Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T06:39:12.0078214Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T06:39:12.0079646Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T06:39:12.0081052Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T06:39:12.0082461Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T06:39:12.0098576Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T06:39:12.0179297Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T06:39:12.0180848Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:39:12.0182514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:39:12.0183787Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T06:39:12.0184928Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T06:39:12.0186062Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T06:39:12.0187210Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T06:39:12.0188294Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T06:39:12.0189260Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T06:39:12.0190299Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T06:39:12.0252341Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T06:39:12.0253315Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T06:39:12.0254267Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T06:39:12.0255395Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:39:12.0262629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:39:12.0263933Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T06:39:12.0265225Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:39:12.0266579Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:39:12.0267934Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:39:12.0269414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:39:12.0270942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:39:12.0272315Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:39:12.0273655Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:39:12.0275234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:39:12.0276807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:39:12.0278360Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:39:12.0279912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:39:12.0281488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:39:12.0283084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:39:12.0284500Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:39:12.0285854Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:39:12.0341027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:39:12.0343198Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:39:12.0345494Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:39:12.0347514Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:39:12.0349504Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:39:12.0351390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:39:12.0353284Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:39:12.0355169Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:39:12.0357066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:39:12.0397965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:39:12.0447980Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:39:12.0449859Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:39:12.0451616Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:39:12.0453313Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:39:12.0454977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:39:12.0456658Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:39:12.0458289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:39:12.0459955Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:39:12.0531216Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:39:12.0607038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:39:12.0608667Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:39:12.0610030Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:39:12.0611226Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:39:12.0612462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T06:39:12.0613769Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T06:39:12.0615240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T06:39:12.0616855Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T06:39:12.0618426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T06:39:12.0619903Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T06:39:12.0621442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T06:39:12.0623615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T06:39:12.0625197Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T06:39:12.0626730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T06:39:12.0628254Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T06:39:12.0629857Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T06:39:12.0780143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T06:39:12.0781669Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T06:39:12.0783455Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T06:39:12.0785162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T06:39:12.0786820Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T06:39:12.0788379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T06:39:12.0789968Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T06:39:12.0791580Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T06:39:12.0793198Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T06:39:12.0794854Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T06:39:12.0796442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T06:39:12.0813752Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T06:39:12.0883199Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T06:39:12.0884897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:39:12.0886369Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:39:12.0887696Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T06:39:12.0888947Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T06:39:12.0890189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T06:39:12.0891418Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T06:39:12.0892593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T06:39:12.0893664Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T06:39:12.0894750Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T06:39:12.1022177Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T06:39:12.1076308Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T06:39:12.1077130Z creating: build/lib/ 2024-06-26T06:39:12.1088287Z inflating: build/lib/libpthreadpool.a 2024-06-26T06:39:12.1089154Z inflating: build/lib/libclog.a 2024-06-26T06:39:12.1099655Z inflating: build/lib/libcpuinfo.a 2024-06-26T06:39:12.1109756Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T06:39:12.1112574Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T06:39:12.1224034Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T06:39:12.1792525Z inflating: build/lib/libprotobuf.a 2024-06-26T06:39:12.1873611Z inflating: build/lib/libgtest.a 2024-06-26T06:39:12.1966701Z inflating: build/lib/libbenchmark.a 2024-06-26T06:39:12.1976121Z inflating: build/lib/libittnotify.a 2024-06-26T06:39:12.2010756Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T06:39:12.2089933Z inflating: build/lib/libasmjit.a 2024-06-26T06:39:12.2248317Z inflating: build/lib/libgloo.a 2024-06-26T06:39:12.2275425Z inflating: build/lib/libfmt.a 2024-06-26T06:39:12.2277764Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T06:39:12.2392256Z inflating: build/lib/libc10.so 2024-06-26T06:39:12.2393355Z inflating: build/lib/libfoxi_loader.a 2024-06-26T06:39:12.2394868Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T06:39:12.2419191Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T06:39:12.2441666Z inflating: build/lib/libgmock.a 2024-06-26T06:39:12.2442461Z inflating: build/lib/libgtest_main.a 2024-06-26T06:39:12.2443326Z inflating: build/lib/libbenchmark_main.a 2024-06-26T06:39:12.3070436Z inflating: build/lib/libprotoc.a 2024-06-26T06:39:12.3092850Z inflating: build/lib/libnnpack.a 2024-06-26T06:39:13.5720848Z inflating: build/lib/libdnnl.a 2024-06-26T06:39:13.6433665Z inflating: build/lib/libtensorpipe.a 2024-06-26T06:39:13.6501399Z inflating: build/lib/libc10_cuda.so 2024-06-26T06:39:13.6502631Z inflating: build/lib/libgmock_main.a 2024-06-26T06:39:13.8061073Z inflating: build/lib/libfbgemm.a 2024-06-26T06:39:13.8702335Z inflating: build/lib/libkineto.a 2024-06-26T06:39:13.9020644Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T06:39:13.9250214Z inflating: build/lib/libXNNPACK.a 2024-06-26T06:39:13.9733989Z inflating: build/lib/libgloo_cuda.a 2024-06-26T06:39:13.9786929Z inflating: build/lib/libonnx_proto.a 2024-06-26T06:39:14.0647825Z inflating: build/lib/libonnx.a 2024-06-26T06:39:17.0768409Z inflating: build/lib/libtorch_cpu.so 2024-06-26T06:39:17.0773238Z inflating: build/lib/libshm.so 2024-06-26T06:39:17.0779173Z inflating: build/lib/libunbox_lib.a 2024-06-26T06:39:19.5348439Z inflating: build/lib/libtorch_cuda.so 2024-06-26T06:39:19.5349472Z inflating: build/lib/libtorch.so 2024-06-26T06:39:19.5352838Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T06:39:20.6737675Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T06:39:20.6763467Z inflating: build/lib/libjitbackend_test.so 2024-06-26T06:39:20.6849118Z inflating: build/lib/libtorchbind_test.so 2024-06-26T06:39:20.6880973Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T06:39:20.9334495Z inflating: build/lib/libtorch_python.so 2024-06-26T06:39:20.9377533Z inflating: build/lib/libnnapi_backend.so 2024-06-26T06:39:20.9378114Z creating: build/bin/ 2024-06-26T06:39:20.9440742Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T06:39:20.9505261Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T06:39:20.9568066Z inflating: build/bin/c10_Device_test 2024-06-26T06:39:20.9641928Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T06:39:20.9709687Z inflating: build/bin/c10_Scalar_test 2024-06-26T06:39:20.9770890Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T06:39:20.9834728Z inflating: build/bin/c10_SymInt_test 2024-06-26T06:39:20.9902654Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T06:39:20.9972840Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T06:39:21.0043234Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T06:39:21.0131773Z inflating: build/bin/c10_cow_test 2024-06-26T06:39:21.0198237Z inflating: build/bin/c10_Bitset_test 2024-06-26T06:39:21.0259342Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T06:39:21.0321611Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T06:39:21.0385141Z inflating: build/bin/c10_Half_test 2024-06-26T06:39:21.0455473Z inflating: build/bin/c10_LeftRight_test 2024-06-26T06:39:21.0524313Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T06:39:21.0586871Z inflating: build/bin/c10_Synchronized_test 2024-06-26T06:39:21.0656396Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T06:39:21.0720815Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T06:39:21.0784810Z inflating: build/bin/c10_TypeList_test 2024-06-26T06:39:21.0845856Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T06:39:21.0910577Z inflating: build/bin/c10_accumulate_test 2024-06-26T06:39:21.0979640Z inflating: build/bin/c10_bfloat16_test 2024-06-26T06:39:21.1042990Z inflating: build/bin/c10_bit_cast_test 2024-06-26T06:39:21.1113441Z inflating: build/bin/c10_complex_math_test 2024-06-26T06:39:21.1182878Z inflating: build/bin/c10_complex_test 2024-06-26T06:39:21.1248721Z inflating: build/bin/c10_exception_test 2024-06-26T06:39:21.1311864Z inflating: build/bin/c10_flags_test 2024-06-26T06:39:21.1374483Z inflating: build/bin/c10_generic_math_test 2024-06-26T06:39:21.1581563Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T06:39:21.1645887Z inflating: build/bin/c10_irange_test 2024-06-26T06:39:21.1713099Z inflating: build/bin/c10_lazy_test 2024-06-26T06:39:21.1784348Z inflating: build/bin/c10_logging_test 2024-06-26T06:39:21.1877809Z inflating: build/bin/c10_optional_test 2024-06-26T06:39:21.1955788Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T06:39:21.2022837Z inflating: build/bin/c10_registry_test 2024-06-26T06:39:21.2212026Z inflating: build/bin/c10_small_vector_test 2024-06-26T06:39:21.2276602Z inflating: build/bin/c10_ssize_test 2024-06-26T06:39:21.2341409Z inflating: build/bin/c10_string_util_test 2024-06-26T06:39:21.2415391Z inflating: build/bin/c10_string_view_test 2024-06-26T06:39:21.2478462Z inflating: build/bin/c10_tempfile_test 2024-06-26T06:39:21.2539170Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T06:39:21.2609401Z inflating: build/bin/c10_typeid_test 2024-06-26T06:39:21.3166922Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T06:39:21.3723180Z inflating: build/bin/protoc 2024-06-26T06:39:21.3789601Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T06:39:21.3855528Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T06:39:21.3921586Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T06:39:21.3986739Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T06:39:21.4052656Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T06:39:21.4118774Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T06:39:21.4184759Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T06:39:21.4246724Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T06:39:21.4661833Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T06:39:21.5092474Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T06:39:21.5541257Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T06:39:21.5608572Z inflating: build/bin/HashStoreTest 2024-06-26T06:39:21.5674853Z inflating: build/bin/FileStoreTest 2024-06-26T06:39:21.5744464Z inflating: build/bin/TCPStoreTest 2024-06-26T06:39:21.5761333Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T06:39:21.5766593Z inflating: build/bin/torch_shm_manager 2024-06-26T06:39:21.5833978Z inflating: build/bin/test_edge_op_registration 2024-06-26T06:39:21.5837482Z inflating: build/bin/example_allreduce 2024-06-26T06:39:21.5902498Z inflating: build/bin/Dimname_test 2024-06-26T06:39:21.5995455Z inflating: build/bin/Dict_test 2024-06-26T06:39:21.6076285Z inflating: build/bin/MaybeOwned_test 2024-06-26T06:39:21.6150210Z inflating: build/bin/apply_utils_test 2024-06-26T06:39:21.6222264Z inflating: build/bin/NamedTensor_test 2024-06-26T06:39:21.6296601Z inflating: build/bin/atest 2024-06-26T06:39:21.6375498Z inflating: build/bin/basic 2024-06-26T06:39:21.6444235Z inflating: build/bin/broadcast_test 2024-06-26T06:39:21.6507703Z inflating: build/bin/cpu_allocator_test 2024-06-26T06:39:21.6580347Z inflating: build/bin/cpu_generator_test 2024-06-26T06:39:21.6646998Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T06:39:21.6763825Z inflating: build/bin/cpu_rng_test 2024-06-26T06:39:21.6826477Z inflating: build/bin/dispatch_key_set_test 2024-06-26T06:39:21.6889341Z inflating: build/bin/dlconvertor_test 2024-06-26T06:39:21.6962293Z inflating: build/bin/extension_backend_test 2024-06-26T06:39:21.7031429Z inflating: build/bin/half_test 2024-06-26T06:39:21.7151278Z inflating: build/bin/ivalue_test 2024-06-26T06:39:21.7213306Z inflating: build/bin/lazy_tensor_test 2024-06-26T06:39:21.7281502Z inflating: build/bin/math_kernel_test 2024-06-26T06:39:21.7349441Z inflating: build/bin/memory_format_test 2024-06-26T06:39:21.7415510Z inflating: build/bin/memory_overlapping_test 2024-06-26T06:39:21.7481912Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T06:39:21.7545201Z inflating: build/bin/operator_name_test 2024-06-26T06:39:21.7614873Z inflating: build/bin/native_test 2024-06-26T06:39:21.7679675Z inflating: build/bin/operators_test 2024-06-26T06:39:21.7744686Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T06:39:21.7828839Z inflating: build/bin/pow_test 2024-06-26T06:39:21.7900441Z inflating: build/bin/quantized_test 2024-06-26T06:39:21.7963293Z inflating: build/bin/reduce_ops_test 2024-06-26T06:39:21.8027378Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T06:39:21.8097948Z inflating: build/bin/scalar_tensor_test 2024-06-26T06:39:21.8171052Z inflating: build/bin/scalar_test 2024-06-26T06:39:21.8235994Z inflating: build/bin/StorageUtils_test 2024-06-26T06:39:21.8301581Z inflating: build/bin/stride_properties_test 2024-06-26T06:39:21.8400942Z inflating: build/bin/tensor_iterator_test 2024-06-26T06:39:21.8404698Z inflating: build/bin/thread_init_test 2024-06-26T06:39:21.8472587Z inflating: build/bin/test_parallel 2024-06-26T06:39:21.8542292Z inflating: build/bin/type_ptr_test 2024-06-26T06:39:21.8617669Z inflating: build/bin/type_test 2024-06-26T06:39:21.8683084Z inflating: build/bin/undefined_tensor_test 2024-06-26T06:39:21.8685102Z inflating: build/bin/verify_api_visibility 2024-06-26T06:39:21.8749587Z inflating: build/bin/weakref_test 2024-06-26T06:39:21.8836377Z inflating: build/bin/legacy_vmap_test 2024-06-26T06:39:21.8900530Z inflating: build/bin/wrapdim_test 2024-06-26T06:39:21.8965319Z inflating: build/bin/xla_tensor_test 2024-06-26T06:39:21.9040626Z inflating: build/bin/IListRef_test 2024-06-26T06:39:21.9173608Z inflating: build/bin/List_test 2024-06-26T06:39:21.9326063Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T06:39:21.9409326Z inflating: build/bin/KernelFunction_test 2024-06-26T06:39:21.9530492Z inflating: build/bin/kernel_function_test 2024-06-26T06:39:21.9691275Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T06:39:21.9820969Z inflating: build/bin/kernel_lambda_test 2024-06-26T06:39:21.9897500Z inflating: build/bin/kernel_stackbased_test 2024-06-26T06:39:22.0018014Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T06:39:22.0082018Z inflating: build/bin/CppSignature_test 2024-06-26T06:39:22.0151931Z inflating: build/bin/backend_fallback_test 2024-06-26T06:39:22.0212886Z inflating: build/bin/op_allowlist_test 2024-06-26T06:39:22.0292565Z inflating: build/bin/inline_container_test 2024-06-26T06:39:22.0670596Z inflating: build/bin/op_registration_test 2024-06-26T06:39:22.0736558Z inflating: build/bin/cuda_apply_test 2024-06-26T06:39:22.0800185Z inflating: build/bin/cuda_allocator_test 2024-06-26T06:39:22.0874230Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T06:39:22.0942559Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T06:39:22.1031450Z inflating: build/bin/cuda_complex_math_test 2024-06-26T06:39:22.1093475Z inflating: build/bin/cuda_device_test 2024-06-26T06:39:22.1167415Z inflating: build/bin/cuda_complex_test 2024-06-26T06:39:22.1239850Z inflating: build/bin/cuda_cub_test 2024-06-26T06:39:22.1303529Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T06:39:22.1368936Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T06:39:22.1450653Z inflating: build/bin/cuda_distributions_test 2024-06-26T06:39:22.1521553Z inflating: build/bin/cuda_generator_test 2024-06-26T06:39:22.1583927Z inflating: build/bin/cuda_half_test 2024-06-26T06:39:22.1645446Z inflating: build/bin/cuda_optional_test 2024-06-26T06:39:22.1711292Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T06:39:22.1773753Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T06:39:22.1838348Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T06:39:22.1914208Z inflating: build/bin/cuda_stream_test 2024-06-26T06:39:22.1975759Z inflating: build/bin/cuda_cudnn_test 2024-06-26T06:39:22.2040977Z inflating: build/bin/cuda_vectorized_test 2024-06-26T06:39:22.2059973Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T06:39:22.2141741Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T06:39:22.2221350Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T06:39:22.2293708Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T06:39:22.2359638Z inflating: build/bin/ProcessGroupUCCTest 2024-06-26T06:39:22.2437524Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T06:39:22.2506011Z inflating: build/bin/test_dist_autograd 2024-06-26T06:39:22.2592655Z inflating: build/bin/test_cpp_rpc 2024-06-26T06:39:22.2596062Z inflating: build/bin/parallel_benchmark 2024-06-26T06:39:22.3635216Z inflating: build/bin/test_tensorexpr 2024-06-26T06:39:22.3646304Z inflating: build/bin/aot_model_compiler_test 2024-06-26T06:39:22.3729941Z inflating: build/bin/test_mobile_nnc 2024-06-26T06:39:22.4163252Z inflating: build/bin/test_lazy 2024-06-26T06:39:22.5645466Z inflating: build/bin/test_api 2024-06-26T06:39:22.6366842Z inflating: build/bin/test_jit 2024-06-26T06:39:22.6367399Z creating: .additional_ci_files/ 2024-06-26T06:39:22.6436585Z inflating: .additional_ci_files/test-times.json 2024-06-26T06:39:22.6707670Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T06:39:22.6746973Z ##[group]Run rm artifacts.zip 2024-06-26T06:39:22.6747468Z rm artifacts.zip 2024-06-26T06:39:22.6755813Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:22.6756460Z env: 2024-06-26T06:39:22.6756805Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:22.6757346Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:22.6757952Z ##[endgroup] 2024-06-26T06:39:22.7943553Z ##[group]Run df -H 2024-06-26T06:39:22.7943951Z df -H 2024-06-26T06:39:22.7952008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:22.7952632Z env: 2024-06-26T06:39:22.7952968Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:22.7953503Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:22.7954101Z ##[endgroup] 2024-06-26T06:39:22.7988598Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T06:39:22.7989226Z devtmpfs 129G 0 129G 0% /dev 2024-06-26T06:39:22.7990035Z tmpfs 129G 4.1M 129G 1% /dev/shm 2024-06-26T06:39:22.7990641Z tmpfs 129G 533k 129G 1% /run 2024-06-26T06:39:22.7991464Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-06-26T06:39:22.7992199Z /dev/xvda1 162G 37G 125G 23% / 2024-06-26T06:39:22.8060482Z Prepare all required actions 2024-06-26T06:39:22.8061053Z Getting action download info 2024-06-26T06:39:22.9245740Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T06:39:22.9246331Z with: 2024-06-26T06:39:22.9246626Z env: 2024-06-26T06:39:22.9246962Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:22.9247512Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:22.9248119Z ##[endgroup] 2024-06-26T06:39:22.9291436Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T06:39:22.9292001Z with: 2024-06-26T06:39:22.9292311Z name: td_results 2024-06-26T06:39:22.9292703Z s3-bucket: gha-artifacts 2024-06-26T06:39:22.9293132Z region: us-east-1 2024-06-26T06:39:22.9293476Z env: 2024-06-26T06:39:22.9293790Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:22.9294336Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:22.9294922Z ##[endgroup] 2024-06-26T06:39:23.5755426Z (node:102455) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T06:39:23.5756353Z 2024-06-26T06:39:23.5756680Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T06:39:23.5757597Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T06:39:23.5758675Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T06:39:23.6653144Z Found 1 objects with prefix pytorch/pytorch/9673645538/td_results/ 2024-06-26T06:39:23.6654327Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T06:39:23.7290667Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T06:39:23.7301358Z Artifact download has finished successfully 2024-06-26T06:39:23.7489491Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T06:39:23.7490114Z mkdir -p .additional_ci_files 2024-06-26T06:39:23.7490829Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T06:39:23.7499497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:23.7500102Z env: 2024-06-26T06:39:23.7500439Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:23.7501001Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:23.7501586Z ##[endgroup] 2024-06-26T06:39:23.7588734Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T06:39:23.7589370Z .github/scripts/parse_ref.py 2024-06-26T06:39:23.7597173Z shell: /usr/bin/bash -e {0} 2024-06-26T06:39:23.7597622Z env: 2024-06-26T06:39:23.7597968Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:23.7598516Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:23.7599125Z ##[endgroup] 2024-06-26T06:39:23.7911890Z Prepare all required actions 2024-06-26T06:39:23.7963151Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T06:39:23.7963741Z with: 2024-06-26T06:39:23.7964471Z github-token: *** 2024-06-26T06:39:23.7964842Z env: 2024-06-26T06:39:23.7965176Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:23.7965734Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:23.7966316Z ##[endgroup] 2024-06-26T06:39:23.7989291Z ##[group]Run set -eux 2024-06-26T06:39:23.7989712Z set -eux 2024-06-26T06:39:23.7990452Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T06:39:23.7998256Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:23.7998868Z env: 2024-06-26T06:39:23.7999211Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:23.7999779Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:23.8000589Z GITHUB_TOKEN: *** 2024-06-26T06:39:23.8000981Z ##[endgroup] 2024-06-26T06:39:23.8025922Z + python3 .github/scripts/get_workflow_job_id.py 9673645538 i-02a7758459db2295e 2024-06-26T06:39:26.6761549Z setting job-id=26688546431 2024-06-26T06:39:26.6762970Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:26.7026640Z Prepare all required actions 2024-06-26T06:39:26.7027240Z Getting action download info 2024-06-26T06:39:26.8150452Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T06:39:26.8151018Z with: 2024-06-26T06:39:26.8151584Z github-token: *** 2024-06-26T06:39:26.8153403Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-06-26T06:39:26.8155728Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:26.8156626Z env: 2024-06-26T06:39:26.8156961Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:26.8157497Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:26.8158110Z ##[endgroup] 2024-06-26T06:39:26.8212347Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:39:26.8213033Z with: 2024-06-26T06:39:26.8213358Z shell: bash 2024-06-26T06:39:26.8213720Z timeout_minutes: 10 2024-06-26T06:39:26.8214102Z max_attempts: 5 2024-06-26T06:39:26.8214484Z retry_wait_seconds: 30 2024-06-26T06:39:26.8215908Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T06:39:26.8217408Z polling_interval_seconds: 1 2024-06-26T06:39:26.8217851Z warning_on_retry: true 2024-06-26T06:39:26.8218277Z continue_on_error: false 2024-06-26T06:39:26.8218688Z env: 2024-06-26T06:39:26.8218998Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:26.8219549Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:26.8220437Z GITHUB_TOKEN: *** 2024-06-26T06:39:26.8220822Z ##[endgroup] 2024-06-26T06:39:26.8953106Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T06:39:27.1919205Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T06:39:27.2130095Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T06:39:27.2317428Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T06:39:27.2329000Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T06:39:27.2346820Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T06:39:27.2373293Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T06:39:27.2627491Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T06:39:27.8948279Z Command completed after 1 attempt(s). 2024-06-26T06:39:27.9012362Z ##[group]Run set -x 2024-06-26T06:39:27.9012820Z set -x 2024-06-26T06:39:27.9013217Z  2024-06-26T06:39:27.9013935Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T06:39:27.9014799Z # in runner workspace 2024-06-26T06:39:27.9015490Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T06:39:27.9025135Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:27.9025777Z env: 2024-06-26T06:39:27.9026144Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:27.9026740Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:27.9027348Z ##[endgroup] 2024-06-26T06:39:27.9080750Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T06:39:27.9393888Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T06:39:27.9394604Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T06:39:27.9395193Z echo "Job name: ${JOB_NAME}" 2024-06-26T06:39:27.9395781Z  2024-06-26T06:39:27.9396468Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T06:39:27.9397311Z # in runner workspace 2024-06-26T06:39:27.9398039Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T06:39:27.9398853Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T06:39:27.9399407Z  --job-name "${JOB_NAME}" \ 2024-06-26T06:39:27.9401420Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-06-26T06:39:27.9403498Z  --selected-test-configs "" \ 2024-06-26T06:39:27.9404081Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T06:39:27.9404590Z  --tag "${TAG}" \ 2024-06-26T06:39:27.9405088Z  --event-name "${EVENT_NAME}" \ 2024-06-26T06:39:27.9405650Z  --schedule "${SCHEDULE}" \ 2024-06-26T06:39:27.9406164Z  --branch "${HEAD_BRANCH}" 2024-06-26T06:39:27.9414465Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:27.9415106Z env: 2024-06-26T06:39:27.9415431Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:27.9416006Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:27.9416889Z GITHUB_TOKEN: *** 2024-06-26T06:39:27.9417717Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:27.9418638Z PR_NUMBER: 129470 2024-06-26T06:39:27.9419022Z TAG: 2024-06-26T06:39:27.9419378Z EVENT_NAME: pull_request 2024-06-26T06:39:27.9419808Z SCHEDULE: 2024-06-26T06:39:27.9420174Z HEAD_BRANCH: 2024-06-26T06:39:27.9420526Z ##[endgroup] 2024-06-26T06:39:27.9447190Z Workflow: pull 2024-06-26T06:39:27.9448192Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:28.1717384Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T06:39:28.3309166Z ##[group]Run echo "Filtered matrix:" 2024-06-26T06:39:28.3309723Z echo "Filtered matrix:" 2024-06-26T06:39:28.3311634Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-06-26T06:39:28.3313541Z  2024-06-26T06:39:28.3313876Z echo 2024-06-26T06:39:28.3314354Z echo "Is the current job unstable? False" 2024-06-26T06:39:28.3314905Z  2024-06-26T06:39:28.3315239Z echo 2024-06-26T06:39:28.3315676Z echo "Is keep-going label set? False" 2024-06-26T06:39:28.3316210Z  2024-06-26T06:39:28.3316540Z echo 2024-06-26T06:39:28.3316925Z echo "Renabled issues? " 2024-06-26T06:39:28.3324855Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:28.3325478Z env: 2024-06-26T06:39:28.3325818Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:28.3326356Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:28.3326953Z ##[endgroup] 2024-06-26T06:39:28.3352098Z Filtered matrix: 2024-06-26T06:39:28.3354054Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-06-26T06:39:28.3355947Z 2024-06-26T06:39:28.3356267Z Is the current job unstable? False 2024-06-26T06:39:28.3356661Z 2024-06-26T06:39:28.3356944Z Is keep-going label set? False 2024-06-26T06:39:28.3357264Z 2024-06-26T06:39:28.3357428Z Renabled issues? 2024-06-26T06:39:28.3411855Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T06:39:28.3412764Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T06:39:28.3420326Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:39:28.3420928Z env: 2024-06-26T06:39:28.3421265Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:28.3421817Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:28.3422666Z JOB_TIMEOUT: 360 2024-06-26T06:39:28.3423048Z ##[endgroup] 2024-06-26T06:39:28.3517254Z ##[group]Run set -x 2024-06-26T06:39:28.3517784Z set -x 2024-06-26T06:39:28.3518156Z  2024-06-26T06:39:28.3518587Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T06:39:28.3519244Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T06:39:28.3519944Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T06:39:28.3520579Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T06:39:28.3521061Z else 2024-06-26T06:39:28.3521479Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T06:39:28.3521997Z fi 2024-06-26T06:39:28.3522313Z  2024-06-26T06:39:28.3522889Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T06:39:28.3523841Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T06:39:28.3524667Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T06:39:28.3525401Z # shellcheck disable=SC2086,SC2090 2024-06-26T06:39:28.3525964Z container_name=$(docker run \ 2024-06-26T06:39:28.3526483Z  ${GPU_FLAG:-} \ 2024-06-26T06:39:28.3526952Z  -e BUILD_ENVIRONMENT \ 2024-06-26T06:39:28.3527426Z  -e PR_NUMBER \ 2024-06-26T06:39:28.3527873Z  -e GITHUB_ACTIONS \ 2024-06-26T06:39:28.3528355Z  -e GITHUB_REPOSITORY \ 2024-06-26T06:39:28.3528836Z  -e GITHUB_WORKFLOW \ 2024-06-26T06:39:28.3529309Z  -e GITHUB_JOB \ 2024-06-26T06:39:28.3529748Z  -e GITHUB_RUN_ID \ 2024-06-26T06:39:28.3530217Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T06:39:28.3530697Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T06:39:28.3531189Z  -e JOB_ID \ 2024-06-26T06:39:28.3531600Z  -e JOB_NAME \ 2024-06-26T06:39:28.3532005Z  -e BASE_SHA \ 2024-06-26T06:39:28.3532424Z  -e BRANCH \ 2024-06-26T06:39:28.3532831Z  -e SHA1 \ 2024-06-26T06:39:28.3533233Z  -e AWS_DEFAULT_REGION \ 2024-06-26T06:39:28.3533730Z  -e IN_WHEEL_TEST \ 2024-06-26T06:39:28.3534192Z  -e SHARD_NUMBER \ 2024-06-26T06:39:28.3534631Z  -e TEST_CONFIG \ 2024-06-26T06:39:28.3535080Z  -e NUM_TEST_SHARDS \ 2024-06-26T06:39:28.3535563Z  -e REENABLED_ISSUES \ 2024-06-26T06:39:28.3536051Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T06:39:28.3536571Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T06:39:28.3537061Z  -e NO_TEST_TIMEOUT \ 2024-06-26T06:39:28.3537507Z  -e NO_TD \ 2024-06-26T06:39:28.3537925Z  -e TD_DISTRIBUTED \ 2024-06-26T06:39:28.3538392Z  -e PR_LABELS \ 2024-06-26T06:39:28.3538875Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T06:39:28.3539436Z  -e SCCACHE_BUCKET \ 2024-06-26T06:39:28.3539921Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T06:39:28.3540406Z  -e XLA_CUDA \ 2024-06-26T06:39:28.3540901Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T06:39:28.3541531Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T06:39:28.3542159Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T06:39:28.3543416Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T06:39:28.3543994Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T06:39:28.3544488Z  -e DASHBOARD_TAG \ 2024-06-26T06:39:28.3545072Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T06:39:28.3545758Z  --security-opt seccomp=unconfined \ 2024-06-26T06:39:28.3546311Z  --cap-add=SYS_PTRACE \ 2024-06-26T06:39:28.3546794Z  --ipc=host \ 2024-06-26T06:39:28.3547227Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T06:39:28.3547717Z  --tty \ 2024-06-26T06:39:28.3548089Z  --detach \ 2024-06-26T06:39:28.3548529Z  --name="${container_name}" \ 2024-06-26T06:39:28.3549040Z  --user jenkins \ 2024-06-26T06:39:28.3549757Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T06:39:28.3550474Z  -w /var/lib/jenkins/workspace \ 2024-06-26T06:39:28.3551022Z  "${DOCKER_IMAGE}" 2024-06-26T06:39:28.3551428Z ) 2024-06-26T06:39:28.3551924Z # Propagate download.pytorch.org IP to container 2024-06-26T06:39:28.3553084Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T06:39:28.3554288Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T06:39:28.3555430Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T06:39:28.3563545Z shell: /usr/bin/bash -e {0} 2024-06-26T06:39:28.3563984Z env: 2024-06-26T06:39:28.3564317Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:39:28.3564857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:28.3565589Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:39:28.3566192Z PR_NUMBER: 129470 2024-06-26T06:39:28.3566590Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T06:39:28.3567109Z GITHUB_WORKFLOW: pull 2024-06-26T06:39:28.3567522Z GITHUB_JOB: test 2024-06-26T06:39:28.3567886Z GITHUB_RUN_ID: 9673645538 2024-06-26T06:39:28.3568328Z GITHUB_RUN_NUMBER: 219936 2024-06-26T06:39:28.3568771Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T06:39:28.3569161Z JOB_ID: 26688546431 2024-06-26T06:39:28.3569975Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:28.3570887Z BRANCH: pull/129470 2024-06-26T06:39:28.3571333Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:39:28.3571972Z BASE_SHA: 4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:39:28.3572551Z TEST_CONFIG: distributed 2024-06-26T06:39:28.3572959Z SHARD_NUMBER: 1 2024-06-26T06:39:28.3573337Z NUM_TEST_SHARDS: 3 2024-06-26T06:39:28.3573731Z REENABLED_ISSUES: 2024-06-26T06:39:28.3574136Z CONTINUE_THROUGH_ERROR: False 2024-06-26T06:39:28.3574618Z VERBOSE_TEST_LOGS: False 2024-06-26T06:39:28.3575048Z NO_TEST_TIMEOUT: False 2024-06-26T06:39:28.3575449Z NO_TD: False 2024-06-26T06:39:28.3575817Z TD_DISTRIBUTED: False 2024-06-26T06:39:28.3576325Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T06:39:28.3576927Z SCCACHE_S3_KEY_PREFIX: pull 2024-06-26T06:39:28.3577353Z SHM_SIZE: 2g 2024-06-26T06:39:28.3578549Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:39:28.3579861Z XLA_CUDA: 2024-06-26T06:39:28.3580430Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:39:28.3581187Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T06:39:28.3581724Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T06:39:28.3582505Z DASHBOARD_TAG: 2024-06-26T06:39:28.3583023Z HUGGING_FACE_HUB_TOKEN: 2024-06-26T06:39:28.3583449Z ##[endgroup] 2024-06-26T06:39:28.3608497Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-06-26T06:39:28.3609266Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-06-26T06:39:28.3610089Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T06:39:28.3618966Z +++ nproc --ignore=2 2024-06-26T06:39:28.3637417Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673645538 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:39:39.4350829Z + container_name=15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:39:39.4353191Z + grep download.pytorch.org /etc/hosts 2024-06-26T06:39:39.4354972Z + docker exec -i 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T06:39:39.5186327Z + echo DOCKER_CONTAINER_ID=15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:39:39.5190527Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T06:39:39.5193272Z + docker exec -t 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T06:39:40.0717186Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T06:39:41.1980732Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T06:39:41.1987750Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T06:39:41.1991885Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T06:39:41.1996639Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T06:39:41.2001291Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T06:39:41.2006115Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T06:39:41.2044274Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T06:39:41.2128546Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T06:39:41.2739467Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T06:39:41.2969625Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T06:39:42.3985971Z Installing collected packages: torch 2024-06-26T06:39:55.1188884Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T06:39:55.1908125Z ++ dirname .ci/pytorch/test.sh 2024-06-26T06:39:55.1915367Z + source .ci/pytorch/common.sh 2024-06-26T06:39:55.1918708Z +++ dirname .ci/pytorch/common.sh 2024-06-26T06:39:55.1926283Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T06:39:55.1929051Z +++ declare -f -t trap_add 2024-06-26T06:39:55.1936556Z ++ set -ex 2024-06-26T06:39:55.1937087Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:39:55.1937681Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T06:39:55.1938463Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-06-26T06:39:55.1942049Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T06:39:55.1954432Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T06:39:55.1955005Z + trap_add cleanup_workspace EXIT 2024-06-26T06:39:55.1955528Z + trap_add_cmd=cleanup_workspace 2024-06-26T06:39:55.1955964Z + shift 2024-06-26T06:39:55.1956306Z + for trap_add_name in "$@" 2024-06-26T06:39:55.1963322Z +++ trap -p EXIT 2024-06-26T06:39:55.1966712Z ++ eval 'extract_trap_cmd ' 2024-06-26T06:39:55.1967167Z +++ extract_trap_cmd 2024-06-26T06:39:55.1967914Z +++ printf '%s\n' '' 2024-06-26T06:39:55.1968596Z ++ printf '%s\n' cleanup_workspace 2024-06-26T06:39:55.1970003Z + trap -- ' 2024-06-26T06:39:55.1970458Z cleanup_workspace' EXIT 2024-06-26T06:39:55.1971029Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T06:39:55.7381296Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T06:39:55.7399561Z + echo 'Environment variables:' 2024-06-26T06:39:55.7400094Z Environment variables: 2024-06-26T06:39:55.7400495Z + env 2024-06-26T06:39:55.7408442Z INSTALLED_DB=yes 2024-06-26T06:39:55.7409443Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-06-26T06:39:55.7410030Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T06:39:55.7410712Z NV_NVML_DEV_VERSION=11.8.86-1 2024-06-26T06:39:55.7411709Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:39:55.7412422Z CONTINUE_THROUGH_ERROR=False 2024-06-26T06:39:55.7413029Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-06-26T06:39:55.7413672Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:39:55.7414342Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:39:55.7414932Z HOSTNAME=15d2f4a9275f 2024-06-26T06:39:55.7416294Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7417666Z GITHUB_ACTION=__self 2024-06-26T06:39:55.7418095Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T06:39:55.7420814Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-06-26T06:39:55.7424254Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-06-26T06:39:55.7424894Z NV_NVTX_VERSION=11.8.86-1 2024-06-26T06:39:55.7425495Z GITHUB_RUN_NUMBER=219936 2024-06-26T06:39:55.7426311Z TEST_CONFIG=distributed 2024-06-26T06:39:55.7427146Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T06:39:55.7428377Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T06:39:55.7429573Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-06-26T06:39:55.7430553Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-06-26T06:39:55.7431358Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-06-26T06:39:55.7432426Z GITHUB_TRIGGERING_ACTOR=leslie-fang-intel 2024-06-26T06:39:55.7433725Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T06:39:55.7434315Z GITHUB_REF_TYPE=branch 2024-06-26T06:39:55.7434737Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T06:39:55.7435223Z NCCL_VERSION=2.16.2-1 2024-06-26T06:39:55.7436015Z BASE_SHA=4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:39:55.7436563Z XLA_CUDA= 2024-06-26T06:39:55.7436907Z HUGGING_FACE_HUB_TOKEN= 2024-06-26T06:39:55.7439419Z *** 2024-06-26T06:39:55.7439793Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T06:39:55.7440283Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T06:39:55.7440716Z GITHUB_ACTIONS=true 2024-06-26T06:39:55.7441117Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:55.7441754Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-06-26T06:39:55.7442399Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-06-26T06:39:55.7442976Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:39:55.7443607Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T06:39:55.7444193Z GITHUB_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:39:55.7445303Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/129470/merge 2024-06-26T06:39:55.7446141Z UCC_HOME=/usr 2024-06-26T06:39:55.7446561Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-06-26T06:39:55.7447108Z VERBOSE_TEST_LOGS=False 2024-06-26T06:39:55.7447557Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T06:39:55.7448220Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-06-26T06:39:55.7448829Z GITHUB_REF=refs/pull/129470/merge 2024-06-26T06:39:55.7449387Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-06-26T06:39:55.7449879Z SHARD_NUMBER=1 2024-06-26T06:39:55.7450262Z GITHUB_REF_PROTECTED=false 2024-06-26T06:39:55.7450719Z HOME=/var/lib/jenkins 2024-06-26T06:39:55.7451199Z GITHUB_API_URL=https://api.github.com 2024-06-26T06:39:55.7451752Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T06:39:55.7452378Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T06:39:55.7453006Z SCCACHE_S3_KEY_PREFIX=pull 2024-06-26T06:39:55.7453439Z CUDA_VERSION=11.8.0 2024-06-26T06:39:55.7454003Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-06-26T06:39:55.7454597Z NUM_TEST_SHARDS=3 2024-06-26T06:39:55.7454969Z UCX_HOME=/usr 2024-06-26T06:39:55.7455644Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-06-26T06:39:55.7457116Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7458749Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:55.7460385Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7461913Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T06:39:55.7463369Z GITHUB_EVENT_NAME=pull_request 2024-06-26T06:39:55.7463821Z DASHBOARD_TAG= 2024-06-26T06:39:55.7464186Z GITHUB_RUN_ID=9673645538 2024-06-26T06:39:55.7464770Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-06-26T06:39:55.7465431Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-06-26T06:39:55.7466719Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7467912Z GITHUB_ACTOR=leslie-fang-intel 2024-06-26T06:39:55.7468405Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-06-26T06:39:55.7468879Z PR_NUMBER=129470 2024-06-26T06:39:55.7469253Z GITHUB_RUN_ATTEMPT=1 2024-06-26T06:39:55.7469647Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T06:39:55.7470198Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T06:39:55.7470766Z TERM=xterm 2024-06-26T06:39:55.7471175Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-06-26T06:39:55.7471714Z INSTALLED_VISION=yes 2024-06-26T06:39:55.7472131Z BRANCH=pull/129470 2024-06-26T06:39:55.7472531Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T06:39:55.7473057Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T06:39:55.7473617Z CUDA_PATH=/usr/local/cuda 2024-06-26T06:39:55.7474608Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T06:39:55.7475622Z GITHUB_SERVER_URL=https://github.com 2024-06-26T06:39:55.7476258Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T06:39:55.7477061Z REENABLED_ISSUES= 2024-06-26T06:39:55.7477444Z SHLVL=1 2024-06-26T06:39:55.7477771Z MAX_JOBS=30 2024-06-26T06:39:55.7478175Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-06-26T06:39:55.7478644Z NVARCH=x86_64 2024-06-26T06:39:55.7479028Z GITHUB_ACTOR_ID=53841472 2024-06-26T06:39:55.7479616Z GITHUB_WORKFLOW_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:39:55.7480301Z GITHUB_REF_NAME=129470/merge 2024-06-26T06:39:55.7480883Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-06-26T06:39:55.7481743Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:39:55.7482503Z GITHUB_JOB=test 2024-06-26T06:39:55.7483031Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-06-26T06:39:55.7483745Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T06:39:55.7484422Z NO_TEST_TIMEOUT=False 2024-06-26T06:39:55.7485011Z TD_DISTRIBUTED=False 2024-06-26T06:39:55.7485529Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-06-26T06:39:55.7486123Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T06:39:55.7486675Z NV_NVPROF_VERSION=11.8.87-1 2024-06-26T06:39:55.7487130Z GITHUB_RETENTION_DAYS=90 2024-06-26T06:39:55.7487586Z OPENSSL_DIR=/opt/openssl 2024-06-26T06:39:55.7488038Z GITHUB_ACTION_REPOSITORY= 2024-06-26T06:39:55.7489423Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:39:55.7491034Z GITHUB_BASE_REF=gh/leslie-fang-intel/126/base 2024-06-26T06:39:55.7491642Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T06:39:55.7492100Z CI=true 2024-06-26T06:39:55.7492508Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:39:55.7493047Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T06:39:55.7493514Z JOB_ID=26688546431 2024-06-26T06:39:55.7493913Z INSTALLED_PROTOBUF=yes 2024-06-26T06:39:55.7494497Z GITHUB_HEAD_REF=gh/leslie-fang-intel/126/head 2024-06-26T06:39:55.7495066Z GITHUB_ACTION_REF= 2024-06-26T06:39:55.7495638Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T06:39:55.7496257Z GITHUB_WORKFLOW=pull 2024-06-26T06:39:55.7496733Z DEBIAN_FRONTEND=noninteractive 2024-06-26T06:39:55.7497903Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7498952Z NO_TD=False 2024-06-26T06:39:55.7499302Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T06:39:55.7499749Z _=/usr/bin/env 2024-06-26T06:39:55.7500345Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T06:39:55.7629805Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T06:39:55.7632355Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:39:55.7633644Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T06:39:55.7634782Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T06:39:55.7635534Z + BUILD_DIR=build 2024-06-26T06:39:55.7635941Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T06:39:55.7636839Z + BUILD_BIN_DIR=build/bin 2024-06-26T06:39:55.7637652Z + SHARD_NUMBER=1 2024-06-26T06:39:55.7638038Z + NUM_TEST_SHARDS=3 2024-06-26T06:39:55.7638428Z + export VALGRIND=ON 2024-06-26T06:39:55.7638798Z + VALGRIND=ON 2024-06-26T06:39:55.7639340Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-06-26T06:39:55.7639916Z + [[ 0 == \1 ]] 2024-06-26T06:39:55.7640254Z + [[ False == \1 ]] 2024-06-26T06:39:55.7640804Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-06-26T06:39:55.7641417Z ++ realpath build/custom_test_artifacts 2024-06-26T06:39:55.7646414Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T06:39:55.7647630Z + [[ -n '' ]] 2024-06-26T06:39:55.7648057Z + echo 'Environment variables' 2024-06-26T06:39:55.7648508Z Environment variables 2024-06-26T06:39:55.7648902Z + env 2024-06-26T06:39:55.7653844Z INSTALLED_DB=yes 2024-06-26T06:39:55.7654700Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-06-26T06:39:55.7655961Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T06:39:55.7656915Z NV_NVML_DEV_VERSION=11.8.86-1 2024-06-26T06:39:55.7658327Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:39:55.7659265Z CONTINUE_THROUGH_ERROR=False 2024-06-26T06:39:55.7660230Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-06-26T06:39:55.7661582Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:39:55.7662909Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:39:55.7663511Z HOSTNAME=15d2f4a9275f 2024-06-26T06:39:55.7664576Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7665666Z GITHUB_ACTION=__self 2024-06-26T06:39:55.7666092Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T06:39:55.7669178Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-06-26T06:39:55.7672148Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-06-26T06:39:55.7672771Z NV_NVTX_VERSION=11.8.86-1 2024-06-26T06:39:55.7673205Z GITHUB_RUN_NUMBER=219936 2024-06-26T06:39:55.7673630Z TEST_CONFIG=distributed 2024-06-26T06:39:55.7674049Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T06:39:55.7674635Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T06:39:55.7675204Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-06-26T06:39:55.7675733Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-06-26T06:39:55.7676255Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-06-26T06:39:55.7676814Z GITHUB_TRIGGERING_ACTOR=leslie-fang-intel 2024-06-26T06:39:55.7677419Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T06:39:55.7678003Z GITHUB_REF_TYPE=branch 2024-06-26T06:39:55.7678429Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T06:39:55.7678884Z NCCL_VERSION=2.16.2-1 2024-06-26T06:39:55.7679355Z BASE_SHA=4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:39:55.7679900Z XLA_CUDA= 2024-06-26T06:39:55.7680225Z HUGGING_FACE_HUB_TOKEN= 2024-06-26T06:39:55.7680727Z *** 2024-06-26T06:39:55.7681072Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T06:39:55.7681539Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T06:39:55.7681987Z GITHUB_ACTIONS=true 2024-06-26T06:39:55.7682389Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:39:55.7682982Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-06-26T06:39:55.7683647Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-06-26T06:39:55.7684226Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:39:55.7684834Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T06:39:55.7685453Z GITHUB_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:39:55.7686347Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/129470/merge 2024-06-26T06:39:55.7687153Z UCC_HOME=/usr 2024-06-26T06:39:55.7687555Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-06-26T06:39:55.7688051Z VERBOSE_TEST_LOGS=False 2024-06-26T06:39:55.7688471Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T06:39:55.7689018Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-06-26T06:39:55.7689600Z GITHUB_REF=refs/pull/129470/merge 2024-06-26T06:39:55.7690120Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-06-26T06:39:55.7690561Z SHARD_NUMBER=1 2024-06-26T06:39:55.7690931Z GITHUB_REF_PROTECTED=false 2024-06-26T06:39:55.7691361Z HOME=/var/lib/jenkins 2024-06-26T06:39:55.7691792Z GITHUB_API_URL=https://api.github.com 2024-06-26T06:39:55.7692331Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T06:39:55.7692919Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T06:39:55.7693496Z SCCACHE_S3_KEY_PREFIX=pull 2024-06-26T06:39:55.7693928Z CUDA_VERSION=11.8.0 2024-06-26T06:39:55.7694466Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-06-26T06:39:55.7694998Z NUM_TEST_SHARDS=3 2024-06-26T06:39:55.7695528Z UCX_HOME=/usr 2024-06-26T06:39:55.7696155Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-06-26T06:39:55.7697590Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7699152Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:39:55.7700700Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7702112Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T06:39:55.7703451Z GITHUB_EVENT_NAME=pull_request 2024-06-26T06:39:55.7703907Z DASHBOARD_TAG= 2024-06-26T06:39:55.7704255Z GITHUB_RUN_ID=9673645538 2024-06-26T06:39:55.7704978Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-06-26T06:39:55.7705684Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-06-26T06:39:55.7706939Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7708365Z GITHUB_ACTOR=leslie-fang-intel 2024-06-26T06:39:55.7708899Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-06-26T06:39:55.7709348Z PR_NUMBER=129470 2024-06-26T06:39:55.7709720Z GITHUB_RUN_ATTEMPT=1 2024-06-26T06:39:55.7710106Z VALGRIND=ON 2024-06-26T06:39:55.7710450Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T06:39:55.7711001Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T06:39:55.7711570Z TERM=xterm 2024-06-26T06:39:55.7711961Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-06-26T06:39:55.7712470Z INSTALLED_VISION=yes 2024-06-26T06:39:55.7712863Z BRANCH=pull/129470 2024-06-26T06:39:55.7713238Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T06:39:55.7713725Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T06:39:55.7714254Z CUDA_PATH=/usr/local/cuda 2024-06-26T06:39:55.7715187Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T06:39:55.7716146Z GITHUB_SERVER_URL=https://github.com 2024-06-26T06:39:55.7716753Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T06:39:55.7717317Z REENABLED_ISSUES= 2024-06-26T06:39:55.7717715Z SHLVL=1 2024-06-26T06:39:55.7718031Z MAX_JOBS=30 2024-06-26T06:39:55.7718422Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-06-26T06:39:55.7718835Z NVARCH=x86_64 2024-06-26T06:39:55.7719194Z GITHUB_ACTOR_ID=53841472 2024-06-26T06:39:55.7719759Z GITHUB_WORKFLOW_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:39:55.7720385Z GITHUB_REF_NAME=129470/merge 2024-06-26T06:39:55.7720918Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-06-26T06:39:55.7721742Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:39:55.7722440Z GITHUB_JOB=test 2024-06-26T06:39:55.7722931Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-06-26T06:39:55.7723624Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T06:39:55.7724245Z NO_TEST_TIMEOUT=False 2024-06-26T06:39:55.7724653Z TD_DISTRIBUTED=False 2024-06-26T06:39:55.7725124Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-06-26T06:39:55.7725646Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T06:39:55.7726163Z NV_NVPROF_VERSION=11.8.87-1 2024-06-26T06:39:55.7726603Z GITHUB_RETENTION_DAYS=90 2024-06-26T06:39:55.7727014Z OPENSSL_DIR=/opt/openssl 2024-06-26T06:39:55.7727440Z GITHUB_ACTION_REPOSITORY= 2024-06-26T06:39:55.7728769Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:39:55.7730270Z GITHUB_BASE_REF=gh/leslie-fang-intel/126/base 2024-06-26T06:39:55.7730846Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T06:39:55.7731296Z CI=true 2024-06-26T06:39:55.7731663Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:39:55.7732175Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T06:39:55.7732647Z JOB_ID=26688546431 2024-06-26T06:39:55.7733019Z INSTALLED_PROTOBUF=yes 2024-06-26T06:39:55.7733557Z GITHUB_HEAD_REF=gh/leslie-fang-intel/126/head 2024-06-26T06:39:55.7734271Z GITHUB_ACTION_REF= 2024-06-26T06:39:55.7734793Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T06:39:55.7735373Z GITHUB_WORKFLOW=pull 2024-06-26T06:39:55.7735787Z DEBIAN_FRONTEND=noninteractive 2024-06-26T06:39:55.7736925Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_cde5c1a7-50fc-4abb-b6eb-6c6b8784d2bc 2024-06-26T06:39:55.7737978Z NO_TD=False 2024-06-26T06:39:55.7738343Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T06:39:55.7738769Z _=/usr/bin/env 2024-06-26T06:39:55.7739188Z + echo 'Testing pytorch' 2024-06-26T06:39:55.7739614Z Testing pytorch 2024-06-26T06:39:55.7739992Z + export LANG=C.UTF-8 2024-06-26T06:39:55.7740408Z + LANG=C.UTF-8 2024-06-26T06:39:55.7740768Z + PR_NUMBER=129470 2024-06-26T06:39:55.7741158Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-06-26T06:39:55.7741817Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T06:39:55.7742994Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:39:55.7743588Z + [[ distributed == \s\l\o\w ]] 2024-06-26T06:39:55.7744277Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-06-26T06:39:55.7745064Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-06-26T06:39:55.7745692Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T06:39:55.7746277Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T06:39:55.7746801Z + [[ distributed == *crossref* ]] 2024-06-26T06:39:55.7747413Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:39:55.7748132Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-06-26T06:39:55.7748873Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:39:55.7749525Z + pip_install --user ninja==1.10.2 2024-06-26T06:39:55.7750184Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T06:39:56.2926615Z Collecting ninja==1.10.2 2024-06-26T06:39:56.3129167Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T06:39:56.3320849Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T06:39:57.4953625Z Installing collected packages: ninja 2024-06-26T06:39:57.5068828Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T06:39:57.5070462Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T06:39:57.5123961Z Successfully installed ninja-1.10.2 2024-06-26T06:39:57.5819790Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:39:57.5822880Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:39:57.5824750Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-06-26T06:39:57.5825335Z + install_tlparse 2024-06-26T06:39:57.5825797Z + pip_install --user tlparse==0.3.7 2024-06-26T06:39:57.5826474Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T06:39:58.0917992Z Collecting tlparse==0.3.7 2024-06-26T06:39:58.1093287Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T06:39:58.1591750Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T06:39:59.3343778Z Installing collected packages: tlparse 2024-06-26T06:39:59.3851079Z Successfully installed tlparse-0.3.7 2024-06-26T06:39:59.4572038Z ++ python -m site --user-base 2024-06-26T06:39:59.4805446Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:39:59.4807981Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-06-26T06:39:59.4809099Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-06-26T06:39:59.4809849Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:39:59.4810972Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-06-26T06:39:59.4812341Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-06-26T06:39:59.4813197Z + cd test 2024-06-26T06:39:59.4813829Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T06:40:01.5991321Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T06:40:01.5992034Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T06:40:01.5992898Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:40:01.5993656Z + pushd test 2024-06-26T06:40:01.5994485Z ~/workspace/test ~/workspace 2024-06-26T06:40:01.5995618Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T06:40:03.6895647Z + CUDA_VERSION=11.8 2024-06-26T06:40:03.6896799Z + '[' 11.8 == 12.4 ']' 2024-06-26T06:40:03.6897558Z + ISCUDA124= 2024-06-26T06:40:03.6898230Z + popd 2024-06-26T06:40:03.6898835Z ~/workspace 2024-06-26T06:40:03.6906146Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T06:40:03.6907128Z + [[ distributed == *dynamo_eager* ]] 2024-06-26T06:40:03.6908202Z + [[ distributed == *aot_eager* ]] 2024-06-26T06:40:03.6909208Z + [[ distributed == *aot_inductor* ]] 2024-06-26T06:40:03.6910259Z + [[ distributed == *inductor* ]] 2024-06-26T06:40:03.6911260Z + [[ distributed == *dynamic* ]] 2024-06-26T06:40:03.6912246Z + [[ distributed == *cpu_inductor* ]] 2024-06-26T06:40:03.6913488Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T06:40:03.6948573Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-06-26T06:40:03.6949426Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-06-26T06:40:03.6951681Z + cd test 2024-06-26T06:40:03.6952269Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T06:40:05.5726367Z PyTorch built with: 2024-06-26T06:40:05.5727012Z - GCC 9.4 2024-06-26T06:40:05.5727427Z - C++ Version: 201703 2024-06-26T06:40:05.5728495Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T06:40:05.5729876Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T06:40:05.5730721Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T06:40:05.5731347Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T06:40:05.5731956Z - NNPACK is enabled 2024-06-26T06:40:05.5732435Z - CPU capability usage: AVX2 2024-06-26T06:40:05.5732945Z - CUDA Runtime 11.8 2024-06-26T06:40:05.5733593Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-06-26T06:40:05.5734284Z - CuDNN 90.1 2024-06-26T06:40:05.5734694Z - Magma 2.6.1 2024-06-26T06:40:05.5744323Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T06:40:05.5753338Z 2024-06-26T06:40:05.8283358Z + cd test 2024-06-26T06:40:05.8284135Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T06:40:07.6716586Z ATen/Parallel: 2024-06-26T06:40:07.6717137Z at::get_num_threads() : 16 2024-06-26T06:40:07.6717623Z at::get_num_interop_threads() : 16 2024-06-26T06:40:07.6718151Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T06:40:07.6718649Z omp_get_max_threads() : 16 2024-06-26T06:40:07.6719871Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T06:40:07.6720880Z mkl_get_max_threads() : 16 2024-06-26T06:40:07.6722034Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T06:40:07.6722881Z std::thread::hardware_concurrency() : 32 2024-06-26T06:40:07.6723427Z Environment variables: 2024-06-26T06:40:07.6723865Z OMP_NUM_THREADS : [not set] 2024-06-26T06:40:07.6724326Z MKL_NUM_THREADS : [not set] 2024-06-26T06:40:07.6724772Z ATen parallel backend: OpenMP 2024-06-26T06:40:07.6725107Z 2024-06-26T06:40:07.9068622Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-06-26T06:40:07.9069341Z + [[ distributed == *backward* ]] 2024-06-26T06:40:07.9069829Z + [[ distributed == *xla* ]] 2024-06-26T06:40:07.9070310Z + [[ distributed == *executorch* ]] 2024-06-26T06:40:07.9070851Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T06:40:07.9071578Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-06-26T06:40:07.9072214Z + [[ distributed == distributed ]] 2024-06-26T06:40:07.9072699Z + test_distributed 2024-06-26T06:40:07.9073180Z + echo 'Testing distributed python tests' 2024-06-26T06:40:07.9073769Z Testing distributed python tests 2024-06-26T06:40:07.9074533Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-06-26T06:40:08.0397981Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:40:08.0399408Z import pkg_resources 2024-06-26T06:40:12.4442098Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:40:12.4984284Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:40:12.5471393Z Ignoring disabled issues: [''] 2024-06-26T06:40:12.5600146Z Found test times from artifacts 2024-06-26T06:40:12.6093433Z Found test times from artifacts 2024-06-26T06:40:12.6109831Z Running 25% of tests based on TD 2024-06-26T06:40:12.6232639Z Running parallel tests on 2 processes 2024-06-26T06:40:12.6233578Z Name: tests to run (est. time: 5.99min) 2024-06-26T06:40:12.6234172Z Serial tests (13): 2024-06-26T06:40:12.6234659Z distributed/test_inductor_collectives 1/1 2024-06-26T06:40:12.6235314Z distributed/pipelining/test_backward 1/1 2024-06-26T06:40:12.6236133Z distributed/checkpoint/test_nested_dict 1/1 2024-06-26T06:40:12.6237128Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-06-26T06:40:12.6237819Z distributed/_spmd/test_graph_utils 1/1 2024-06-26T06:40:12.6238420Z distributed/_spmd/test_transformation 1/1 2024-06-26T06:40:12.6239030Z distributed/checkpoint/test_save_load_api 1/1 2024-06-26T06:40:12.6239707Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-06-26T06:40:12.6240338Z distributed/_tensor/test_init 1/1 2024-06-26T06:40:12.6240896Z distributed/_tools/test_fsdp2_mem_tracker 1/1 2024-06-26T06:40:12.6241539Z distributed/_tensor/test_experimental_ops 1/1 2024-06-26T06:40:12.6242255Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2024-06-26T06:40:12.6242936Z distributed/_tensor/test_redistribute 1/1 2024-06-26T06:40:12.6243492Z Parallel tests (0): 2024-06-26T06:40:12.6244305Z Name: excluded (est. time: 75.77min) 2024-06-26T06:40:12.6244804Z Serial tests (56): 2024-06-26T06:40:12.6245233Z distributed/test_distributed_spawn 1/9 2024-06-26T06:40:12.6245810Z distributed/test_distributed_spawn 4/9 2024-06-26T06:40:12.6246386Z distributed/test_distributed_spawn 7/9 2024-06-26T06:40:12.6246915Z distributed/test_c10d_gloo 1/1 2024-06-26T06:40:12.6247456Z distributed/fsdp/test_fsdp_state_dict 1/2 2024-06-26T06:40:12.6248085Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-06-26T06:40:12.6248729Z distributed/fsdp/test_fsdp_mixed_precision 1/1 2024-06-26T06:40:12.6249328Z distributed/test_c10d_pypg 1/1 2024-06-26T06:40:12.6249904Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-06-26T06:40:12.6250598Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-06-26T06:40:12.6251450Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-06-26T06:40:12.6252190Z distributed/_composable/test_replicate_with_compiler 1/1 2024-06-26T06:40:12.6252829Z distributed/test_store 1/1 2024-06-26T06:40:12.6253373Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-06-26T06:40:12.6254071Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 2024-06-26T06:40:12.6274659Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-06-26T06:40:12.6275434Z distributed/tensor/parallel/test_fsdp_2d_parallel 1/1 2024-06-26T06:40:12.6276104Z distributed/tensor/parallel/test_tp_style 1/1 2024-06-26T06:40:12.6276691Z distributed/algorithms/test_join 1/1 2024-06-26T06:40:12.6277243Z distributed/fsdp/test_fsdp_meta 1/1 2024-06-26T06:40:12.6277815Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-06-26T06:40:12.6278399Z distributed/test_c10d_spawn_gloo 1/1 2024-06-26T06:40:12.6279052Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-06-26T06:40:12.6279826Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 2024-06-26T06:40:12.6280570Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-06-26T06:40:12.6281217Z distributed/test_c10d_spawn_ucc 1/1 2024-06-26T06:40:12.6281870Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-06-26T06:40:12.6282619Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-06-26T06:40:12.6283383Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-06-26T06:40:12.6284058Z distributed/fsdp/test_fsdp_apply 1/1 2024-06-26T06:40:12.6284604Z distributed/fsdp/test_fsdp_memory 1/1 2024-06-26T06:40:12.6285305Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-06-26T06:40:12.6286168Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-06-26T06:40:12.6286884Z distributed/checkpoint/test_fsspec 1/1 2024-06-26T06:40:12.6287452Z distributed/test_c10d_ops_nccl 1/1 2024-06-26T06:40:12.6288066Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-06-26T06:40:12.6288825Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-06-26T06:40:12.6289638Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-06-26T06:40:12.6290327Z distributed/elastic/timer/local_timer_test 1/1 2024-06-26T06:40:12.6291078Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-06-26T06:40:12.6291827Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-06-26T06:40:12.6292499Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2024-06-26T06:40:12.6293114Z distributed/rpc/test_share_memory 1/1 2024-06-26T06:40:12.6293638Z distributed/test_launcher 1/1 2024-06-26T06:40:12.6294136Z distributed/test_fake_pg 1/1 2024-06-26T06:40:12.6294678Z distributed/_tensor/test_pointwise_ops 1/1 2024-06-26T06:40:12.6295268Z distributed/_tensor/test_op_strategy 1/1 2024-06-26T06:40:12.6295883Z distributed/checkpoint/test_compatibility 1/1 2024-06-26T06:40:12.6296508Z distributed/elastic/utils/util_test 1/1 2024-06-26T06:40:12.6297142Z distributed/_tools/test_memory_tracker 1/1 2024-06-26T06:40:12.6298018Z distributed/checkpoint/test_traverse 1/1 2024-06-26T06:40:12.6298666Z distributed/fsdp/test_distributed_checkpoint 1/1 2024-06-26T06:40:12.6299302Z distributed/optim/test_named_optimizer 1/1 2024-06-26T06:40:12.6299893Z distributed/test_collective_utils 1/1 2024-06-26T06:40:12.6300478Z distributed/rpc/test_tensorpipe_agent 1/1 2024-06-26T06:40:12.6301050Z distributed/elastic/timer/api_test 1/1 2024-06-26T06:40:12.6301590Z Parallel tests (0): 2024-06-26T06:40:12.6302805Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:40:12.6305213Z Running distributed/test_inductor_collectives 1/1 ... [2024-06-26 06:40:12.630157] 2024-06-26T06:40:12.6310797Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:40:12.630576] 2024-06-26T06:40:26.1236682Z 2024-06-26T06:40:26.1238920Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_8eddf3a0a81a18be_.log 2024-06-26T06:40:26.1274786Z Running 47 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-06-26T06:40:26.1308976Z 2024-06-26T06:40:26.1309577Z Running distributed/pipelining/test_backward 1/1 ... [2024-06-26 06:40:26.124204] 2024-06-26T06:40:26.1312045Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:40:26.124622] 2024-06-26T06:40:29.9490036Z 2024-06-26T06:40:29.9491888Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_2b66a4f31e1c4e43_.log 2024-06-26T06:40:29.9494056Z Running 1 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward 2024-06-26T06:40:29.9494953Z 2024-06-26T06:40:29.9495943Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-06-26 06:40:29.949283] 2024-06-26T06:40:29.9501210Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:40:29.949699] 2024-06-26T06:40:33.6726611Z 2024-06-26T06:40:33.6728825Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_b49d14f1af4f1a97_.log 2024-06-26T06:40:33.6732188Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-06-26T06:40:33.6733639Z 2024-06-26T06:40:33.6734322Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-06-26 06:40:33.672973] 2024-06-26T06:40:33.6776381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:40:33.677135] 2024-06-26T06:44:57.5090284Z 2024-06-26T06:44:57.5093167Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_db6b0198bf52ebbb_.log 2024-06-26T06:44:57.5138458Z Running 22 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::test_compiled_fsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_explicit_prefetching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_post_optim_event, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_1f1b_microbatching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardNDTraining::test_2d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCustomForwardMethod::test_register_fsdp_forward_method 2024-06-26T06:44:57.5321071Z 2024-06-26T06:44:57.5327741Z Running distributed/_spmd/test_graph_utils 1/1 ... [2024-06-26 06:44:57.532393] 2024-06-26T06:44:57.5333021Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_graph_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:44:57.532813] 2024-06-26T06:45:00.9005316Z 2024-06-26T06:45:00.9007111Z distributed/_spmd/test_graph_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_graph_utils_1.1_395a544e6e3e583f_.log 2024-06-26T06:45:00.9009764Z 2024-06-26T06:45:00.9011751Z Running distributed/_spmd/test_transformation 1/1 ... [2024-06-26 06:45:00.900779] 2024-06-26T06:45:00.9017573Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_spmd/test_transformation.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:00.901202] 2024-06-26T06:45:05.3942126Z 2024-06-26T06:45:05.3944497Z distributed/_spmd/test_transformation 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._spmd.test_transformation_1.1_0f4beec38f2a0633_.log 2024-06-26T06:45:05.3945923Z 2024-06-26T06:45:05.3948868Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-06-26 06:45:05.394549] 2024-06-26T06:45:05.3954078Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:05.394973] 2024-06-26T06:45:12.6743247Z 2024-06-26T06:45:12.6745396Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_cfb84c3cafef9bd9_.log 2024-06-26T06:45:12.6747622Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-06-26T06:45:12.6748554Z 2024-06-26T06:45:12.6750116Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2024-06-26 06:45:12.674675] 2024-06-26T06:45:12.6755507Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:12.675089] 2024-06-26T06:45:25.9175501Z 2024-06-26T06:45:25.9177650Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_04c5ba18e9f94897_.log 2024-06-26T06:45:25.9181078Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True 2024-06-26T06:45:25.9183515Z 2024-06-26T06:45:25.9184027Z Running distributed/_tensor/test_init 1/1 ... [2024-06-26 06:45:25.917952] 2024-06-26T06:45:25.9188678Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:25.918404] 2024-06-26T06:45:56.4520932Z 2024-06-26T06:45:56.4525127Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_1a7c1ce451a95156_.log 2024-06-26T06:45:56.4531122Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-06-26T06:45:56.4535966Z 2024-06-26T06:45:56.4536821Z Running distributed/_tools/test_fsdp2_mem_tracker 1/1 ... [2024-06-26 06:45:56.452318] 2024-06-26T06:45:56.4539826Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fsdp2_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:56.452732] 2024-06-26T06:46:39.0056841Z 2024-06-26T06:46:39.0060398Z distributed/_tools/test_fsdp2_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_011af83ca1fb677d_.log 2024-06-26T06:46:39.0066535Z Running 3 items in this shard: test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_multi_group_eager, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_non_root_forward_backward, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCompose::test_tracker_with_activation_checkpointing 2024-06-26T06:46:39.0069427Z 2024-06-26T06:46:39.0070043Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-06-26 06:46:39.005951] 2024-06-26T06:46:39.0072514Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:46:39.006397] 2024-06-26T06:46:56.0551880Z 2024-06-26T06:46:56.0553822Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_c119a86ac53e38ae_.log 2024-06-26T06:46:56.0556936Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-06-26T06:46:56.0558924Z 2024-06-26T06:46:56.0559599Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2024-06-26 06:46:56.055460] 2024-06-26T06:46:56.0564258Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:46:56.055909] 2024-06-26T06:47:10.2495788Z 2024-06-26T06:47:10.2498229Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_1d7c1c4300afff9b_.log 2024-06-26T06:47:10.2501592Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2024-06-26T06:47:10.2503819Z 2024-06-26T06:47:10.2504375Z Running distributed/_tensor/test_redistribute 1/1 ... [2024-06-26 06:47:10.249780] 2024-06-26T06:47:10.2506834Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:47:10.250188] 2024-06-26T06:47:56.9667224Z 2024-06-26T06:47:56.9669078Z distributed/_tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_redistribute_1.1_11bcea3292b11d31_.log 2024-06-26T06:47:56.9679919Z Running 11 items in this shard: test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/_tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward, test/distributed/_tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh 2024-06-26T06:47:56.9687531Z 2024-06-26T06:47:57.8954611Z 2024-06-26T06:47:57.8955081Z real 7m49.988s 2024-06-26T06:47:57.8955533Z user 15m33.679s 2024-06-26T06:47:57.8955885Z sys 5m10.349s 2024-06-26T06:47:57.8956250Z + assert_git_not_dirty 2024-06-26T06:47:57.8956957Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-06-26T06:47:57.8961741Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-06-26T06:47:57.8962452Z ++ git status --porcelain 2024-06-26T06:47:57.8962940Z ++ grep -v '?? third_party' 2024-06-26T06:48:00.7779701Z ++ true 2024-06-26T06:48:00.7780390Z + git_status= 2024-06-26T06:48:00.7780939Z + [[ -n '' ]] 2024-06-26T06:48:00.7781476Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-06-26T06:48:00.7782096Z + [[ 1 == 1 ]] 2024-06-26T06:48:00.7782911Z + echo 'Testing distributed C++ tests' 2024-06-26T06:48:00.7783432Z Testing distributed C++ tests 2024-06-26T06:48:00.7788741Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:48:00.7798494Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:48:00.7808419Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:48:00.7809608Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:48:00.7810614Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-06-26T06:48:00.9099915Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:48:00.9101355Z import pkg_resources 2024-06-26T06:48:05.2351275Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:48:05.2353466Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:48:05.2476350Z Found test times from artifacts 2024-06-26T06:48:05.2970407Z Found test times from artifacts 2024-06-26T06:48:05.2986148Z Running 25% of tests based on TD 2024-06-26T06:48:05.2989108Z Running parallel tests on 2 processes 2024-06-26T06:48:05.2989692Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:48:05.2990185Z Serial tests (1): 2024-06-26T06:48:05.2990598Z cpp/FileStoreTest 1/1 2024-06-26T06:48:05.2991043Z Parallel tests (0): 2024-06-26T06:48:05.2991455Z Name: excluded (est. time: 0.0min) 2024-06-26T06:48:05.2991941Z Serial tests (0): 2024-06-26T06:48:05.2992317Z Parallel tests (0): 2024-06-26T06:48:05.2993088Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:48:05.3056575Z Running cpp/FileStoreTest 1/1 ... [2024-06-26 06:48:05.305311] 2024-06-26T06:48:05.3064330Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-df2b75bad467124f.xml', '-x', '--reruns=2'] ... [2024-06-26 06:48:05.305930] 2024-06-26T06:48:09.1292879Z 2024-06-26T06:48:09.1295127Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_2c458c1c7d9c131d_.log 2024-06-26T06:48:09.1296302Z 2024-06-26T06:48:10.0684048Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-06-26T06:48:10.1992430Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:48:10.1993908Z import pkg_resources 2024-06-26T06:48:14.5253294Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:48:14.5255496Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:48:14.5376685Z Found test times from artifacts 2024-06-26T06:48:14.5866973Z Found test times from artifacts 2024-06-26T06:48:14.5882106Z Running 25% of tests based on TD 2024-06-26T06:48:14.5885465Z Running parallel tests on 2 processes 2024-06-26T06:48:14.5886225Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:48:14.5886752Z Serial tests (1): 2024-06-26T06:48:14.5887159Z cpp/HashStoreTest 1/1 2024-06-26T06:48:14.5887580Z Parallel tests (0): 2024-06-26T06:48:14.5888007Z Name: excluded (est. time: 0.0min) 2024-06-26T06:48:14.5888488Z Serial tests (0): 2024-06-26T06:48:14.5888853Z Parallel tests (0): 2024-06-26T06:48:14.5889608Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:48:14.5949932Z Running cpp/HashStoreTest 1/1 ... [2024-06-26 06:48:14.594652] 2024-06-26T06:48:14.5956942Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-a9439d68954ce166.xml', '-x', '--reruns=2'] ... [2024-06-26 06:48:14.595223] 2024-06-26T06:48:18.0179950Z 2024-06-26T06:48:18.0181464Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_5c0d2062834b767e_.log 2024-06-26T06:48:18.0182954Z 2024-06-26T06:48:18.9488154Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-06-26T06:48:19.0777765Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:48:19.0779209Z import pkg_resources 2024-06-26T06:48:23.4416345Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:48:23.4418529Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:48:23.4539044Z Found test times from artifacts 2024-06-26T06:48:23.5035142Z Found test times from artifacts 2024-06-26T06:48:23.5050547Z Running 25% of tests based on TD 2024-06-26T06:48:23.5053626Z Running parallel tests on 2 processes 2024-06-26T06:48:23.5054166Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:48:23.5054698Z Serial tests (1): 2024-06-26T06:48:23.5055110Z cpp/TCPStoreTest 1/1 2024-06-26T06:48:23.5055535Z Parallel tests (0): 2024-06-26T06:48:23.5055966Z Name: excluded (est. time: 0.0min) 2024-06-26T06:48:23.5056446Z Serial tests (0): 2024-06-26T06:48:23.5056823Z Parallel tests (0): 2024-06-26T06:48:23.5057598Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:48:23.5118843Z Running cpp/TCPStoreTest 1/1 ... [2024-06-26 06:48:23.511556] 2024-06-26T06:48:23.5125870Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-79a282a975698dd7.xml', '-x', '--reruns=2'] ... [2024-06-26 06:48:23.512116] 2024-06-26T06:48:39.9120586Z 2024-06-26T06:48:39.9122362Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_d2c3bddc5a57b983_.log 2024-06-26T06:48:39.9123849Z 2024-06-26T06:48:40.8701246Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-06-26T06:48:40.8704832Z ++ command -v mpiexec 2024-06-26T06:48:40.8709142Z + MPIEXEC=/usr/bin/mpiexec 2024-06-26T06:48:40.8709673Z + [[ -n /usr/bin/mpiexec ]] 2024-06-26T06:48:40.8710808Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-06-26T06:48:40.8712549Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-06-26T06:48:40.8714194Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-06-26T06:48:44.0214364Z Test successful 2024-06-26T06:48:44.0266817Z Test successful 2024-06-26T06:48:44.1798603Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-06-26T06:48:44.3110729Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:48:44.3112210Z import pkg_resources 2024-06-26T06:48:48.6206141Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:48:48.6208329Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:48:48.6328332Z Found test times from artifacts 2024-06-26T06:48:48.6820416Z Found test times from artifacts 2024-06-26T06:48:48.6835983Z Running 25% of tests based on TD 2024-06-26T06:48:48.6839140Z Running parallel tests on 2 processes 2024-06-26T06:48:48.6839713Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:48:48.6840234Z Serial tests (1): 2024-06-26T06:48:48.6840672Z cpp/ProcessGroupGlooTest 1/1 2024-06-26T06:48:48.6841160Z Parallel tests (0): 2024-06-26T06:48:48.6841600Z Name: excluded (est. time: 0.0min) 2024-06-26T06:48:48.6842104Z Serial tests (0): 2024-06-26T06:48:48.6842483Z Parallel tests (0): 2024-06-26T06:48:48.6843249Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:48:48.6900653Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-06-26 06:48:48.689709] 2024-06-26T06:48:48.6909212Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-59c44c594d978436.xml', '-x', '--reruns=2'] ... [2024-06-26 06:48:48.690408] 2024-06-26T06:48:57.8755719Z 2024-06-26T06:48:57.8757728Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_965f38b6028e7a84_.log 2024-06-26T06:48:57.8758972Z 2024-06-26T06:48:58.8174139Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-06-26T06:48:58.9471324Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:48:58.9472784Z import pkg_resources 2024-06-26T06:49:03.3065295Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:49:03.3068067Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:49:03.3188439Z Found test times from artifacts 2024-06-26T06:49:03.3683988Z Found test times from artifacts 2024-06-26T06:49:03.3698217Z Running 25% of tests based on TD 2024-06-26T06:49:03.3701598Z Running parallel tests on 2 processes 2024-06-26T06:49:03.3702176Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:49:03.3703240Z Serial tests (1): 2024-06-26T06:49:03.3703693Z cpp/ProcessGroupNCCLTest 1/1 2024-06-26T06:49:03.3704443Z Parallel tests (0): 2024-06-26T06:49:03.3704884Z Name: excluded (est. time: 0.0min) 2024-06-26T06:49:03.3705648Z Serial tests (0): 2024-06-26T06:49:03.3706053Z Parallel tests (0): 2024-06-26T06:49:03.3707094Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:49:03.3765806Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-06-26 06:49:03.376220] 2024-06-26T06:49:03.3772732Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e2c66f2f721171af.xml', '-x', '--reruns=2'] ... [2024-06-26 06:49:03.376790] 2024-06-26T06:49:28.2906175Z 2024-06-26T06:49:28.2908060Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_cd89e102f6f41037_.log 2024-06-26T06:49:29.2748489Z 2024-06-26T06:49:29.2749533Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-06-26T06:49:29.4043147Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:49:29.4044627Z import pkg_resources 2024-06-26T06:49:33.8000488Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:49:33.8002671Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:49:33.8123486Z Found test times from artifacts 2024-06-26T06:49:33.8620633Z Found test times from artifacts 2024-06-26T06:49:33.8636928Z Running 25% of tests based on TD 2024-06-26T06:49:33.8640494Z Running parallel tests on 2 processes 2024-06-26T06:49:33.8641064Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:49:33.8641577Z Serial tests (1): 2024-06-26T06:49:33.8642014Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-06-26T06:49:33.8642903Z Parallel tests (0): 2024-06-26T06:49:33.8643340Z Name: excluded (est. time: 0.0min) 2024-06-26T06:49:33.8643805Z Serial tests (0): 2024-06-26T06:49:33.8644205Z Parallel tests (0): 2024-06-26T06:49:33.8703962Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:49:33.8705117Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-06-26 06:49:33.869925] 2024-06-26T06:49:33.8710886Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-555b47c455094751.xml', '-x', '--reruns=2'] ... [2024-06-26 06:49:33.870540] 2024-06-26T06:49:46.1630154Z 2024-06-26T06:49:46.1632188Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_c1af13a103fdfea8_.log 2024-06-26T06:49:46.1633555Z 2024-06-26T06:49:47.0949209Z + [[ 1 == 1 ]] 2024-06-26T06:49:47.0949689Z + test_rpc 2024-06-26T06:49:47.0950227Z + echo 'Testing RPC C++ tests' 2024-06-26T06:49:47.0950692Z Testing RPC C++ tests 2024-06-26T06:49:47.0956309Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:49:47.0968359Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:49:47.0980105Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:49:47.0981704Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-06-26T06:49:47.2267797Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:49:47.2269297Z import pkg_resources 2024-06-26T06:49:51.5853245Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:49:51.5855455Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:49:51.5977442Z Found test times from artifacts 2024-06-26T06:49:51.6474469Z Found test times from artifacts 2024-06-26T06:49:51.6489995Z Running 25% of tests based on TD 2024-06-26T06:49:51.6493305Z Running parallel tests on 2 processes 2024-06-26T06:49:51.6494238Z Name: tests to run (est. time: 0.0min) 2024-06-26T06:49:51.6494751Z Serial tests (1): 2024-06-26T06:49:51.6495181Z cpp/test_cpp_rpc 1/1 2024-06-26T06:49:51.6495619Z Parallel tests (0): 2024-06-26T06:49:51.6496051Z Name: excluded (est. time: 0.0min) 2024-06-26T06:49:51.6496514Z Serial tests (0): 2024-06-26T06:49:51.6496908Z Parallel tests (0): 2024-06-26T06:49:51.6497733Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:49:51.6558250Z Running cpp/test_cpp_rpc 1/1 ... [2024-06-26 06:49:51.655466] 2024-06-26T06:49:51.6565381Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-07478d07e55b956f.xml', '-x', '--reruns=2'] ... [2024-06-26 06:49:51.656036] 2024-06-26T06:49:57.6844889Z 2024-06-26T06:49:57.6846847Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_101efb65b16c8836_.log 2024-06-26T06:49:57.6848613Z 2024-06-26T06:49:58.6376821Z + cleanup_workspace 2024-06-26T06:49:58.6378029Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T06:49:58.6379442Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T06:49:58.6380609Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T06:49:58.6381823Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T06:49:58.6383717Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T06:49:58.6384854Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T06:49:58.6385715Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T06:49:59.1754490Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T06:49:59.1755475Z cat test/**/*_toprint.log || true 2024-06-26T06:49:59.1763680Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.1764290Z env: 2024-06-26T06:49:59.1764626Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.1765181Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.1766101Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.1766929Z ##[endgroup] 2024-06-26T06:49:59.1838943Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T06:49:59.1877818Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T06:49:59.1878421Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T06:49:59.1886143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.1886769Z env: 2024-06-26T06:49:59.1887110Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.1887648Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.1888592Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.1889445Z MONITOR_SCRIPT_PID: 102367 2024-06-26T06:49:59.1889869Z ##[endgroup] 2024-06-26T06:49:59.2133919Z Prepare all required actions 2024-06-26T06:49:59.2134504Z Getting action download info 2024-06-26T06:49:59.3830542Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T06:49:59.5641848Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T06:49:59.5642415Z with: 2024-06-26T06:49:59.5643015Z file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431 2024-06-26T06:49:59.5643807Z s3-bucket: gha-artifacts 2024-06-26T06:49:59.5644198Z env: 2024-06-26T06:49:59.5644532Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.5645079Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.5645995Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.5646812Z ##[endgroup] 2024-06-26T06:49:59.5681350Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T06:49:59.5682182Z # Remove any previous test jsons if they exist 2024-06-26T06:49:59.5682792Z rm -f test-jsons-*.zip 2024-06-26T06:49:59.5683423Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T06:49:59.5691506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.5692107Z env: 2024-06-26T06:49:59.5692443Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.5692995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.5693951Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.5695020Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431 2024-06-26T06:49:59.5695758Z ##[endgroup] 2024-06-26T06:49:59.5940000Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T06:49:59.5975459Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T06:49:59.5976351Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T06:49:59.5983225Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T06:49:59.5986035Z adding: test/test-reports/td_exclusions-d1898c1d20de51924eb8.json (deflated 86%) 2024-06-26T06:49:59.5987265Z adding: test/test-reports/td_exclusions-927660d5b1f032087208.json (deflated 15%) 2024-06-26T06:49:59.5988468Z adding: test/test-reports/td_exclusions-4652aa1e768a3da2ec43.json (deflated 14%) 2024-06-26T06:49:59.5989953Z adding: test/test-reports/td_exclusions-bf8d8fda7bed5002e589.json (deflated 14%) 2024-06-26T06:49:59.5991157Z adding: test/test-reports/td_exclusions-3613cc192f2191ab04bb.json (deflated 12%) 2024-06-26T06:49:59.5992355Z adding: test/test-reports/td_exclusions-790a0a707b9a32471fd5.json (deflated 12%) 2024-06-26T06:49:59.5993553Z adding: test/test-reports/td_exclusions-7537a4f9de4bd108c5da.json (deflated 11%) 2024-06-26T06:49:59.5994722Z adding: test/test-reports/td_exclusions-3298d39717a42c793fda.json (deflated 18%) 2024-06-26T06:49:59.5995714Z adding: test/.pytorch-slow-tests.json (deflated 77%) 2024-06-26T06:49:59.5997931Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-06-26T06:49:59.6033381Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T06:49:59.6034191Z # Remove any previous test reports if they exist 2024-06-26T06:49:59.6034838Z rm -f test-reports-*.zip 2024-06-26T06:49:59.6035569Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T06:49:59.6043475Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.6044097Z env: 2024-06-26T06:49:59.6044436Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.6044968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.6045906Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.6046993Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431 2024-06-26T06:49:59.6047735Z ##[endgroup] 2024-06-26T06:49:59.6271704Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-92a6baaabc4d3615.xml (deflated 89%) 2024-06-26T06:49:59.6276820Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-cbc8dddfa2ad6132.xml (deflated 37%) 2024-06-26T06:49:59.6281325Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-99785055e5e471bf.xml (deflated 50%) 2024-06-26T06:49:59.6283869Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-6a7212fcc675a14e.xml (deflated 81%) 2024-06-26T06:49:59.6286386Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-859bffb027ad4b90.xml (deflated 49%) 2024-06-26T06:49:59.6288794Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-83fb3555e7476732.xml (deflated 53%) 2024-06-26T06:49:59.6291002Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-0c9a70bd2a154944.xml (deflated 72%) 2024-06-26T06:49:59.6293113Z adding: test/test-reports/python-pytest/distributed._tools.test_fsdp2_mem_tracker/distributed._tools.test_fsdp2_mem_tracker-002035022d298cd0.xml (deflated 56%) 2024-06-26T06:49:59.6295392Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-e4414b8751233113.xml (deflated 57%) 2024-06-26T06:49:59.6297841Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_overlap/distributed._composable.fsdp.test_fully_shard_overlap-06a13c730cff6a36.xml (deflated 53%) 2024-06-26T06:49:59.6300263Z adding: test/test-reports/python-pytest/distributed._tensor.test_redistribute/distributed._tensor.test_redistribute-efda0c8722131d6a.xml (deflated 79%) 2024-06-26T06:49:59.6302110Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-df2b75bad467124f.xml (deflated 60%) 2024-06-26T06:49:59.6303930Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-a9439d68954ce166.xml (deflated 60%) 2024-06-26T06:49:59.6305448Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-79a282a975698dd7.xml (deflated 69%) 2024-06-26T06:49:59.6307174Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-59c44c594d978436.xml (deflated 78%) 2024-06-26T06:49:59.6308661Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-e2c66f2f721171af.xml (deflated 74%) 2024-06-26T06:49:59.6310174Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-555b47c455094751.xml (deflated 76%) 2024-06-26T06:49:59.6311684Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-07478d07e55b956f.xml (deflated 64%) 2024-06-26T06:49:59.6342129Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T06:49:59.6343300Z # Remove any previous usage logs if they exist 2024-06-26T06:49:59.6343920Z rm -f logs-*.zip 2024-06-26T06:49:59.6344747Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T06:49:59.6345728Z # so check to see if the file exists first 2024-06-26T06:49:59.6346340Z if [ -f 'usage_log.txt' ]; then 2024-06-26T06:49:59.6346990Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T06:49:59.6347588Z fi 2024-06-26T06:49:59.6348008Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T06:49:59.6348716Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T06:49:59.6349329Z fi 2024-06-26T06:49:59.6356753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.6357371Z env: 2024-06-26T06:49:59.6357705Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.6358388Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.6359347Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.6360427Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431 2024-06-26T06:49:59.6361162Z ##[endgroup] 2024-06-26T06:49:59.6408771Z adding: usage_log.txt (deflated 92%) 2024-06-26T06:49:59.6633797Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_8eddf3a0a81a18be_.log (deflated 87%) 2024-06-26T06:49:59.6637123Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_2b66a4f31e1c4e43_.log (deflated 54%) 2024-06-26T06:49:59.6640303Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_b49d14f1af4f1a97_.log (deflated 59%) 2024-06-26T06:49:59.6819491Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_db6b0198bf52ebbb_.log (deflated 95%) 2024-06-26T06:49:59.6821457Z adding: test/test-reports/distributed._spmd.test_graph_utils_1.1_395a544e6e3e583f_.log (stored 0%) 2024-06-26T06:49:59.6825019Z adding: test/test-reports/distributed._spmd.test_transformation_1.1_0f4beec38f2a0633_.log (stored 0%) 2024-06-26T06:49:59.6827499Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_cfb84c3cafef9bd9_.log (deflated 50%) 2024-06-26T06:49:59.6831195Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_04c5ba18e9f94897_.log (deflated 91%) 2024-06-26T06:49:59.6834570Z adding: test/test-reports/distributed._tensor.test_init_1.1_1a7c1ce451a95156_.log (deflated 69%) 2024-06-26T06:49:59.6836138Z adding: test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_011af83ca1fb677d_.log (deflated 65%) 2024-06-26T06:49:59.6837925Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_c119a86ac53e38ae_.log (deflated 61%) 2024-06-26T06:49:59.6839607Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_1d7c1c4300afff9b_.log (deflated 63%) 2024-06-26T06:49:59.6841218Z adding: test/test-reports/distributed._tensor.test_redistribute_1.1_11bcea3292b11d31_.log (deflated 93%) 2024-06-26T06:49:59.6842559Z adding: test/test-reports/cpp.FileStoreTest_1.1_2c458c1c7d9c131d_.log (deflated 71%) 2024-06-26T06:49:59.6843791Z adding: test/test-reports/cpp.HashStoreTest_1.1_5c0d2062834b767e_.log (deflated 71%) 2024-06-26T06:49:59.6845023Z adding: test/test-reports/cpp.TCPStoreTest_1.1_d2c3bddc5a57b983_.log (deflated 79%) 2024-06-26T06:49:59.6846574Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_965f38b6028e7a84_.log (deflated 85%) 2024-06-26T06:49:59.6847892Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_cd89e102f6f41037_.log (deflated 82%) 2024-06-26T06:49:59.6849299Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_c1af13a103fdfea8_.log (deflated 78%) 2024-06-26T06:49:59.6850625Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_101efb65b16c8836_.log (deflated 79%) 2024-06-26T06:49:59.6880895Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T06:49:59.6881792Z # Remove any previous debugging artifacts if they exist 2024-06-26T06:49:59.6882474Z rm -f debug-*.zip 2024-06-26T06:49:59.6882933Z if [ -d 'test/debug' ]; then 2024-06-26T06:49:59.6883529Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T06:49:59.6884115Z fi 2024-06-26T06:49:59.6891432Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:59.6892031Z env: 2024-06-26T06:49:59.6892363Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.6892913Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.6893834Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.6894912Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431 2024-06-26T06:49:59.6895653Z ##[endgroup] 2024-06-26T06:49:59.6993810Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:49:59.6994341Z with: 2024-06-26T06:49:59.6994831Z s3-bucket: gha-artifacts 2024-06-26T06:49:59.6995379Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:59.6995935Z retention-days: 14 2024-06-26T06:49:59.6996346Z if-no-files-found: warn 2024-06-26T06:49:59.6996786Z path: test-jsons-*.zip 2024-06-26T06:49:59.6997183Z name: artifact 2024-06-26T06:49:59.6997549Z region: us-east-1 2024-06-26T06:49:59.6997922Z env: 2024-06-26T06:49:59.6998224Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:59.6998774Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:59.6999708Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:49:59.7000514Z ##[endgroup] 2024-06-26T06:50:00.1454258Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:50:00.1455257Z With the provided path, there will be 1 file uploaded 2024-06-26T06:50:00.1456040Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:00.1503538Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:00.2939510Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:00.3109260Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:50:00.3109794Z with: 2024-06-26T06:50:00.3110132Z s3-bucket: gha-artifacts 2024-06-26T06:50:00.3110658Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:00.3111238Z retention-days: 14 2024-06-26T06:50:00.3111645Z if-no-files-found: error 2024-06-26T06:50:00.3112089Z path: test-reports-*.zip 2024-06-26T06:50:00.3112498Z name: artifact 2024-06-26T06:50:00.3112870Z region: us-east-1 2024-06-26T06:50:00.3113231Z env: 2024-06-26T06:50:00.3113542Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:00.3114091Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:00.3115033Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:00.3115855Z ##[endgroup] 2024-06-26T06:50:00.7605547Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:50:00.7606522Z With the provided path, there will be 1 file uploaded 2024-06-26T06:50:00.7607308Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:00.7655357Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:00.9124191Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:00.9292891Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:50:00.9293426Z with: 2024-06-26T06:50:00.9293762Z s3-bucket: gha-artifacts 2024-06-26T06:50:00.9294284Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:00.9294849Z retention-days: 14 2024-06-26T06:50:00.9295261Z if-no-files-found: ignore 2024-06-26T06:50:00.9295698Z path: logs-*.zip 2024-06-26T06:50:00.9296054Z name: artifact 2024-06-26T06:50:00.9296419Z region: us-east-1 2024-06-26T06:50:00.9296802Z env: 2024-06-26T06:50:00.9297115Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:00.9297666Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:00.9298610Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:00.9299427Z ##[endgroup] 2024-06-26T06:50:01.3806644Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:50:01.3807621Z With the provided path, there will be 1 file uploaded 2024-06-26T06:50:01.3808401Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:01.3856158Z Starting upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:01.5020047Z Finished upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_26688546431.zip 2024-06-26T06:50:01.5213899Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:50:01.5214448Z with: 2024-06-26T06:50:01.5214770Z s3-bucket: gha-artifacts 2024-06-26T06:50:01.5215462Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:50:01.5216071Z retention-days: 14 2024-06-26T06:50:01.5216459Z if-no-files-found: ignore 2024-06-26T06:50:01.5216891Z path: debug-*.zip 2024-06-26T06:50:01.5217269Z name: artifact 2024-06-26T06:50:01.5217617Z region: us-east-1 2024-06-26T06:50:01.5217983Z env: 2024-06-26T06:50:01.5218317Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:01.5218847Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:01.5219877Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:01.5220705Z ##[endgroup] 2024-06-26T06:50:01.9568624Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T06:50:01.9735760Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T06:50:01.9736388Z # shellcheck disable=SC2156 2024-06-26T06:50:01.9737390Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T06:50:01.9746651Z shell: /usr/bin/bash -e {0} 2024-06-26T06:50:01.9747089Z env: 2024-06-26T06:50:01.9747421Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:01.9747953Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:01.9748901Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:01.9749736Z ##[endgroup] 2024-06-26T06:50:02.2753134Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T06:50:02.2753851Z with: 2024-06-26T06:50:02.2754161Z env: 2024-06-26T06:50:02.2754489Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:02.2755029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:02.2755977Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:02.2756802Z ##[endgroup] 2024-06-26T06:50:02.2782640Z ##[group]Run set -eou pipefail 2024-06-26T06:50:02.2783153Z set -eou pipefail 2024-06-26T06:50:02.2783583Z  2024-06-26T06:50:02.2784215Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T06:50:02.2784998Z for _ in $(seq 1440); do 2024-06-26T06:50:02.2785576Z  # Break if no ssh session exists anymore 2024-06-26T06:50:02.2786193Z  if [ "$(who)" = "" ]; then 2024-06-26T06:50:02.2786676Z  break 2024-06-26T06:50:02.2787360Z  fi 2024-06-26T06:50:02.2787713Z  echo "." 2024-06-26T06:50:02.2788114Z  sleep 5 2024-06-26T06:50:02.2788496Z done 2024-06-26T06:50:02.2796518Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:50:02.2797139Z env: 2024-06-26T06:50:02.2797476Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:02.2798013Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:02.2798955Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:02.2799790Z ##[endgroup] 2024-06-26T06:50:02.2826680Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T06:50:02.2877967Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:50:02.2878930Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:50:02.2879667Z # shellcheck disable=SC2046 2024-06-26T06:50:02.2880239Z docker stop $(docker ps -q) || true 2024-06-26T06:50:02.2880826Z # Prune all of the docker images 2024-06-26T06:50:02.2881381Z docker system prune -af 2024-06-26T06:50:02.2889070Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:50:02.2889673Z env: 2024-06-26T06:50:02.2890008Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:50:02.2890564Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:50:02.2891491Z DOCKER_CONTAINER_ID: 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:02.2892324Z ##[endgroup] 2024-06-26T06:50:02.6426440Z 15d2f4a9275f 2024-06-26T06:50:03.2289393Z Deleted Containers: 2024-06-26T06:50:03.2290124Z 15d2f4a9275fe5edccee85c3bbbba27a4ad3cd172b9d14a97b95d27396776dd7 2024-06-26T06:50:03.2290706Z 2024-06-26T06:50:09.9512837Z Deleted Images: 2024-06-26T06:50:09.9515715Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:50:09.9519407Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:493391c3b293fbc39c8ca4208a5b70ab17031329cb9c855a641ab5a208c52fe6 2024-06-26T06:50:09.9521320Z deleted: sha256:1fde65d68e1ba467b7c461d097349da1965328db5e6e5252ccb25702fcc3b83e 2024-06-26T06:50:09.9522424Z deleted: sha256:d7f4fd7abbe071a8cbdda2bae83a151d7bcde19abdef895ea3019cdeadd13a18 2024-06-26T06:50:09.9523503Z deleted: sha256:ee085d46602902c38ef79d90a27d337dae20947f1eeb9c4fccabd1a92e01e58e 2024-06-26T06:50:09.9524562Z deleted: sha256:e860cd33597e4e0f5736fd4d6e2f4702fa16a9940209ad8e214ca7081f44a287 2024-06-26T06:50:09.9525623Z deleted: sha256:a18d95c977fe0d23a7007fdc3ee2379aa8a8c2d42e59e8ef95c3e677d2578fcd 2024-06-26T06:50:09.9526677Z deleted: sha256:52e65c84df1c54be8f442eee573bca8c158f9fc906b5fbee0ff6953553aea0e4 2024-06-26T06:50:09.9527756Z deleted: sha256:bb7adddb154f86f311acf257b7a21b39fa5702dda66249163efd1ae6729d3782 2024-06-26T06:50:09.9529098Z deleted: sha256:1997f0e776385e3c40db4b42b13e2e9006fb6204d0d1dcf32f29db95c117372a 2024-06-26T06:50:09.9530192Z deleted: sha256:cad19b1260ec60216e3d0b105b8f813b992973a7c792a45774531d346384edc1 2024-06-26T06:50:09.9531223Z deleted: sha256:ecc44fe33e9088e5eb858752a97ca6e2d729a64cd42d6a5225b567f8da1ec73f 2024-06-26T06:50:09.9532306Z deleted: sha256:f352c3ff12d96e8bd1de0ee0fddddf2df4aaadbad5433589930269739439c345 2024-06-26T06:50:09.9533397Z deleted: sha256:e6f2acebe916afc5f24b61dd806cd8194326c8dd940a061c8b6acd11a1af4683 2024-06-26T06:50:09.9534441Z deleted: sha256:c94a6342cc6b933354589534bdead86649bbc1464c794cc9d21577fe82437b3f 2024-06-26T06:50:09.9535499Z deleted: sha256:73c84e0e40a121ef226ee59ad4e41c40f6f8e3dd3aa6e40b580b2342181ff842 2024-06-26T06:50:09.9536567Z deleted: sha256:5f650bdb1edb4afa0f22a8c2de8706ea07e360c55298d0bac711f5f4dd750406 2024-06-26T06:50:09.9537625Z deleted: sha256:e3c78b112f621be151f2591f8a82ab325ffedd19be99e3b478c3017e051b5355 2024-06-26T06:50:09.9539075Z deleted: sha256:e058ad0c4f09075c613e2fb7abd0ef0e7465733d555b7778902ed13846dfacb6 2024-06-26T06:50:09.9540492Z deleted: sha256:2b91783cec90ca4caa17a1a03b734c733623e7f8f185334d5bbb21fe9a0f1e5d 2024-06-26T06:50:09.9541561Z deleted: sha256:014aca11d6f33d59cda58c68a53f4a22dadeedb3f6b487fa66f556fcec903607 2024-06-26T06:50:09.9543174Z deleted: sha256:32b4d98aa83b992ac9dd109c618da5a02af41078f6d5bd2a174b69f208fedb0c 2024-06-26T06:50:09.9544263Z deleted: sha256:f4dad79cdbcd5b49f4b3526d3f3c0dae331ddad5ee9be01b21d289ce2be060c0 2024-06-26T06:50:09.9545356Z deleted: sha256:755dbc164caaa04e7f3ac761fcfa41914cb61bbefd288772a7fae92e6429c790 2024-06-26T06:50:09.9546405Z deleted: sha256:3bb6a2f1711ddd563f93bd2b39fb20a667aad822781c1586c9d04a049ffff691 2024-06-26T06:50:09.9547465Z deleted: sha256:25731f18356bbce9f9cf003c08edb416482d8c85becb79fd1af3b83806d147b0 2024-06-26T06:50:09.9548523Z deleted: sha256:6f3c5cbd50ad5b397dcac877079fae290058a3af2bb7c5390c202f221fe1d8f2 2024-06-26T06:50:09.9549552Z deleted: sha256:ea584ac8474876e97af40f7148363e9a6df9e84b52540b61d3488f76fe3fda19 2024-06-26T06:50:09.9550622Z deleted: sha256:c99897453df1a53f8d8caf80177dfabc306d7ad968db5e62b0084b2fcc38ff9b 2024-06-26T06:50:09.9551673Z deleted: sha256:0fa44105790cb410839a6f2cc4410c3be63657d803ef603c9ef7a4b355d5f380 2024-06-26T06:50:09.9553889Z deleted: sha256:2fd5ab6ef7d9cb567b70dddfa5a1a5e65e324446bb14f6feba5599f4e730b35b 2024-06-26T06:50:09.9555195Z deleted: sha256:b99692dfc620bf7adfe66568b6fe1f742be148802aa2d7c6028593bef0210571 2024-06-26T06:50:09.9556262Z deleted: sha256:21daffc134560cc20e34bee6ed4684f5c0de46241501e2a7a89cb7bf546d4227 2024-06-26T06:50:09.9557301Z deleted: sha256:819249fb48326412a787749ddc95957655df6636f2e0f1c5265bf6881c7111e9 2024-06-26T06:50:09.9558309Z deleted: sha256:610d96248b1aa37c8d0b05fdf39c38fa1d491f505e454525de8f8ac3b0bd48f4 2024-06-26T06:50:09.9559366Z deleted: sha256:22f47f538c346eb9b1142a65fdc01f85cd5e10621a8a28cd8aae7e205880bada 2024-06-26T06:50:09.9560430Z deleted: sha256:09c6cd1d8ea04ed3b74420884bf62eaed12a455f041633ebec3a2cf41468e057 2024-06-26T06:50:09.9561490Z deleted: sha256:2f81f71a0bfa140425b1f34efbb9e8a09ce133c20bce173876ae51ec55c53767 2024-06-26T06:50:09.9562548Z deleted: sha256:7083f25055a8577c549ee03196e895f72ec1e26bb509cf4f531e68a9458ec9ff 2024-06-26T06:50:09.9563584Z deleted: sha256:2d353f3abd6ba69b7052291fab5b672c6a31571f9ca14eb579f1324928be9a17 2024-06-26T06:50:09.9565144Z deleted: sha256:f8cb1af08a381526d68d11ccadf89749f855c9f643bced1fa51c146109d894fb 2024-06-26T06:50:09.9566682Z deleted: sha256:538faaa902f68d805ad2a2dc8511f8fea33499e28ed8dd54a2e885b6069b94ed 2024-06-26T06:50:09.9567762Z deleted: sha256:6c0cabeb2da5f02c2781f2f0ca3706fdb328da082a3decdbd8f82ae596f513f3 2024-06-26T06:50:09.9568834Z deleted: sha256:5722d912cb634e82f4826ea4170754898a77efee6df3b6ea282ba78b4aaa43d2 2024-06-26T06:50:09.9569864Z deleted: sha256:dc0beaa37049a8d70d2e4da52db402c3876391d86720df30a9b686ae6ac5cb24 2024-06-26T06:50:09.9570912Z deleted: sha256:a2793888ee929a46917ad906e46fa352b0b00c68298d9c47b995cdd3715f0b44 2024-06-26T06:50:09.9572157Z deleted: sha256:fccc9e0f8fd9de6f71e5b000f1d79435e896c89cd8554c718604b56c993d93ed 2024-06-26T06:50:09.9573234Z deleted: sha256:99df2ce205527dab671443417aa5c24070144ad78f7597f7b01a2fbdc75db4a0 2024-06-26T06:50:09.9574274Z deleted: sha256:ab3da06e7304132443910308818f60abaee5e6aaea7aadc74ff905e3071786a6 2024-06-26T06:50:09.9575332Z deleted: sha256:c43d30eed3c50d7f00e7d83ef78a94dcdf07c763c37d83f9185e72377d1a5f20 2024-06-26T06:50:09.9576378Z deleted: sha256:8f2d8265fe3b438415439e09c19ba6a68b4150eaa562224d7e8ac2710dc2eb14 2024-06-26T06:50:09.9577389Z deleted: sha256:28f2bc432836b7a4424f890747021c36c0b33371c62b6fdb714244d6b90203aa 2024-06-26T06:50:09.9578422Z deleted: sha256:f6286286a1dbe990ec2f0b147e01827e45b747cdb10499b6102daa2b25922727 2024-06-26T06:50:09.9579482Z deleted: sha256:c6e97d8ca374fb6c9ca255b80d6c36af85de5fd66e448ea23491ad9958f19b66 2024-06-26T06:50:09.9580548Z deleted: sha256:b88b1ab78ee5abbcbfd6500e00629d125d67a3037ce62eddb8e0866ffef82fe9 2024-06-26T06:50:09.9581612Z deleted: sha256:1512ffb3610f68874de48684966014158e1bdd95d80f21f97570bd10e88b4ed4 2024-06-26T06:50:09.9583078Z deleted: sha256:fd6f5133b44e40b8da2ecb4666c25328e71d7c29daf9247f740daba07621adc2 2024-06-26T06:50:09.9584119Z deleted: sha256:f9f206b62ad20270a766d66358f0a8cda72f3819c9805b5fe5e929be08487da6 2024-06-26T06:50:09.9585156Z deleted: sha256:f73b05159be211077e8f3d9ff623e66c1c4735d4450aeb21229f951235dc0adb 2024-06-26T06:50:09.9586202Z deleted: sha256:ef71994fba318c56792ce905d971dc876347c60961aa4fd4a2bdf22f026df5ba 2024-06-26T06:50:09.9587252Z deleted: sha256:2c70714aac7394ad9e1e42a55b728f30f38fbd46e545b118bf0d625f4b1eec4e 2024-06-26T06:50:09.9588294Z deleted: sha256:49adb4771e439b38cd6de3c8ef02299fc5672fdfe39ed6a08ecc41893a00d16f 2024-06-26T06:50:09.9589360Z deleted: sha256:fa77ab1294ff3ad236f4a809e697cb4b9b150b11fc7641ea3c508198e0059582 2024-06-26T06:50:09.9590404Z deleted: sha256:06332c1fdd9151f70f43605c3bd5a2967db0f7d03398d97147e645c6ad1034b6 2024-06-26T06:50:09.9591450Z deleted: sha256:c217de6cbf77b426bac257f8f17b0ff3ca3a514a5511deaa7bdfc6f42d405375 2024-06-26T06:50:09.9592522Z deleted: sha256:1248854931e90cf6124b5f6d68fe5bcb43a328f6c246cad949ef775809a2dd40 2024-06-26T06:50:09.9593580Z deleted: sha256:903fa51ff1654718fa673df7ccde5b99f43fc40cd5d5c541f674554c68b9dd00 2024-06-26T06:50:09.9594607Z deleted: sha256:80c567c8b1b790b0859417382b69fb69bd01a15552aadf1ab1aba69882cba1d4 2024-06-26T06:50:09.9595668Z deleted: sha256:8fc1dd27a3b4d2bbf11127923b7bd5063a9faacb77ed389a2eb18bc7d9732abc 2024-06-26T06:50:09.9596740Z deleted: sha256:08147e1d91de147ab86d7cab3a87eac310ab1d06d857b2506890fd3ea12fbf6c 2024-06-26T06:50:09.9597790Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-06-26T06:50:09.9598834Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-06-26T06:50:09.9599884Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-06-26T06:50:09.9600926Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-06-26T06:50:09.9601977Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-06-26T06:50:09.9603061Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-06-26T06:50:09.9604124Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-06-26T06:50:09.9605149Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-06-26T06:50:09.9606203Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-06-26T06:50:09.9607246Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-06-26T06:50:09.9608282Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T06:50:09.9608914Z 2024-06-26T06:50:09.9609099Z Total reclaimed space: 25.98GB 2024-06-26T06:50:09.9694250Z Post job cleanup. 2024-06-26T06:50:09.9759983Z Post job cleanup. 2024-06-26T06:50:10.0970258Z [command]/usr/bin/git version 2024-06-26T06:50:10.1026892Z git version 2.40.1 2024-06-26T06:50:10.1092468Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c1d437ea-fa9f-460f-bb3b-bfe8aa2b51e0' before making global git config changes 2024-06-26T06:50:10.1094622Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:50:10.1100590Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:50:10.1144574Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:50:10.1182608Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:50:10.1504146Z Entering 'android/libs/fbjni' 2024-06-26T06:50:10.1555820Z Entering 'third_party/FP16' 2024-06-26T06:50:10.1607685Z Entering 'third_party/FXdiv' 2024-06-26T06:50:10.1659611Z Entering 'third_party/NNPACK' 2024-06-26T06:50:10.1713911Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:50:10.1765870Z Entering 'third_party/XNNPACK' 2024-06-26T06:50:10.1837400Z Entering 'third_party/benchmark' 2024-06-26T06:50:10.1890282Z Entering 'third_party/cpp-httplib' 2024-06-26T06:50:10.1941198Z Entering 'third_party/cpuinfo' 2024-06-26T06:50:10.1993152Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:50:10.2045565Z Entering 'third_party/cutlass' 2024-06-26T06:50:10.2106919Z Entering 'third_party/eigen' 2024-06-26T06:50:10.2160157Z Entering 'third_party/fbgemm' 2024-06-26T06:50:10.2212969Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:50:10.2263320Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:50:10.2316092Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:50:10.2376017Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:50:10.2427613Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:50:10.2479557Z Entering 'third_party/flatbuffers' 2024-06-26T06:50:10.2534969Z Entering 'third_party/fmt' 2024-06-26T06:50:10.2587692Z Entering 'third_party/foxi' 2024-06-26T06:50:10.2639800Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:50:10.2691952Z Entering 'third_party/gloo' 2024-06-26T06:50:10.2743942Z Entering 'third_party/googletest' 2024-06-26T06:50:10.2797335Z Entering 'third_party/ideep' 2024-06-26T06:50:10.2849726Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:50:10.2909699Z Entering 'third_party/ittapi' 2024-06-26T06:50:10.2962201Z Entering 'third_party/kineto' 2024-06-26T06:50:10.3015860Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:50:10.3068373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:50:10.3123842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:50:10.3178539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:50:10.3231898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:50:10.3283775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:50:10.3339067Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:50:10.3392456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:50:10.3445401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:50:10.3499556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:50:10.3554948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:50:10.3608405Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:50:10.3663562Z Entering 'third_party/mimalloc' 2024-06-26T06:50:10.3718280Z Entering 'third_party/nccl/nccl' 2024-06-26T06:50:10.3772755Z Entering 'third_party/nlohmann' 2024-06-26T06:50:10.3828740Z Entering 'third_party/onnx' 2024-06-26T06:50:10.3899169Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:50:10.3953228Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:50:10.4008021Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:50:10.4061641Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:50:10.4113043Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:50:10.4163984Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:50:10.4215843Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:50:10.4269905Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:50:10.4321139Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:50:10.4371381Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:50:10.4422112Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:50:10.4476240Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:50:10.4529302Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:50:10.4605644Z Entering 'third_party/pocketfft' 2024-06-26T06:50:10.4658974Z Entering 'third_party/protobuf' 2024-06-26T06:50:10.4714121Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:50:10.4765051Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:50:10.4817153Z Entering 'third_party/psimd' 2024-06-26T06:50:10.4869717Z Entering 'third_party/pthreadpool' 2024-06-26T06:50:10.4922205Z Entering 'third_party/pybind11' 2024-06-26T06:50:10.4973715Z Entering 'third_party/python-peachpy' 2024-06-26T06:50:10.5026139Z Entering 'third_party/sleef' 2024-06-26T06:50:10.5077583Z Entering 'third_party/tensorpipe' 2024-06-26T06:50:10.5128598Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:50:10.5180360Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:50:10.5231178Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:50:10.5281943Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:50:10.5332439Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:50:10.5406311Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:50:10.5433430Z http.https://github.com/.extraheader 2024-06-26T06:50:10.5445765Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T06:50:10.5485161Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:50:10.5795089Z Entering 'android/libs/fbjni' 2024-06-26T06:50:10.5827971Z http.https://github.com/.extraheader 2024-06-26T06:50:10.5862421Z Entering 'third_party/FP16' 2024-06-26T06:50:10.5896425Z http.https://github.com/.extraheader 2024-06-26T06:50:10.5931205Z Entering 'third_party/FXdiv' 2024-06-26T06:50:10.5964967Z http.https://github.com/.extraheader 2024-06-26T06:50:10.5998694Z Entering 'third_party/NNPACK' 2024-06-26T06:50:10.6033731Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6068631Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:50:10.6101301Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6137080Z Entering 'third_party/XNNPACK' 2024-06-26T06:50:10.6171151Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6224076Z Entering 'third_party/benchmark' 2024-06-26T06:50:10.6256601Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6291925Z Entering 'third_party/cpp-httplib' 2024-06-26T06:50:10.6325917Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6360098Z Entering 'third_party/cpuinfo' 2024-06-26T06:50:10.6393850Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6429511Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:50:10.6462592Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6496840Z Entering 'third_party/cutlass' 2024-06-26T06:50:10.6530552Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6574702Z Entering 'third_party/eigen' 2024-06-26T06:50:10.6608877Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6645390Z Entering 'third_party/fbgemm' 2024-06-26T06:50:10.6678584Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6712953Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:50:10.6746885Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6781177Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:50:10.6815388Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6850079Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:50:10.6883027Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6923393Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:50:10.6956172Z http.https://github.com/.extraheader 2024-06-26T06:50:10.6990384Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:50:10.7023666Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7059419Z Entering 'third_party/flatbuffers' 2024-06-26T06:50:10.7093564Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7131485Z Entering 'third_party/fmt' 2024-06-26T06:50:10.7164630Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7199224Z Entering 'third_party/foxi' 2024-06-26T06:50:10.7232078Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7266184Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:50:10.7299761Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7334966Z Entering 'third_party/gloo' 2024-06-26T06:50:10.7368624Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7403207Z Entering 'third_party/googletest' 2024-06-26T06:50:10.7438202Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7472803Z Entering 'third_party/ideep' 2024-06-26T06:50:10.7505869Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7538453Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:50:10.7572645Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7614640Z Entering 'third_party/ittapi' 2024-06-26T06:50:10.7647953Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7682342Z Entering 'third_party/kineto' 2024-06-26T06:50:10.7715745Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7750348Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:50:10.7782606Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7816739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:50:10.7849848Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7886489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:50:10.7919279Z http.https://github.com/.extraheader 2024-06-26T06:50:10.7954667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:50:10.7986464Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8021745Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:50:10.8054260Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8089907Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:50:10.8122912Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8160344Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:50:10.8192865Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8227291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:50:10.8258930Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8295318Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:50:10.8327656Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8363636Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:50:10.8396701Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8433407Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:50:10.8466111Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8500711Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:50:10.8534104Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8569412Z Entering 'third_party/mimalloc' 2024-06-26T06:50:10.8602286Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8637136Z Entering 'third_party/nccl/nccl' 2024-06-26T06:50:10.8669999Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8704608Z Entering 'third_party/nlohmann' 2024-06-26T06:50:10.8737846Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8774191Z Entering 'third_party/onnx' 2024-06-26T06:50:10.8807474Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8860680Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:50:10.8893768Z http.https://github.com/.extraheader 2024-06-26T06:50:10.8928886Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:50:10.8962347Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9000182Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:50:10.9035017Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9071386Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:50:10.9102871Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9137494Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:50:10.9170862Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9204869Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:50:10.9237286Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9271585Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:50:10.9303766Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9340616Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:50:10.9372692Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9406967Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:50:10.9439643Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9473473Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:50:10.9505563Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9538529Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:50:10.9572161Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9609192Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:50:10.9641547Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9678497Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:50:10.9710889Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9767660Z Entering 'third_party/pocketfft' 2024-06-26T06:50:10.9801247Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9835818Z Entering 'third_party/protobuf' 2024-06-26T06:50:10.9870024Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9907712Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:50:10.9940122Z http.https://github.com/.extraheader 2024-06-26T06:50:10.9974221Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:50:11.0006646Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0042983Z Entering 'third_party/psimd' 2024-06-26T06:50:11.0076087Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0110190Z Entering 'third_party/pthreadpool' 2024-06-26T06:50:11.0142859Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0176990Z Entering 'third_party/pybind11' 2024-06-26T06:50:11.0209984Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0244458Z Entering 'third_party/python-peachpy' 2024-06-26T06:50:11.0278988Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0312683Z Entering 'third_party/sleef' 2024-06-26T06:50:11.0346169Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0380060Z Entering 'third_party/tensorpipe' 2024-06-26T06:50:11.0413403Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0447774Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:50:11.0481119Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0515037Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:50:11.0547573Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0580849Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:50:11.0612711Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0646590Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:50:11.0679619Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0712259Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:50:11.0745550Z http.https://github.com/.extraheader 2024-06-26T06:50:11.0887355Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T06:50:11.0936318Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:50:11.0944709Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:50:11.0945357Z ##[endgroup] 2024-06-26T06:50:11.8537536Z Cleaning up orphan processes