2024-08-07T17:43:45.3820196Z Current runner version: '2.318.0' 2024-08-07T17:43:45.3830253Z Runner name: 'i-095fa8d57edae1a62' 2024-08-07T17:43:45.3831629Z Runner group name: 'Default' 2024-08-07T17:43:45.3833498Z Machine name: 'ip-10-0-53-8' 2024-08-07T17:43:45.3840274Z ##[group]GITHUB_TOKEN Permissions 2024-08-07T17:43:45.3843728Z Actions: read 2024-08-07T17:43:45.3844837Z Attestations: read 2024-08-07T17:43:45.3845829Z Checks: read 2024-08-07T17:43:45.3846809Z Contents: read 2024-08-07T17:43:45.3847849Z Deployments: read 2024-08-07T17:43:45.3848857Z Discussions: read 2024-08-07T17:43:45.3849863Z Issues: read 2024-08-07T17:43:45.3850849Z Metadata: read 2024-08-07T17:43:45.3851833Z Packages: read 2024-08-07T17:43:45.3852826Z Pages: read 2024-08-07T17:43:45.3853835Z PullRequests: read 2024-08-07T17:43:45.3854861Z RepositoryProjects: read 2024-08-07T17:43:45.3855950Z SecurityEvents: read 2024-08-07T17:43:45.3856993Z Statuses: read 2024-08-07T17:43:45.3857953Z ##[endgroup] 2024-08-07T17:43:45.3862707Z Secret source: Actions 2024-08-07T17:43:45.3864312Z Prepare workflow directory 2024-08-07T17:43:45.7920753Z Prepare all required actions 2024-08-07T17:43:45.7974963Z Getting action download info 2024-08-07T17:43:45.9656761Z Download action repository 'pytorch/test-infra@main' (SHA:a1f5a89251fc4258ab59806094fe3108f7d6741a) 2024-08-07T17:43:46.4210245Z Download action repository 'pytorch/pytorch@main' (SHA:a62710c82039b798befd86e938d2137af3978c93) 2024-08-07T17:43:50.1664746Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-07T17:43:50.3184391Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-07T17:43:50.5921242Z Getting action download info 2024-08-07T17:43:50.6787684Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-07T17:43:50.9714470Z Getting action download info 2024-08-07T17:43:51.0642578Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-07T17:43:51.2183740Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/131248/merge (f779f6b7738020e244184bded4026b37de3f9f24) 2024-08-07T17:43:51.2186978Z ##[group] Inputs 2024-08-07T17:43:51.2187654Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:43:51.2189415Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:43:51.2191532Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:43:51.2192718Z sync-tag: 2024-08-07T17:43:51.2194463Z timeout-minutes: 360 2024-08-07T17:43:51.2195029Z use-gha: 2024-08-07T17:43:51.2195552Z dashboard-tag: 2024-08-07T17:43:51.2196080Z s3-bucket: gha-artifacts 2024-08-07T17:43:51.2196606Z aws-role-to-assume: 2024-08-07T17:43:51.2197136Z ##[endgroup] 2024-08-07T17:43:51.2197933Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:43:51.2996531Z A job started hook has been configured by the self-hosted runner administrator 2024-08-07T17:43:51.3166148Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-07T17:43:51.3179084Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:51.3180188Z ##[endgroup] 2024-08-07T17:43:52.8829972Z Runner Type: linux.8xlarge.nvidia.gpu 2024-08-07T17:43:52.8830651Z Instance Type: g3.8xlarge 2024-08-07T17:43:52.8831074Z AMI Name: amzn2-ami-hvm-2.0.20240306.2-x86_64-ebs 2024-08-07T17:43:52.8831554Z AMI ID: ami-0ce0c36d7a00b20e2 2024-08-07T17:43:57.8912286Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-07T17:43:57.8913145Z with: 2024-08-07T17:43:57.8914363Z github-secret: *** 2024-08-07T17:43:57.8915400Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:57.8916418Z activate-with-label: false 2024-08-07T17:43:57.8916846Z label: with-ssh 2024-08-07T17:43:57.8917249Z remove-existing-keys: true 2024-08-07T17:43:57.8917694Z fail-silently: true 2024-08-07T17:43:57.8918070Z env: 2024-08-07T17:43:57.8918434Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:57.8918865Z ##[endgroup] 2024-08-07T17:43:58.0030400Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-07T17:43:58.3037386Z Grabbing public ssh keys from https://github.com/zdevito.keys 2024-08-07T17:43:58.3717371Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-07T17:43:58.3738631Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-07T17:43:58.3781934Z Login using: ssh ec2-user@ec2-18-208-249-13.compute-1.amazonaws.com 2024-08-07T17:43:58.3782795Z All testing is done inside the container, to start an interactive session run: 2024-08-07T17:43:58.3783579Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-07T17:43:58.3923996Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-07T17:43:58.3924663Z with: 2024-08-07T17:43:58.3925020Z submodules: recursive 2024-08-07T17:43:58.3925439Z fetch-depth: 0 2024-08-07T17:43:58.3925833Z env: 2024-08-07T17:43:58.3926177Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:58.3926595Z ##[endgroup] 2024-08-07T17:43:58.4043509Z ##[group]Run retry () { 2024-08-07T17:43:58.4043986Z retry () { 2024-08-07T17:43:58.4044504Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-07T17:43:58.4045085Z } 2024-08-07T17:43:58.4045483Z echo "${GITHUB_WORKSPACE}" 2024-08-07T17:43:58.4045982Z if [ -z "${NO_SUDO}" ]; then 2024-08-07T17:43:58.4046508Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:58.4047024Z else 2024-08-07T17:43:58.4047430Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-07T17:43:58.4047923Z fi 2024-08-07T17:43:58.4048385Z mkdir "${GITHUB_WORKSPACE}" 2024-08-07T17:43:58.4056865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:43:58.4057437Z env: 2024-08-07T17:43:58.4057789Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:43:58.4058216Z NO_SUDO: 2024-08-07T17:43:58.4058589Z ##[endgroup] 2024-08-07T17:43:58.4087217Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:44:01.6056758Z ##[group]Run malfet/checkout@silent-checkout 2024-08-07T17:44:01.6057273Z with: 2024-08-07T17:44:01.6057660Z ref: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:01.6058160Z fetch-depth: 0 2024-08-07T17:44:01.6058552Z submodules: recursive 2024-08-07T17:44:01.6058974Z quiet-checkout: true 2024-08-07T17:44:01.6059399Z repository: pytorch/pytorch 2024-08-07T17:44:01.6060002Z token: *** 2024-08-07T17:44:01.6060386Z ssh-strict: true 2024-08-07T17:44:01.6060771Z persist-credentials: true 2024-08-07T17:44:01.6061210Z clean: true 2024-08-07T17:44:01.6061607Z sparse-checkout-cone-mode: true 2024-08-07T17:44:01.6062065Z lfs: false 2024-08-07T17:44:01.6062446Z set-safe-directory: true 2024-08-07T17:44:01.6062850Z env: 2024-08-07T17:44:01.6063207Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:44:01.6063628Z ##[endgroup] 2024-08-07T17:44:01.7369706Z Syncing repository: pytorch/pytorch 2024-08-07T17:44:01.7371737Z ##[group]Getting Git version info 2024-08-07T17:44:01.7372445Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:44:01.7373425Z [command]/usr/bin/git version 2024-08-07T17:44:01.7373927Z git version 2.40.1 2024-08-07T17:44:01.7410002Z ##[endgroup] 2024-08-07T17:44:01.7433107Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7c7addb9-5234-4dac-b8c5-1bf68a05713e' before making global git config changes 2024-08-07T17:44:01.7435018Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T17:44:01.7441323Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:44:01.7487129Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-07T17:44:01.7493736Z ##[group]Initializing the repository 2024-08-07T17:44:01.7497960Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:44:01.7529326Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-07T17:44:01.7530086Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-07T17:44:01.7530814Z hint: of your new repositories, which will suppress this warning, call: 2024-08-07T17:44:01.7531333Z hint: 2024-08-07T17:44:01.7531764Z hint: git config --global init.defaultBranch 2024-08-07T17:44:01.7532234Z hint: 2024-08-07T17:44:01.7532658Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-07T17:44:01.7533376Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-07T17:44:01.7534198Z hint: 2024-08-07T17:44:01.7534529Z hint: git branch -m 2024-08-07T17:44:01.7535198Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-07T17:44:01.7543404Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-07T17:44:01.7577065Z ##[endgroup] 2024-08-07T17:44:01.7577638Z ##[group]Disabling automatic garbage collection 2024-08-07T17:44:01.7582094Z [command]/usr/bin/git config --local gc.auto 0 2024-08-07T17:44:01.7614835Z ##[endgroup] 2024-08-07T17:44:01.7615358Z ##[group]Setting up auth 2024-08-07T17:44:01.7625114Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T17:44:01.7660147Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T17:44:01.7948598Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T17:44:01.7979480Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T17:44:01.8269227Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:44:01.8320767Z ##[endgroup] 2024-08-07T17:44:01.8321447Z ##[group]Fetching the repository 2024-08-07T17:44:01.8330060Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-07T17:44:03.8233883Z remote: Enumerating objects: 1008246 2024-08-07T17:44:03.8234518Z remote: Enumerating objects: 1011035, done. 2024-08-07T17:44:03.8235831Z remote: Counting objects: 0% (1/2789) 2024-08-07T17:44:03.8236395Z remote: Counting objects: 1% (28/2789) 2024-08-07T17:44:03.8237082Z remote: Counting objects: 2% (56/2789) 2024-08-07T17:44:03.8237922Z remote: Counting objects: 3% (84/2789) 2024-08-07T17:44:03.8238631Z remote: Counting objects: 4% (112/2789) 2024-08-07T17:44:03.8240872Z remote: Counting objects: 5% (140/2789) 2024-08-07T17:44:03.8241429Z remote: Counting objects: 6% (168/2789) 2024-08-07T17:44:03.8241994Z remote: Counting objects: 7% (196/2789) 2024-08-07T17:44:03.8242546Z remote: Counting objects: 8% (224/2789) 2024-08-07T17:44:03.8243409Z remote: Counting objects: 9% (252/2789) 2024-08-07T17:44:03.8243998Z remote: Counting objects: 10% (279/2789) 2024-08-07T17:44:03.8244960Z remote: Counting objects: 11% (307/2789) 2024-08-07T17:44:03.8245520Z remote: Counting objects: 12% (335/2789) 2024-08-07T17:44:03.8246061Z remote: Counting objects: 13% (363/2789) 2024-08-07T17:44:03.8246615Z remote: Counting objects: 14% (391/2789) 2024-08-07T17:44:03.8247180Z remote: Counting objects: 15% (419/2789) 2024-08-07T17:44:03.8247742Z remote: Counting objects: 16% (447/2789) 2024-08-07T17:44:03.8248301Z remote: Counting objects: 17% (475/2789) 2024-08-07T17:44:03.8248860Z remote: Counting objects: 18% (503/2789) 2024-08-07T17:44:03.8249399Z remote: Counting objects: 19% (530/2789) 2024-08-07T17:44:03.8249952Z remote: Counting objects: 20% (558/2789) 2024-08-07T17:44:03.8250505Z remote: Counting objects: 21% (586/2789) 2024-08-07T17:44:03.8251046Z remote: Counting objects: 22% (614/2789) 2024-08-07T17:44:03.8251601Z remote: Counting objects: 23% (642/2789) 2024-08-07T17:44:03.8252162Z remote: Counting objects: 24% (670/2789) 2024-08-07T17:44:03.8252694Z remote: Counting objects: 25% (698/2789) 2024-08-07T17:44:03.8253247Z remote: Counting objects: 26% (726/2789) 2024-08-07T17:44:03.8253892Z remote: Counting objects: 27% (754/2789) 2024-08-07T17:44:03.8254650Z remote: Counting objects: 28% (781/2789) 2024-08-07T17:44:03.8255235Z remote: Counting objects: 29% (809/2789) 2024-08-07T17:44:03.8255781Z remote: Counting objects: 30% (837/2789) 2024-08-07T17:44:03.8256313Z remote: Counting objects: 31% (865/2789) 2024-08-07T17:44:03.8256868Z remote: Counting objects: 32% (893/2789) 2024-08-07T17:44:03.8257425Z remote: Counting objects: 33% (921/2789) 2024-08-07T17:44:03.8257959Z remote: Counting objects: 34% (949/2789) 2024-08-07T17:44:03.8258511Z remote: Counting objects: 35% (977/2789) 2024-08-07T17:44:03.8259061Z remote: Counting objects: 36% (1005/2789) 2024-08-07T17:44:03.8259609Z remote: Counting objects: 37% (1032/2789) 2024-08-07T17:44:03.8260164Z remote: Counting objects: 38% (1060/2789) 2024-08-07T17:44:03.8260715Z remote: Counting objects: 39% (1088/2789) 2024-08-07T17:44:03.8261282Z remote: Counting objects: 40% (1116/2789) 2024-08-07T17:44:03.8261819Z remote: Counting objects: 41% (1144/2789) 2024-08-07T17:44:03.8262370Z remote: Counting objects: 42% (1172/2789) 2024-08-07T17:44:03.8262920Z remote: Counting objects: 43% (1200/2789) 2024-08-07T17:44:03.8263455Z remote: Counting objects: 44% (1228/2789) 2024-08-07T17:44:03.8264007Z remote: Counting objects: 45% (1256/2789) 2024-08-07T17:44:03.8264602Z remote: Counting objects: 46% (1283/2789) 2024-08-07T17:44:03.8265146Z remote: Counting objects: 47% (1311/2789) 2024-08-07T17:44:03.8265706Z remote: Counting objects: 48% (1339/2789) 2024-08-07T17:44:03.8266261Z remote: Counting objects: 49% (1367/2789) 2024-08-07T17:44:03.8266862Z remote: Counting objects: 50% (1395/2789) 2024-08-07T17:44:03.8267420Z remote: Counting objects: 51% (1423/2789) 2024-08-07T17:44:03.8267981Z remote: Counting objects: 52% (1451/2789) 2024-08-07T17:44:03.8268527Z remote: Counting objects: 53% (1479/2789) 2024-08-07T17:44:03.8269086Z remote: Counting objects: 54% (1507/2789) 2024-08-07T17:44:03.8269644Z remote: Counting objects: 55% (1534/2789) 2024-08-07T17:44:03.8270178Z remote: Counting objects: 56% (1562/2789) 2024-08-07T17:44:03.8270744Z remote: Counting objects: 57% (1590/2789) 2024-08-07T17:44:03.8271301Z remote: Counting objects: 58% (1618/2789) 2024-08-07T17:44:03.8271845Z remote: Counting objects: 59% (1646/2789) 2024-08-07T17:44:03.8272502Z remote: Counting objects: 60% (1674/2789) 2024-08-07T17:44:03.8273063Z remote: Counting objects: 61% (1702/2789) 2024-08-07T17:44:03.8273852Z remote: Counting objects: 62% (1730/2789) 2024-08-07T17:44:03.8274626Z remote: Counting objects: 63% (1758/2789) 2024-08-07T17:44:03.8275188Z remote: Counting objects: 64% (1785/2789) 2024-08-07T17:44:03.8275746Z remote: Counting objects: 65% (1813/2789) 2024-08-07T17:44:03.8276295Z remote: Counting objects: 66% (1841/2789) 2024-08-07T17:44:03.8276858Z remote: Counting objects: 67% (1869/2789) 2024-08-07T17:44:03.8277416Z remote: Counting objects: 68% (1897/2789) 2024-08-07T17:44:03.8277961Z remote: Counting objects: 69% (1925/2789) 2024-08-07T17:44:03.8278516Z remote: Counting objects: 70% (1953/2789) 2024-08-07T17:44:03.8279071Z remote: Counting objects: 71% (1981/2789) 2024-08-07T17:44:03.8279606Z remote: Counting objects: 72% (2009/2789) 2024-08-07T17:44:03.8280160Z remote: Counting objects: 73% (2036/2789) 2024-08-07T17:44:03.8280731Z remote: Counting objects: 74% (2064/2789) 2024-08-07T17:44:03.8281277Z remote: Counting objects: 75% (2092/2789) 2024-08-07T17:44:03.8281879Z remote: Counting objects: 76% (2120/2789) 2024-08-07T17:44:03.8282432Z remote: Counting objects: 77% (2148/2789) 2024-08-07T17:44:03.8282984Z remote: Counting objects: 78% (2176/2789) 2024-08-07T17:44:03.8283648Z remote: Counting objects: 79% (2204/2789) 2024-08-07T17:44:03.8284224Z remote: Counting objects: 80% (2232/2789) 2024-08-07T17:44:03.8284778Z remote: Counting objects: 81% (2260/2789) 2024-08-07T17:44:03.8285310Z remote: Counting objects: 82% (2287/2789) 2024-08-07T17:44:03.8285860Z remote: Counting objects: 83% (2315/2789) 2024-08-07T17:44:03.8286418Z remote: Counting objects: 84% (2343/2789) 2024-08-07T17:44:03.8286954Z remote: Counting objects: 85% (2371/2789) 2024-08-07T17:44:03.8287515Z remote: Counting objects: 86% (2399/2789) 2024-08-07T17:44:03.8288070Z remote: Counting objects: 87% (2427/2789) 2024-08-07T17:44:03.8288618Z remote: Counting objects: 88% (2455/2789) 2024-08-07T17:44:03.8289172Z remote: Counting objects: 89% (2483/2789) 2024-08-07T17:44:03.8289730Z remote: Counting objects: 90% (2511/2789) 2024-08-07T17:44:03.8290265Z remote: Counting objects: 91% (2538/2789) 2024-08-07T17:44:03.8290823Z remote: Counting objects: 92% (2566/2789) 2024-08-07T17:44:03.8291382Z remote: Counting objects: 93% (2594/2789) 2024-08-07T17:44:03.8291917Z remote: Counting objects: 94% (2622/2789) 2024-08-07T17:44:03.8292471Z remote: Counting objects: 95% (2650/2789) 2024-08-07T17:44:03.8293024Z remote: Counting objects: 96% (2678/2789) 2024-08-07T17:44:03.8293560Z remote: Counting objects: 97% (2706/2789) 2024-08-07T17:44:03.8294113Z remote: Counting objects: 98% (2734/2789) 2024-08-07T17:44:03.8294668Z remote: Counting objects: 99% (2762/2789) 2024-08-07T17:44:03.8295204Z remote: Counting objects: 100% (2789/2789) 2024-08-07T17:44:03.8295808Z remote: Counting objects: 100% (2789/2789), done. 2024-08-07T17:44:03.8623013Z remote: Compressing objects: 0% (1/1486) 2024-08-07T17:44:03.8810337Z remote: Compressing objects: 1% (15/1486) 2024-08-07T17:44:03.9896836Z remote: Compressing objects: 2% (30/1486) 2024-08-07T17:44:04.1789571Z remote: Compressing objects: 3% (45/1486) 2024-08-07T17:44:04.2919161Z remote: Compressing objects: 4% (60/1486) 2024-08-07T17:44:04.3658605Z remote: Compressing objects: 5% (75/1486) 2024-08-07T17:44:04.4303772Z remote: Compressing objects: 6% (90/1486) 2024-08-07T17:44:04.4773908Z remote: Compressing objects: 7% (105/1486) 2024-08-07T17:44:04.5193613Z remote: Compressing objects: 8% (119/1486) 2024-08-07T17:44:04.5481170Z remote: Compressing objects: 9% (134/1486) 2024-08-07T17:44:04.5751541Z remote: Compressing objects: 10% (149/1486) 2024-08-07T17:44:04.5978735Z remote: Compressing objects: 11% (164/1486) 2024-08-07T17:44:04.6186438Z remote: Compressing objects: 12% (179/1486) 2024-08-07T17:44:04.6375761Z remote: Compressing objects: 13% (194/1486) 2024-08-07T17:44:04.6520962Z remote: Compressing objects: 14% (209/1486) 2024-08-07T17:44:04.6650703Z remote: Compressing objects: 15% (223/1486) 2024-08-07T17:44:04.6741678Z remote: Compressing objects: 16% (238/1486) 2024-08-07T17:44:04.6813502Z remote: Compressing objects: 17% (253/1486) 2024-08-07T17:44:04.6866952Z remote: Compressing objects: 18% (268/1486) 2024-08-07T17:44:04.6905019Z remote: Compressing objects: 19% (283/1486) 2024-08-07T17:44:04.6927536Z remote: Compressing objects: 20% (298/1486) 2024-08-07T17:44:04.6937374Z remote: Compressing objects: 21% (313/1486) 2024-08-07T17:44:04.6944548Z remote: Compressing objects: 22% (327/1486) 2024-08-07T17:44:04.6975113Z remote: Compressing objects: 23% (342/1486) 2024-08-07T17:44:04.6997510Z remote: Compressing objects: 24% (357/1486) 2024-08-07T17:44:04.7019307Z remote: Compressing objects: 25% (372/1486) 2024-08-07T17:44:04.7026500Z remote: Compressing objects: 26% (387/1486) 2024-08-07T17:44:04.7048368Z remote: Compressing objects: 27% (402/1486) 2024-08-07T17:44:04.7062450Z remote: Compressing objects: 28% (417/1486) 2024-08-07T17:44:04.7075589Z remote: Compressing objects: 29% (431/1486) 2024-08-07T17:44:04.7098901Z remote: Compressing objects: 30% (446/1486) 2024-08-07T17:44:04.7114696Z remote: Compressing objects: 31% (461/1486) 2024-08-07T17:44:04.7127149Z remote: Compressing objects: 32% (476/1486) 2024-08-07T17:44:04.7137106Z remote: Compressing objects: 33% (491/1486) 2024-08-07T17:44:04.7148530Z remote: Compressing objects: 34% (506/1486) 2024-08-07T17:44:04.7158828Z remote: Compressing objects: 35% (521/1486) 2024-08-07T17:44:04.7168910Z remote: Compressing objects: 36% (535/1486) 2024-08-07T17:44:04.7176185Z remote: Compressing objects: 37% (550/1486) 2024-08-07T17:44:04.7187463Z remote: Compressing objects: 38% (565/1486) 2024-08-07T17:44:04.7198680Z remote: Compressing objects: 39% (580/1486) 2024-08-07T17:44:04.7205280Z remote: Compressing objects: 40% (595/1486) 2024-08-07T17:44:04.7215088Z remote: Compressing objects: 41% (610/1486) 2024-08-07T17:44:04.7221584Z remote: Compressing objects: 42% (625/1486) 2024-08-07T17:44:04.7229908Z remote: Compressing objects: 43% (639/1486) 2024-08-07T17:44:04.7236294Z remote: Compressing objects: 44% (654/1486) 2024-08-07T17:44:04.7242349Z remote: Compressing objects: 45% (669/1486) 2024-08-07T17:44:04.7247078Z remote: Compressing objects: 46% (684/1486) 2024-08-07T17:44:04.7251954Z remote: Compressing objects: 47% (699/1486) 2024-08-07T17:44:04.7257413Z remote: Compressing objects: 48% (714/1486) 2024-08-07T17:44:04.7261609Z remote: Compressing objects: 49% (729/1486) 2024-08-07T17:44:04.7266381Z remote: Compressing objects: 50% (743/1486) 2024-08-07T17:44:04.7269985Z remote: Compressing objects: 51% (758/1486) 2024-08-07T17:44:04.7274216Z remote: Compressing objects: 52% (773/1486) 2024-08-07T17:44:04.7277308Z remote: Compressing objects: 53% (788/1486) 2024-08-07T17:44:04.7279719Z remote: Compressing objects: 54% (803/1486) 2024-08-07T17:44:04.7281697Z remote: Compressing objects: 55% (818/1486) 2024-08-07T17:44:04.7282878Z remote: Compressing objects: 56% (833/1486) 2024-08-07T17:44:04.7286318Z remote: Compressing objects: 57% (848/1486) 2024-08-07T17:44:04.7286916Z remote: Compressing objects: 58% (862/1486) 2024-08-07T17:44:04.7287500Z remote: Compressing objects: 59% (877/1486) 2024-08-07T17:44:04.7288057Z remote: Compressing objects: 60% (892/1486) 2024-08-07T17:44:04.7288646Z remote: Compressing objects: 61% (907/1486) 2024-08-07T17:44:04.7289449Z remote: Compressing objects: 62% (922/1486) 2024-08-07T17:44:04.7290008Z remote: Compressing objects: 63% (937/1486) 2024-08-07T17:44:04.7290586Z remote: Compressing objects: 64% (952/1486) 2024-08-07T17:44:04.7291175Z remote: Compressing objects: 65% (966/1486) 2024-08-07T17:44:04.7307181Z remote: Compressing objects: 66% (981/1486) 2024-08-07T17:44:04.7318106Z remote: Compressing objects: 67% (996/1486) 2024-08-07T17:44:04.7328099Z remote: Compressing objects: 68% (1011/1486) 2024-08-07T17:44:04.7336802Z remote: Compressing objects: 69% (1026/1486) 2024-08-07T17:44:04.7345165Z remote: Compressing objects: 70% (1041/1486) 2024-08-07T17:44:04.7351447Z remote: Compressing objects: 71% (1056/1486) 2024-08-07T17:44:04.7357624Z remote: Compressing objects: 72% (1070/1486) 2024-08-07T17:44:04.7363006Z remote: Compressing objects: 73% (1085/1486) 2024-08-07T17:44:04.7366209Z remote: Compressing objects: 74% (1100/1486) 2024-08-07T17:44:04.7369625Z remote: Compressing objects: 75% (1115/1486) 2024-08-07T17:44:04.7373356Z remote: Compressing objects: 76% (1130/1486) 2024-08-07T17:44:04.7377960Z remote: Compressing objects: 77% (1145/1486) 2024-08-07T17:44:04.7381100Z remote: Compressing objects: 78% (1160/1486) 2024-08-07T17:44:04.7385475Z remote: Compressing objects: 79% (1174/1486) 2024-08-07T17:44:04.7388433Z remote: Compressing objects: 80% (1189/1486) 2024-08-07T17:44:04.7391586Z remote: Compressing objects: 81% (1204/1486) 2024-08-07T17:44:04.7394908Z remote: Compressing objects: 82% (1219/1486) 2024-08-07T17:44:04.7397477Z remote: Compressing objects: 83% (1234/1486) 2024-08-07T17:44:04.7400792Z remote: Compressing objects: 84% (1249/1486) 2024-08-07T17:44:04.7403299Z remote: Compressing objects: 85% (1264/1486) 2024-08-07T17:44:04.7405371Z remote: Compressing objects: 86% (1278/1486) 2024-08-07T17:44:04.7407431Z remote: Compressing objects: 87% (1293/1486) 2024-08-07T17:44:04.7408836Z remote: Compressing objects: 88% (1308/1486) 2024-08-07T17:44:04.7411599Z remote: Compressing objects: 89% (1323/1486) 2024-08-07T17:44:04.7413779Z remote: Compressing objects: 90% (1338/1486) 2024-08-07T17:44:04.7416256Z remote: Compressing objects: 91% (1353/1486) 2024-08-07T17:44:04.7417653Z remote: Compressing objects: 92% (1368/1486) 2024-08-07T17:44:04.7420485Z remote: Compressing objects: 93% (1382/1486) 2024-08-07T17:44:04.7421682Z remote: Compressing objects: 94% (1397/1486) 2024-08-07T17:44:04.7423989Z remote: Compressing objects: 95% (1412/1486) 2024-08-07T17:44:04.7425945Z remote: Compressing objects: 96% (1427/1486) 2024-08-07T17:44:04.7427149Z remote: Compressing objects: 97% (1442/1486) 2024-08-07T17:44:04.7429272Z remote: Compressing objects: 98% (1457/1486) 2024-08-07T17:44:04.7433438Z remote: Compressing objects: 99% (1472/1486) 2024-08-07T17:44:04.7434286Z remote: Compressing objects: 100% (1486/1486) 2024-08-07T17:44:04.7434919Z remote: Compressing objects: 100% (1486/1486), done. 2024-08-07T17:44:31.3064559Z remote: Total 1011035 (delta 1772), reused 2144 (delta 1296), pack-reused 1008246 2024-08-07T17:44:54.1828990Z [command]/usr/bin/git rev-parse --verify --quiet 016588f53c6904b840aa56aa86f95460b4d9c996^{object} 2024-08-07T17:44:54.1856941Z 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:54.1863303Z ##[endgroup] 2024-08-07T17:44:54.1863860Z ##[group]Determining the checkout info 2024-08-07T17:44:54.1865991Z ##[endgroup] 2024-08-07T17:44:54.1866520Z ##[group]Checking out the ref 2024-08-07T17:44:54.1870738Z [command]/usr/bin/git checkout --quiet --force 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:44:56.0055389Z ##[endgroup] 2024-08-07T17:44:56.0056136Z ##[group]Setting up auth for fetching submodules 2024-08-07T17:44:56.0061981Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-07T17:44:56.0117292Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-07T17:44:56.0148948Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-07T17:44:56.0181754Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-07T17:44:56.0212564Z ##[endgroup] 2024-08-07T17:44:56.0213186Z ##[group]Fetching submodules 2024-08-07T17:44:56.0272993Z [command]/usr/bin/git submodule sync --recursive 2024-08-07T17:44:56.0544456Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-07T17:44:56.0849638Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-07T17:44:56.0852223Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-07T17:44:56.0855279Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-07T17:44:56.0858670Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-07T17:44:56.0862181Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-07T17:44:56.0865276Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-07T17:44:56.0868835Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-07T17:44:56.0872654Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-07T17:44:56.0878096Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-07T17:44:56.0882280Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-07T17:44:56.0886255Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-07T17:44:56.0890473Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-07T17:44:56.0894869Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-07T17:44:56.0899486Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-07T17:44:56.0904049Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-07T17:44:56.0908840Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-08-07T17:44:56.0913940Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:44:56.0919299Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-07T17:44:56.0924480Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-07T17:44:56.0929691Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-07T17:44:56.0935148Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-07T17:44:56.0940841Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-07T17:44:56.0946847Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-07T17:44:56.0952631Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-07T17:44:56.0959029Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-07T17:44:56.0965094Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-07T17:44:56.0971638Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-07T17:44:56.0977856Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-07T17:44:56.0984378Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-07T17:44:56.0991042Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-07T17:44:56.0998435Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-07T17:44:56.1005044Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-07T17:44:56.1012072Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-07T17:44:56.1019080Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-07T17:44:56.1026441Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-07T17:44:56.1054846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-07T17:44:56.3485435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-07T17:44:56.5105474Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-07T17:44:56.7113968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-07T17:44:56.9590138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-07T17:44:58.9045796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-07T17:45:10.8811555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-07T17:45:11.2967092Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-07T17:45:11.7287291Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-07T17:45:12.3800204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-07T17:45:13.8358686Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-07T17:45:16.1715595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-07T17:45:23.2134114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-07T17:45:24.6211340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-07T17:45:26.4656583Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-07T17:45:27.7431072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-08-07T17:45:29.0535704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-07T17:45:29.4826992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-07T17:45:29.7708674Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-07T17:45:30.8963946Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-07T17:45:31.9336751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-07T17:45:32.1725925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-07T17:45:33.9834182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-07T17:45:34.7418092Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-07T17:45:35.4744875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-07T17:45:42.3270598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-07T17:45:44.4133986Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-07T17:45:48.1260030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-07T17:45:48.4526034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-07T17:45:58.2018341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-07T17:45:58.3629161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-07T17:45:58.5570478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-07T17:45:59.4774300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-07T17:45:59.8274940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-07T17:46:00.5178380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-07T17:46:00.9585176Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-07T17:46:00.9711252Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-07T17:46:00.9806993Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-07T17:46:01.0086898Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-07T17:46:01.0532990Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-07T17:46:02.2630967Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-07T17:46:02.2899874Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-07T17:46:02.3447472Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-07T17:46:02.4666947Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-07T17:46:02.5063408Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019' 2024-08-07T17:46:03.1550266Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-07T17:46:03.4705622Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-07T17:46:03.5684698Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-07T17:46:03.5703062Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:46:03.5705897Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:46:03.5709138Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:46:03.5712554Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:46:03.5716879Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:46:03.5742560Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-07T17:46:04.5734497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-07T17:46:05.2648877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-07T17:46:07.5698535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-07T17:46:08.7095489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-07T17:46:09.0679935Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-07T17:46:09.1868216Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-07T17:46:09.7088667Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-07T17:46:09.7843658Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-07T17:46:09.7973658Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-07T17:46:09.9348380Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-07T17:46:09.9821976Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-07T17:46:09.9921475Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-08-07T17:46:10.0380413Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-07T17:46:10.0674536Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-07T17:46:10.1205384Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:46:10.1343001Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-07T17:46:10.1359442Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-07T17:46:10.1384464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-07T17:46:25.1232550Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-07T17:46:25.1436803Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-07T17:46:25.2573961Z Submodule path 'third_party/kineto': checked out 'da2f2682cabaf95d601fa2a9b7e0979f84fe7667' 2024-08-07T17:46:25.2592215Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:46:25.2595968Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:46:25.2599279Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:46:25.2625125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-07T17:46:25.8053873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-07T17:46:26.9850554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-07T17:46:28.2063152Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-07T17:46:28.2079865Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:46:28.2083266Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:46:28.2086593Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:46:28.2090059Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:46:28.2093614Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:46:28.2097558Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:46:28.2101156Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:46:28.2104867Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:46:28.2132687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-07T17:46:29.2098094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-07T17:46:29.5667372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-07T17:46:30.8189756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-07T17:46:31.0965726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-07T17:46:31.6466452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-07T17:46:32.8159347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-07T17:46:39.7738326Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-07T17:46:40.2036963Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-07T17:46:40.2248578Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-07T17:46:40.2675514Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-07T17:46:40.2818057Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-07T17:46:40.2834022Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:46:40.2860399Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-07T17:46:40.5857473Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-07T17:46:40.6059886Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-07T17:46:40.6532748Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-07T17:46:40.7823061Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-07T17:46:40.8005782Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-07T17:46:40.8465916Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-07T17:46:40.9147808Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-07T17:46:40.9585083Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-07T17:46:40.9855833Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-07T17:46:41.1146701Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-07T17:46:41.5494417Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-07T17:46:41.5531593Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-07T17:46:41.5534598Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-07T17:46:41.5562582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-07T17:46:41.9808508Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-07T17:46:42.9195468Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-07T17:46:42.9590913Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-07T17:46:43.0392489Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-07T17:46:43.0411062Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:46:43.0414187Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:46:43.0417351Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:46:43.0420782Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:46:43.0424483Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:46:43.0428062Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:46:43.0431738Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:46:43.0435912Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:46:43.0462121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-07T17:46:43.8366023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-07T17:46:45.0627454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-07T17:46:45.5238774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-07T17:46:52.3940817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-07T17:46:52.6425708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-07T17:46:52.8574646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-07T17:46:53.1582996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-07T17:47:00.2205019Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-07T17:47:00.2685257Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-07T17:47:00.2861478Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-07T17:47:00.4135412Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-07T17:47:00.4276992Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-07T17:47:00.4438497Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-07T17:47:00.4609035Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-07T17:47:00.4625791Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:00.4629062Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:00.4655940Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-07T17:47:02.4934545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-07T17:47:04.0294663Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-07T17:47:04.0834426Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-07T17:47:04.6586805Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-07T17:47:04.6711440Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-07T17:47:04.9994709Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-07T17:47:05.0017938Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:05.0020511Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:05.0048053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-07T17:47:05.4573636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-07T17:47:06.6228501Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-07T17:47:06.7089066Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-07T17:47:06.7180491Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-07T17:47:06.7309656Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-07T17:47:06.7744694Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-07T17:47:06.8073213Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-07T17:47:06.8575885Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-07T17:47:06.8874214Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-07T17:47:06.8891770Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:06.8894926Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:06.8898247Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:06.8901811Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:06.8928397Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-07T17:47:08.0364008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-07T17:47:08.3025458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-07T17:47:09.5709960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-07T17:47:10.5766104Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-07T17:47:10.5931669Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-07T17:47:10.6753855Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-07T17:47:10.7084043Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-07T17:47:10.7099775Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:10.7125849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-07T17:47:10.9052308Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-07T17:47:10.9086937Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-07T17:47:10.9406754Z Entering 'android/libs/fbjni' 2024-08-07T17:47:10.9448521Z Entering 'third_party/FP16' 2024-08-07T17:47:10.9489713Z Entering 'third_party/FXdiv' 2024-08-07T17:47:10.9531329Z Entering 'third_party/NNPACK' 2024-08-07T17:47:10.9573908Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:47:10.9616416Z Entering 'third_party/XNNPACK' 2024-08-07T17:47:10.9676786Z Entering 'third_party/benchmark' 2024-08-07T17:47:10.9719187Z Entering 'third_party/cpp-httplib' 2024-08-07T17:47:10.9760508Z Entering 'third_party/cpuinfo' 2024-08-07T17:47:10.9802716Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:47:10.9844395Z Entering 'third_party/cutlass' 2024-08-07T17:47:10.9893453Z Entering 'third_party/eigen' 2024-08-07T17:47:10.9938758Z Entering 'third_party/fbgemm' 2024-08-07T17:47:10.9980143Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:47:11.0021214Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:47:11.0063237Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:47:11.0110032Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:47:11.0150549Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:47:11.0193219Z Entering 'third_party/flatbuffers' 2024-08-07T17:47:11.0237983Z Entering 'third_party/fmt' 2024-08-07T17:47:11.0278897Z Entering 'third_party/foxi' 2024-08-07T17:47:11.0321742Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:47:11.0362696Z Entering 'third_party/gloo' 2024-08-07T17:47:11.0404101Z Entering 'third_party/googletest' 2024-08-07T17:47:11.0445783Z Entering 'third_party/ideep' 2024-08-07T17:47:11.0484996Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:47:11.0533093Z Entering 'third_party/ittapi' 2024-08-07T17:47:11.0573751Z Entering 'third_party/kineto' 2024-08-07T17:47:11.0615125Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:47:11.0656958Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:47:11.0700499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:47:11.0741422Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:47:11.0783649Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:47:11.0822680Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:47:11.0865184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:47:11.0905576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:47:11.0946854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:47:11.0989536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:47:11.1033414Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:47:11.1075811Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:47:11.1117334Z Entering 'third_party/mimalloc' 2024-08-07T17:47:11.1158966Z Entering 'third_party/nccl/nccl' 2024-08-07T17:47:11.1200050Z Entering 'third_party/nlohmann' 2024-08-07T17:47:11.1243050Z Entering 'third_party/onnx' 2024-08-07T17:47:11.1301269Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:47:11.1343935Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:47:11.1386752Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:47:11.1429687Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:47:11.1469238Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:47:11.1509030Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:47:11.1548268Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:47:11.1589423Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:47:11.1628628Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:47:11.1669098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:47:11.1708197Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:11.1752130Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:11.1795869Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:47:11.1857843Z Entering 'third_party/pocketfft' 2024-08-07T17:47:11.1899363Z Entering 'third_party/protobuf' 2024-08-07T17:47:11.1944617Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:11.1985000Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:11.2026634Z Entering 'third_party/psimd' 2024-08-07T17:47:11.2067710Z Entering 'third_party/pthreadpool' 2024-08-07T17:47:11.2108357Z Entering 'third_party/pybind11' 2024-08-07T17:47:11.2150304Z Entering 'third_party/python-peachpy' 2024-08-07T17:47:11.2191033Z Entering 'third_party/sleef' 2024-08-07T17:47:11.2233326Z Entering 'third_party/tensorpipe' 2024-08-07T17:47:11.2275543Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:11.2317165Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:11.2357935Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:11.2398628Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:11.2438090Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:11.2492423Z ##[endgroup] 2024-08-07T17:47:11.2496480Z ##[group]Persisting credentials for submodules 2024-08-07T17:47:11.2501910Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-07T17:47:11.2814643Z Entering 'android/libs/fbjni' 2024-08-07T17:47:11.2866359Z Entering 'third_party/FP16' 2024-08-07T17:47:11.2918481Z Entering 'third_party/FXdiv' 2024-08-07T17:47:11.2969614Z Entering 'third_party/NNPACK' 2024-08-07T17:47:11.3021946Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:47:11.3072441Z Entering 'third_party/XNNPACK' 2024-08-07T17:47:11.3142030Z Entering 'third_party/benchmark' 2024-08-07T17:47:11.3192971Z Entering 'third_party/cpp-httplib' 2024-08-07T17:47:11.3244311Z Entering 'third_party/cpuinfo' 2024-08-07T17:47:11.3295892Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:47:11.3347962Z Entering 'third_party/cutlass' 2024-08-07T17:47:11.3407432Z Entering 'third_party/eigen' 2024-08-07T17:47:11.3461354Z Entering 'third_party/fbgemm' 2024-08-07T17:47:11.3512108Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:47:11.3562937Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:47:11.3612781Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:47:11.3670081Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:47:11.3720844Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:47:11.3771628Z Entering 'third_party/flatbuffers' 2024-08-07T17:47:11.3825584Z Entering 'third_party/fmt' 2024-08-07T17:47:11.3877123Z Entering 'third_party/foxi' 2024-08-07T17:47:11.3927501Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:47:11.3978673Z Entering 'third_party/gloo' 2024-08-07T17:47:11.4030112Z Entering 'third_party/googletest' 2024-08-07T17:47:11.4082172Z Entering 'third_party/ideep' 2024-08-07T17:47:11.4131589Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:47:11.4189919Z Entering 'third_party/ittapi' 2024-08-07T17:47:11.4241272Z Entering 'third_party/kineto' 2024-08-07T17:47:11.4291600Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:47:11.4342885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:47:11.4395909Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:47:11.4445986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:47:11.4497462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:47:11.4546321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:47:11.4598737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:47:11.4648484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:47:11.4698884Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:47:11.4749685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:47:11.4803118Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:47:11.4852513Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:47:11.4904036Z Entering 'third_party/mimalloc' 2024-08-07T17:47:11.4954417Z Entering 'third_party/nccl/nccl' 2024-08-07T17:47:11.5005096Z Entering 'third_party/nlohmann' 2024-08-07T17:47:11.5057176Z Entering 'third_party/onnx' 2024-08-07T17:47:11.5125613Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:47:11.5176162Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:47:11.5228512Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:47:11.5281085Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:47:11.5331251Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:47:11.5382095Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:47:11.5430914Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:47:11.5483159Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:47:11.5533688Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:47:11.5583811Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:47:11.5635117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:11.5687555Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:11.5739288Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:47:11.5812319Z Entering 'third_party/pocketfft' 2024-08-07T17:47:11.5863189Z Entering 'third_party/protobuf' 2024-08-07T17:47:11.5918453Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:11.5968901Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:11.6020033Z Entering 'third_party/psimd' 2024-08-07T17:47:11.6070420Z Entering 'third_party/pthreadpool' 2024-08-07T17:47:11.6121865Z Entering 'third_party/pybind11' 2024-08-07T17:47:11.6173454Z Entering 'third_party/python-peachpy' 2024-08-07T17:47:11.6225045Z Entering 'third_party/sleef' 2024-08-07T17:47:11.6278030Z Entering 'third_party/tensorpipe' 2024-08-07T17:47:11.6327500Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:11.6377621Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:11.6427248Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:11.6477981Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:11.6526773Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:11.6595100Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-07T17:47:11.6900257Z Entering 'android/libs/fbjni' 2024-08-07T17:47:11.6948501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-07T17:47:11.6968121Z Entering 'third_party/FP16' 2024-08-07T17:47:11.7016142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-07T17:47:11.7034596Z Entering 'third_party/FXdiv' 2024-08-07T17:47:11.7083643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-07T17:47:11.7101333Z Entering 'third_party/NNPACK' 2024-08-07T17:47:11.7149130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-07T17:47:11.7168341Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:47:11.7217096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-07T17:47:11.7235943Z Entering 'third_party/XNNPACK' 2024-08-07T17:47:11.7283635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-07T17:47:11.7320421Z Entering 'third_party/benchmark' 2024-08-07T17:47:11.7367005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:47:11.7385573Z Entering 'third_party/cpp-httplib' 2024-08-07T17:47:11.7432352Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-07T17:47:11.7450690Z Entering 'third_party/cpuinfo' 2024-08-07T17:47:11.7498932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:47:11.7518703Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:47:11.7565879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-07T17:47:11.7584225Z Entering 'third_party/cutlass' 2024-08-07T17:47:11.7631348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:47:11.7657854Z Entering 'third_party/eigen' 2024-08-07T17:47:11.7704942Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-07T17:47:11.7726138Z Entering 'third_party/fbgemm' 2024-08-07T17:47:11.7774467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-07T17:47:11.7792469Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:47:11.7840329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-07T17:47:11.7858667Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:47:11.7906347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-07T17:47:11.7926246Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:47:11.7972951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-07T17:47:11.7997946Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:47:11.8044541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:11.8062555Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:47:11.8108869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-07T17:47:11.8128143Z Entering 'third_party/flatbuffers' 2024-08-07T17:47:11.8176046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-07T17:47:11.8197351Z Entering 'third_party/fmt' 2024-08-07T17:47:11.8245008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:47:11.8263491Z Entering 'third_party/foxi' 2024-08-07T17:47:11.8310730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-08-07T17:47:11.8329749Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:47:11.8376950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-07T17:47:11.8395987Z Entering 'third_party/gloo' 2024-08-07T17:47:11.8443427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-07T17:47:11.8461757Z Entering 'third_party/googletest' 2024-08-07T17:47:11.8508309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:11.8527200Z Entering 'third_party/ideep' 2024-08-07T17:47:11.8573726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-07T17:47:11.8591468Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:47:11.8639187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-07T17:47:11.8665801Z Entering 'third_party/ittapi' 2024-08-07T17:47:11.8712469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-07T17:47:11.8734572Z Entering 'third_party/kineto' 2024-08-07T17:47:11.8782658Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-07T17:47:11.8801425Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:47:11.8850775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-07T17:47:11.8868507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:47:11.8918543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-07T17:47:11.8938088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:47:11.8985494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-07T17:47:11.9004239Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:47:11.9052137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-07T17:47:11.9070155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:47:11.9118005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-07T17:47:11.9134739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:47:11.9182511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-07T17:47:11.9202987Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:47:11.9250062Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-07T17:47:11.9267886Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:47:11.9316552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:11.9334678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:47:11.9382079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-07T17:47:11.9402148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:47:11.9450710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-07T17:47:11.9470034Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:47:11.9516979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-07T17:47:11.9534770Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:47:11.9582139Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-07T17:47:11.9602480Z Entering 'third_party/mimalloc' 2024-08-07T17:47:11.9648914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-07T17:47:11.9667195Z Entering 'third_party/nccl/nccl' 2024-08-07T17:47:11.9714387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-07T17:47:11.9733757Z Entering 'third_party/nlohmann' 2024-08-07T17:47:11.9781049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-07T17:47:11.9801591Z Entering 'third_party/onnx' 2024-08-07T17:47:11.9848261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-07T17:47:11.9885582Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:47:11.9932740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:47:11.9950797Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:47:11.9999018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:47:12.0019578Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:47:12.0068178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-07T17:47:12.0088842Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:47:12.0137622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:47:12.0156747Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:47:12.0204527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:12.0223344Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:47:12.0269720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-07T17:47:12.0287536Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:47:12.0337059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-07T17:47:12.0357287Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:47:12.0404224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-07T17:47:12.0422049Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:47:12.0469136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-07T17:47:12.0488538Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:47:12.0535619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-07T17:47:12.0552615Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:12.0600033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-07T17:47:12.0619812Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:12.0667142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-07T17:47:12.0686840Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:47:12.0733407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-07T17:47:12.0774262Z Entering 'third_party/pocketfft' 2024-08-07T17:47:12.0821044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-07T17:47:12.0841593Z Entering 'third_party/protobuf' 2024-08-07T17:47:12.0887836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-07T17:47:12.0910008Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:12.0959310Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-07T17:47:12.0977044Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:12.1023621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:12.1043907Z Entering 'third_party/psimd' 2024-08-07T17:47:12.1090953Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-07T17:47:12.1109248Z Entering 'third_party/pthreadpool' 2024-08-07T17:47:12.1157160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-07T17:47:12.1175115Z Entering 'third_party/pybind11' 2024-08-07T17:47:12.1222418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:47:12.1241758Z Entering 'third_party/python-peachpy' 2024-08-07T17:47:12.1288487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-07T17:47:12.1306494Z Entering 'third_party/sleef' 2024-08-07T17:47:12.1353820Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-07T17:47:12.1372669Z Entering 'third_party/tensorpipe' 2024-08-07T17:47:12.1421159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-07T17:47:12.1439428Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:12.1485579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-07T17:47:12.1503706Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:12.1551520Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-07T17:47:12.1569618Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:12.1615996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-07T17:47:12.1633881Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:12.1682184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-07T17:47:12.1699176Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:12.1747209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-07T17:47:12.2811547Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-07T17:47:12.3121346Z Entering 'android/libs/fbjni' 2024-08-07T17:47:12.3162410Z Entering 'third_party/FP16' 2024-08-07T17:47:12.3205356Z Entering 'third_party/FXdiv' 2024-08-07T17:47:12.3246973Z Entering 'third_party/NNPACK' 2024-08-07T17:47:12.3288835Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:47:12.3330447Z Entering 'third_party/XNNPACK' 2024-08-07T17:47:12.3389233Z Entering 'third_party/benchmark' 2024-08-07T17:47:12.3430943Z Entering 'third_party/cpp-httplib' 2024-08-07T17:47:12.3471745Z Entering 'third_party/cpuinfo' 2024-08-07T17:47:12.3513220Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:47:12.3554846Z Entering 'third_party/cutlass' 2024-08-07T17:47:12.3604543Z Entering 'third_party/eigen' 2024-08-07T17:47:12.3648261Z Entering 'third_party/fbgemm' 2024-08-07T17:47:12.3691810Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:47:12.3733284Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:47:12.3774784Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:47:12.3823127Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:47:12.3865462Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:47:12.3906445Z Entering 'third_party/flatbuffers' 2024-08-07T17:47:12.3950786Z Entering 'third_party/fmt' 2024-08-07T17:47:12.3991726Z Entering 'third_party/foxi' 2024-08-07T17:47:12.4033336Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:47:12.4074932Z Entering 'third_party/gloo' 2024-08-07T17:47:12.4117283Z Entering 'third_party/googletest' 2024-08-07T17:47:12.4158645Z Entering 'third_party/ideep' 2024-08-07T17:47:12.4198730Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:47:12.4247357Z Entering 'third_party/ittapi' 2024-08-07T17:47:12.4289386Z Entering 'third_party/kineto' 2024-08-07T17:47:12.4330041Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:47:12.4372650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:47:12.4415301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:47:12.4457264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:47:12.4497647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:47:12.4538118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:47:12.4582203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:47:12.4622540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:47:12.4663646Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:47:12.4705117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:47:12.4747322Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:47:12.4787692Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:47:12.4829877Z Entering 'third_party/mimalloc' 2024-08-07T17:47:12.4871040Z Entering 'third_party/nccl/nccl' 2024-08-07T17:47:12.4913582Z Entering 'third_party/nlohmann' 2024-08-07T17:47:12.4957122Z Entering 'third_party/onnx' 2024-08-07T17:47:12.5014920Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:47:12.5057695Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:47:12.5101948Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:47:12.5146133Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:47:12.5186555Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:47:12.5227228Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:47:12.5267906Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:47:12.5310541Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:47:12.5350630Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:47:12.5391915Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:47:12.5431205Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:12.5475019Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:12.5518444Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:47:12.5581471Z Entering 'third_party/pocketfft' 2024-08-07T17:47:12.5622697Z Entering 'third_party/protobuf' 2024-08-07T17:47:12.5668680Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:12.5708969Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:12.5751384Z Entering 'third_party/psimd' 2024-08-07T17:47:12.5792806Z Entering 'third_party/pthreadpool' 2024-08-07T17:47:12.5836248Z Entering 'third_party/pybind11' 2024-08-07T17:47:12.5878371Z Entering 'third_party/python-peachpy' 2024-08-07T17:47:12.5919605Z Entering 'third_party/sleef' 2024-08-07T17:47:12.5961486Z Entering 'third_party/tensorpipe' 2024-08-07T17:47:12.6002441Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:12.6043459Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:12.6083859Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:12.6125579Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:12.6165142Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:12.6223271Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-07T17:47:12.6532584Z Entering 'android/libs/fbjni' 2024-08-07T17:47:12.6573837Z Entering 'third_party/FP16' 2024-08-07T17:47:12.6616729Z Entering 'third_party/FXdiv' 2024-08-07T17:47:12.6658498Z Entering 'third_party/NNPACK' 2024-08-07T17:47:12.6700298Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T17:47:12.6742067Z Entering 'third_party/XNNPACK' 2024-08-07T17:47:12.6801395Z Entering 'third_party/benchmark' 2024-08-07T17:47:12.6842711Z Entering 'third_party/cpp-httplib' 2024-08-07T17:47:12.6883675Z Entering 'third_party/cpuinfo' 2024-08-07T17:47:12.6926342Z Entering 'third_party/cudnn_frontend' 2024-08-07T17:47:12.6967513Z Entering 'third_party/cutlass' 2024-08-07T17:47:12.7016611Z Entering 'third_party/eigen' 2024-08-07T17:47:12.7059754Z Entering 'third_party/fbgemm' 2024-08-07T17:47:12.7101499Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T17:47:12.7141990Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T17:47:12.7183629Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T17:47:12.7230151Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T17:47:12.7270493Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T17:47:12.7311420Z Entering 'third_party/flatbuffers' 2024-08-07T17:47:12.7355254Z Entering 'third_party/fmt' 2024-08-07T17:47:12.7395736Z Entering 'third_party/foxi' 2024-08-07T17:47:12.7436447Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T17:47:12.7479386Z Entering 'third_party/gloo' 2024-08-07T17:47:12.7520507Z Entering 'third_party/googletest' 2024-08-07T17:47:12.7561565Z Entering 'third_party/ideep' 2024-08-07T17:47:12.7601214Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T17:47:12.7649020Z Entering 'third_party/ittapi' 2024-08-07T17:47:12.7690184Z Entering 'third_party/kineto' 2024-08-07T17:47:12.7730571Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T17:47:12.7771324Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T17:47:12.7814054Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T17:47:12.7856154Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T17:47:12.7897226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T17:47:12.7938058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T17:47:12.7981898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T17:47:12.8023058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T17:47:12.8064632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T17:47:12.8107014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T17:47:12.8149198Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T17:47:12.8188972Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T17:47:12.8230952Z Entering 'third_party/mimalloc' 2024-08-07T17:47:12.8272182Z Entering 'third_party/nccl/nccl' 2024-08-07T17:47:12.8313603Z Entering 'third_party/nlohmann' 2024-08-07T17:47:12.8355366Z Entering 'third_party/onnx' 2024-08-07T17:47:12.8413130Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T17:47:12.8455598Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T17:47:12.8499648Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T17:47:12.8542873Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T17:47:12.8582916Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T17:47:12.8623363Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T17:47:12.8663149Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T17:47:12.8704814Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T17:47:12.8744646Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T17:47:12.8785126Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T17:47:12.8824347Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T17:47:12.8867484Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T17:47:12.8910209Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T17:47:12.8972843Z Entering 'third_party/pocketfft' 2024-08-07T17:47:12.9014811Z Entering 'third_party/protobuf' 2024-08-07T17:47:12.9059152Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T17:47:12.9100555Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T17:47:12.9142582Z Entering 'third_party/psimd' 2024-08-07T17:47:12.9183347Z Entering 'third_party/pthreadpool' 2024-08-07T17:47:12.9224552Z Entering 'third_party/pybind11' 2024-08-07T17:47:12.9265838Z Entering 'third_party/python-peachpy' 2024-08-07T17:47:12.9306924Z Entering 'third_party/sleef' 2024-08-07T17:47:12.9348568Z Entering 'third_party/tensorpipe' 2024-08-07T17:47:12.9389714Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T17:47:12.9430808Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T17:47:12.9470982Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T17:47:12.9511538Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T17:47:12.9550079Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T17:47:12.9603505Z ##[endgroup] 2024-08-07T17:47:12.9644303Z [command]/usr/bin/git log -1 --format='%H' 2024-08-07T17:47:12.9670284Z '016588f53c6904b840aa56aa86f95460b4d9c996' 2024-08-07T17:47:12.9856722Z Prepare all required actions 2024-08-07T17:47:12.9857590Z Getting action download info 2024-08-07T17:47:13.1181065Z ##[group]Run ./.github/actions/setup-linux 2024-08-07T17:47:13.1181492Z env: 2024-08-07T17:47:13.1181769Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:13.1182117Z ##[endgroup] 2024-08-07T17:47:13.1263546Z ##[group]Run set -euo pipefail 2024-08-07T17:47:13.1264010Z set -euo pipefail 2024-08-07T17:47:13.1264387Z function get_ec2_metadata() { 2024-08-07T17:47:13.1264886Z  # Pulled from instance metadata endpoint for EC2 2024-08-07T17:47:13.1265677Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-07T17:47:13.1266387Z  category=$1 2024-08-07T17:47:13.1266845Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-07T17:47:13.1267409Z  runner_name_str=i-095fa8d57edae1a62 2024-08-07T17:47:13.1267925Z  if [[ -f /.inarc ]]; then 2024-08-07T17:47:13.1268387Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-07T17:47:13.1268891Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-07T17:47:13.1269484Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-07T17:47:13.1270026Z  else 2024-08-07T17:47:13.1270456Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-07T17:47:13.1270981Z  fi 2024-08-07T17:47:13.1271272Z } 2024-08-07T17:47:13.1271607Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-07T17:47:13.1272159Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-07T17:47:13.1272766Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-07T17:47:13.1273539Z echo "system info $(uname -a)" 2024-08-07T17:47:13.1281752Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:13.1282242Z env: 2024-08-07T17:47:13.1282531Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:13.1282895Z ##[endgroup] 2024-08-07T17:47:13.1380189Z ami-id: ami-0ce0c36d7a00b20e2 2024-08-07T17:47:13.1445390Z instance-id: i-095fa8d57edae1a62 2024-08-07T17:47:13.1507366Z instance-type: g3.8xlarge 2024-08-07T17:47:13.1516165Z system info Linux ip-10-0-53-8.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-07T17:47:13.1547416Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:47:13.1548234Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-07T17:47:13.1557340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:13.1557830Z env: 2024-08-07T17:47:13.1558126Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:13.1558460Z ##[endgroup] 2024-08-07T17:47:13.1638625Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-07T17:47:13.1639209Z if systemctl is-active --quiet docker; then 2024-08-07T17:47:13.1639705Z  echo "Docker daemon is running..."; 2024-08-07T17:47:13.1640134Z else 2024-08-07T17:47:13.1640590Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-07T17:47:13.1641107Z fi 2024-08-07T17:47:13.1648197Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:13.1648686Z env: 2024-08-07T17:47:13.1648962Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:13.1649317Z ##[endgroup] 2024-08-07T17:47:13.1694764Z Docker daemon is running... 2024-08-07T17:47:13.1751004Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:47:13.1751518Z with: 2024-08-07T17:47:13.1751804Z shell: bash 2024-08-07T17:47:13.1752101Z timeout_minutes: 5 2024-08-07T17:47:13.1752406Z max_attempts: 3 2024-08-07T17:47:13.1752715Z retry_wait_seconds: 30 2024-08-07T17:47:13.1754482Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-08-07T17:47:13.1755916Z polling_interval_seconds: 1 2024-08-07T17:47:13.1756276Z warning_on_retry: true 2024-08-07T17:47:13.1756620Z continue_on_error: false 2024-08-07T17:47:13.1756942Z env: 2024-08-07T17:47:13.1757209Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:13.1757548Z AWS_RETRY_MODE: standard 2024-08-07T17:47:13.1757880Z AWS_MAX_ATTEMPTS: 5 2024-08-07T17:47:13.1758198Z AWS_DEFAULT_REGION: us-east-1 2024-08-07T17:47:13.1758547Z ##[endgroup] 2024-08-07T17:47:14.1785119Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:47:14.1785915Z Configure a credential helper to remove this warning. See 2024-08-07T17:47:14.1786625Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:47:14.1787131Z 2024-08-07T17:47:14.1787308Z Login Succeeded 2024-08-07T17:47:14.2469186Z Command completed after 1 attempt(s). 2024-08-07T17:47:14.2529620Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:47:14.2530283Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:47:14.2530849Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-07T17:47:14.2538981Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:14.2539472Z env: 2024-08-07T17:47:14.2539744Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.2540098Z ##[endgroup] 2024-08-07T17:47:14.2625576Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:47:14.2626295Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T17:47:14.2626819Z # shellcheck disable=SC2046 2024-08-07T17:47:14.2627254Z docker stop $(docker ps -q) || true 2024-08-07T17:47:14.2627716Z # Prune all of the docker images 2024-08-07T17:47:14.2628131Z docker system prune -af 2024-08-07T17:47:14.2636265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:14.2636760Z env: 2024-08-07T17:47:14.2637039Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.2637392Z ##[endgroup] 2024-08-07T17:47:14.2968398Z "docker stop" requires at least 1 argument. 2024-08-07T17:47:14.2968872Z See 'docker stop --help'. 2024-08-07T17:47:14.2969127Z 2024-08-07T17:47:14.2969345Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-07T17:47:14.2969693Z 2024-08-07T17:47:14.2969872Z Stop one or more running containers 2024-08-07T17:47:14.3154662Z Total reclaimed space: 0B 2024-08-07T17:47:14.3196420Z ##[group]Run set +e 2024-08-07T17:47:14.3196802Z set +e 2024-08-07T17:47:14.3197165Z set -x 2024-08-07T17:47:14.3197456Z  2024-08-07T17:47:14.3197795Z PT_DOMAIN=download.pytorch.org 2024-08-07T17:47:14.3198550Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-07T17:47:14.3199485Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-07T17:47:14.3200156Z # one is returned at random 2024-08-07T17:47:14.3200668Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-07T17:47:14.3201141Z  2024-08-07T17:47:14.3201462Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:47:14.3202016Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-07T17:47:14.3202664Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-07T17:47:14.3203187Z  2024-08-07T17:47:14.3203509Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-07T17:47:14.3204020Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-07T17:47:14.3204501Z  exit 1 2024-08-07T17:47:14.3204809Z  fi 2024-08-07T17:47:14.3205111Z fi 2024-08-07T17:47:14.3205602Z  2024-08-07T17:47:14.3206157Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-07T17:47:14.3206642Z  # Clean up any old records first 2024-08-07T17:47:14.3207117Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-07T17:47:14.3207532Z fi 2024-08-07T17:47:14.3207825Z  2024-08-07T17:47:14.3208240Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-07T17:47:14.3208743Z cat /etc/hosts 2024-08-07T17:47:14.3216200Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:14.3216697Z env: 2024-08-07T17:47:14.3216999Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.3217338Z ##[endgroup] 2024-08-07T17:47:14.3240573Z + PT_DOMAIN=download.pytorch.org 2024-08-07T17:47:14.3245632Z ++ dig -4 +short download.pytorch.org 2024-08-07T17:47:14.3246202Z ++ tail -n1 2024-08-07T17:47:14.3375426Z + RESOLVED_IP=18.160.10.36 2024-08-07T17:47:14.3375802Z + '[' -z 18.160.10.36 ']' 2024-08-07T17:47:14.3376188Z + grep -r download.pytorch.org /etc/hosts 2024-08-07T17:47:14.3385044Z 18.160.10.36 download.pytorch.org 2024-08-07T17:47:14.3386142Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-07T17:47:14.3510175Z + echo '18.160.10.36 download.pytorch.org' 2024-08-07T17:47:14.3511228Z + sudo tee -a /etc/hosts 2024-08-07T17:47:14.3602566Z 18.160.10.36 download.pytorch.org 2024-08-07T17:47:14.3619080Z + cat /etc/hosts 2024-08-07T17:47:14.3625102Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-07T17:47:14.3630226Z ::1 localhost6 localhost6.localdomain6 2024-08-07T17:47:14.3630730Z 18.160.10.36 download.pytorch.org 2024-08-07T17:47:14.3786572Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-07T17:47:14.3787200Z with: 2024-08-07T17:47:14.3788107Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3789108Z docker-build-dir: .ci/docker 2024-08-07T17:47:14.3789510Z working-directory: . 2024-08-07T17:47:14.3789968Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.3790461Z force-push: false 2024-08-07T17:47:14.3790771Z env: 2024-08-07T17:47:14.3791062Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.3791395Z ##[endgroup] 2024-08-07T17:47:14.3822737Z ##[group]Run set -ex 2024-08-07T17:47:14.3823132Z set -ex 2024-08-07T17:47:14.3823437Z  2024-08-07T17:47:14.3823946Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-07T17:47:14.3824827Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-07T17:47:14.3825569Z # job could then download the pre-built image as usual 2024-08-07T17:47:14.3826219Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-07T17:47:14.3826814Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3827406Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3827913Z  2024-08-07T17:47:14.3828375Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-07T17:47:14.3828938Z  exit 0 2024-08-07T17:47:14.3829229Z else 2024-08-07T17:47:14.3829581Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3830003Z fi 2024-08-07T17:47:14.3830273Z  2024-08-07T17:47:14.3830711Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-07T17:47:14.3831464Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-07T17:47:14.3832123Z  # use it as it is, but first let's extract the tag 2024-08-07T17:47:14.3832730Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-07T17:47:14.3833686Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3834487Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3835014Z else 2024-08-07T17:47:14.3835435Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-07T17:47:14.3836010Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3836805Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3837504Z fi 2024-08-07T17:47:14.3844905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:14.3845383Z env: 2024-08-07T17:47:14.3845678Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.3846017Z REPO_NAME: pytorch 2024-08-07T17:47:14.3846955Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3847961Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:47:14.3848452Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.3848939Z ##[endgroup] 2024-08-07T17:47:14.3871900Z + [[ ! -d .ci/docker ]] 2024-08-07T17:47:14.3872275Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-07T17:47:14.3872647Z + echo skip=false 2024-08-07T17:47:14.3874141Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-07T17:47:14.3879209Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3880199Z ++ awk -F '[:,]' '{print $2}' 2024-08-07T17:47:14.3890852Z + DOCKER_TAG=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3891406Z + echo docker-tag=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3892760Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3930674Z ##[group]Run set +e 2024-08-07T17:47:14.3931088Z set +e 2024-08-07T17:47:14.3931459Z set -x 2024-08-07T17:47:14.3931750Z  2024-08-07T17:47:14.3932042Z login() { 2024-08-07T17:47:14.3932646Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:47:14.3933282Z } 2024-08-07T17:47:14.3933572Z  2024-08-07T17:47:14.3933873Z retry () { 2024-08-07T17:47:14.3934223Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:47:14.3934641Z } 2024-08-07T17:47:14.3934925Z  2024-08-07T17:47:14.3935224Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:47:14.3935630Z  2024-08-07T17:47:14.3936066Z # Check if image already exists, if it does then skip building it 2024-08-07T17:47:14.3936701Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-07T17:47:14.3937178Z  exit 0 2024-08-07T17:47:14.3937482Z fi 2024-08-07T17:47:14.3937751Z  2024-08-07T17:47:14.3938206Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-07T17:47:14.3938925Z # be empty. The default action would be to continue rebuild the image 2024-08-07T17:47:14.3939561Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-07T17:47:14.3940151Z  # if we're on the base branch then use the parent commit 2024-08-07T17:47:14.3940674Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-07T17:47:14.3941074Z else 2024-08-07T17:47:14.3941507Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-07T17:47:14.3942112Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-07T17:47:14.3942586Z fi 2024-08-07T17:47:14.3943059Z  2024-08-07T17:47:14.3943382Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-07T17:47:14.3943850Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3944271Z  2024-08-07T17:47:14.3944863Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-07T17:47:14.3945551Z  exit 0 2024-08-07T17:47:14.3945841Z fi 2024-08-07T17:47:14.3946135Z  2024-08-07T17:47:14.3946541Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-07T17:47:14.3947396Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-07T17:47:14.3948136Z  exit 1 2024-08-07T17:47:14.3948448Z fi 2024-08-07T17:47:14.3948723Z  2024-08-07T17:47:14.3949192Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-07T17:47:14.3950024Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-07T17:47:14.3950745Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-07T17:47:14.3951596Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-07T17:47:14.3952542Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-07T17:47:14.3953135Z fi 2024-08-07T17:47:14.3953701Z  2024-08-07T17:47:14.3954209Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-07T17:47:14.3961392Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:14.3961889Z env: 2024-08-07T17:47:14.3962189Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:14.3962541Z DOCKER_BUILD_DIR: .ci/docker 2024-08-07T17:47:14.3962994Z BASE_REVISION: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:47:14.3964059Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3965107Z DOCKER_TAG: 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:14.3965673Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.3966177Z ##[endgroup] 2024-08-07T17:47:14.3989037Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.3989604Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.3991459Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:47:14.3993635Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:14.8754002Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:47:14.8754775Z Configure a credential helper to remove this warning. See 2024-08-07T17:47:14.8755499Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:47:14.8756028Z 2024-08-07T17:47:14.8756236Z Login Succeeded 2024-08-07T17:47:14.8769396Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.1059325Z { 2024-08-07T17:47:15.1059631Z "schemaVersion": 2, 2024-08-07T17:47:15.1060211Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-07T17:47:15.1060787Z "config": { 2024-08-07T17:47:15.1061242Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-07T17:47:15.1061773Z "size": 47454, 2024-08-07T17:47:15.1062302Z "digest": "sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129" 2024-08-07T17:47:15.1062880Z }, 2024-08-07T17:47:15.1063163Z "layers": [ 2024-08-07T17:47:15.1063461Z { 2024-08-07T17:47:15.1063884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1064735Z "size": 28580681, 2024-08-07T17:47:15.1065270Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-07T17:47:15.1065849Z }, 2024-08-07T17:47:15.1066120Z { 2024-08-07T17:47:15.1066545Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1067063Z "size": 7943353, 2024-08-07T17:47:15.1067595Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-08-07T17:47:15.1068186Z }, 2024-08-07T17:47:15.1068434Z { 2024-08-07T17:47:15.1068872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1069404Z "size": 56226389, 2024-08-07T17:47:15.1069918Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-08-07T17:47:15.1070504Z }, 2024-08-07T17:47:15.1070762Z { 2024-08-07T17:47:15.1071171Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1071699Z "size": 187, 2024-08-07T17:47:15.1072382Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-08-07T17:47:15.1073025Z }, 2024-08-07T17:47:15.1074036Z { 2024-08-07T17:47:15.1074714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1075409Z "size": 6885, 2024-08-07T17:47:15.1076009Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-08-07T17:47:15.1076718Z }, 2024-08-07T17:47:15.1077134Z { 2024-08-07T17:47:15.1077622Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1078443Z "size": 1383662475, 2024-08-07T17:47:15.1079146Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-08-07T17:47:15.1079786Z }, 2024-08-07T17:47:15.1106303Z { 2024-08-07T17:47:15.1106796Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1107350Z "size": 62282, 2024-08-07T17:47:15.1107968Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-08-07T17:47:15.1108667Z }, 2024-08-07T17:47:15.1108939Z { 2024-08-07T17:47:15.1109374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1109891Z "size": 1684, 2024-08-07T17:47:15.1110425Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-08-07T17:47:15.1111022Z }, 2024-08-07T17:47:15.1111278Z { 2024-08-07T17:47:15.1111707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1112241Z "size": 1522, 2024-08-07T17:47:15.1112758Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-08-07T17:47:15.1113718Z }, 2024-08-07T17:47:15.1113987Z { 2024-08-07T17:47:15.1114399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1114936Z "size": 2445884909, 2024-08-07T17:47:15.1115473Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-08-07T17:47:15.1116046Z }, 2024-08-07T17:47:15.1116314Z { 2024-08-07T17:47:15.1116716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1117246Z "size": 85882, 2024-08-07T17:47:15.1117774Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-08-07T17:47:15.1118354Z }, 2024-08-07T17:47:15.1118599Z { 2024-08-07T17:47:15.1119023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1119532Z "size": 1820, 2024-08-07T17:47:15.1120062Z "digest": "sha256:ceceb076946ee95300822daccb5d3e4e45e6857d34996ef89305d9a078410f8e" 2024-08-07T17:47:15.1120653Z }, 2024-08-07T17:47:15.1120895Z { 2024-08-07T17:47:15.1121320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1121853Z "size": 654860881, 2024-08-07T17:47:15.1122374Z "digest": "sha256:889eb8142c5187639e02cdc07adc75024843fd85a1f5c77d24c1a1123f5246bb" 2024-08-07T17:47:15.1123217Z }, 2024-08-07T17:47:15.1123480Z { 2024-08-07T17:47:15.1123891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1124507Z + exit 0 2024-08-07T17:47:15.1124791Z "size": 542, 2024-08-07T17:47:15.1125306Z "digest": "sha256:af8081143f008c5cb1bd61f6d92fde22553cfc007b6a77c2fd2aaf6a9f0088a9" 2024-08-07T17:47:15.1125896Z }, 2024-08-07T17:47:15.1126154Z { 2024-08-07T17:47:15.1126564Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1127094Z "size": 1284, 2024-08-07T17:47:15.1127623Z "digest": "sha256:cc64a79090ad4059e2f3dbc1949b010b2733ab614bc9518ff6256ce2bed52115" 2024-08-07T17:47:15.1128197Z }, 2024-08-07T17:47:15.1128461Z { 2024-08-07T17:47:15.1128887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1129397Z "size": 483, 2024-08-07T17:47:15.1129906Z "digest": "sha256:ad26542e0d01609840496a12691483f2c50eb8e23002a205becb5b695f9159fb" 2024-08-07T17:47:15.1130500Z }, 2024-08-07T17:47:15.1130745Z { 2024-08-07T17:47:15.1131174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1131708Z "size": 91711921, 2024-08-07T17:47:15.1132226Z "digest": "sha256:7b42d8ae23671f21f6523a7e71cc50d5f87c4e993027a9bd924985cda6b57de9" 2024-08-07T17:47:15.1132814Z }, 2024-08-07T17:47:15.1133079Z { 2024-08-07T17:47:15.1133491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1134025Z "size": 3229, 2024-08-07T17:47:15.1134693Z "digest": "sha256:6d46e3b97d9fd2bd0e7ceec44e4884756b26c594a1c1c51d1426ddd13f749db5" 2024-08-07T17:47:15.1135298Z }, 2024-08-07T17:47:15.1135561Z { 2024-08-07T17:47:15.1135988Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1136506Z "size": 1907, 2024-08-07T17:47:15.1137030Z "digest": "sha256:91f9dddffbee46981d9e74382704c1b53c87f1808350cf1dd518687805d3aa59" 2024-08-07T17:47:15.1137618Z }, 2024-08-07T17:47:15.1137874Z { 2024-08-07T17:47:15.1138300Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1138814Z "size": 699, 2024-08-07T17:47:15.1139344Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:47:15.1139941Z }, 2024-08-07T17:47:15.1140189Z { 2024-08-07T17:47:15.1140611Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1141130Z "size": 2813413845, 2024-08-07T17:47:15.1141677Z "digest": "sha256:eeedca475f8cafec096dfa0abc5c6f08938eb47c5d9d96fb001fd7f13a2531ca" 2024-08-07T17:47:15.1142294Z }, 2024-08-07T17:47:15.1142558Z { 2024-08-07T17:47:15.1142967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1143496Z "size": 378, 2024-08-07T17:47:15.1144020Z "digest": "sha256:202c71391ade27762d5e9cafde793db67b4d5e4d6869debe2643d5092101c461" 2024-08-07T17:47:15.1144591Z }, 2024-08-07T17:47:15.1144852Z { 2024-08-07T17:47:15.1145283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1145795Z "size": 12893, 2024-08-07T17:47:15.1146314Z "digest": "sha256:5af882ef27a2db3ee5e89c8730de161c0718b951f3f8477e63fe21517c9167f7" 2024-08-07T17:47:15.1146903Z }, 2024-08-07T17:47:15.1147149Z { 2024-08-07T17:47:15.1147576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1148106Z "size": 802, 2024-08-07T17:47:15.1148615Z "digest": "sha256:bac001ac2fee5df3367ff9332bc1fcd706a8de3ad05a1885903f338197619520" 2024-08-07T17:47:15.1149208Z }, 2024-08-07T17:47:15.1149470Z { 2024-08-07T17:47:15.1149889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1150426Z "size": 106, 2024-08-07T17:47:15.1150932Z "digest": "sha256:05896f1d52f7a8e2c4a53777e5575da3dc8fca8a786d14075b5bb204bcb4cf04" 2024-08-07T17:47:15.1151504Z }, 2024-08-07T17:47:15.1151765Z { 2024-08-07T17:47:15.1152187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1152864Z "size": 502, 2024-08-07T17:47:15.1153760Z "digest": "sha256:49a0e418a175ac12f1b4b503e80e22679bc321e6163ed5b203af848204f0606a" 2024-08-07T17:47:15.1154415Z }, 2024-08-07T17:47:15.1154664Z { 2024-08-07T17:47:15.1155091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1155607Z "size": 121477364, 2024-08-07T17:47:15.1156137Z "digest": "sha256:b5632121ff7a146647683888294bcca31252dd90ca681099dfb81d75c4710d52" 2024-08-07T17:47:15.1156717Z }, 2024-08-07T17:47:15.1156963Z { 2024-08-07T17:47:15.1157398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1157931Z "size": 109, 2024-08-07T17:47:15.1158427Z "digest": "sha256:deceb46806b535404e025b9b532009eb211e79f85021171997af825253327d1a" 2024-08-07T17:47:15.1159009Z }, 2024-08-07T17:47:15.1159275Z { 2024-08-07T17:47:15.1159685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1160229Z "size": 487, 2024-08-07T17:47:15.1160756Z "digest": "sha256:a2630e5fe166ae8ff146e4c14db0bdc1299ed2b6120108351c961adcd2375288" 2024-08-07T17:47:15.1161335Z }, 2024-08-07T17:47:15.1161603Z { 2024-08-07T17:47:15.1162034Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1162546Z "size": 297, 2024-08-07T17:47:15.1163066Z "digest": "sha256:603261fb379fff6cf0eb63de1787e7c98932f81e44e6af3f1645c9a977e0ca8c" 2024-08-07T17:47:15.1163653Z }, 2024-08-07T17:47:15.1163894Z { 2024-08-07T17:47:15.1164454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1165012Z "size": 103, 2024-08-07T17:47:15.1165510Z "digest": "sha256:d8ac71337d62da7c4624f267380da27d834fcf174b873c8c53a963e6cc7193a4" 2024-08-07T17:47:15.1166096Z }, 2024-08-07T17:47:15.1166356Z { 2024-08-07T17:47:15.1166766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1167293Z "size": 1470, 2024-08-07T17:47:15.1167837Z "digest": "sha256:2c2e35b979f5fc0ecccef6efc157a97f33db01a77958fa33855dc367544a5c46" 2024-08-07T17:47:15.1168428Z }, 2024-08-07T17:47:15.1168692Z { 2024-08-07T17:47:15.1169116Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1169630Z "size": 424146101, 2024-08-07T17:47:15.1170165Z "digest": "sha256:e559222c0ec62d839f78e4215ae553de028be354b7031e7393d1c6be6e91fd70" 2024-08-07T17:47:15.1170748Z }, 2024-08-07T17:47:15.1170994Z { 2024-08-07T17:47:15.1171420Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1171954Z "size": 159, 2024-08-07T17:47:15.1172465Z "digest": "sha256:c4dcb7e9d189eca2e414c757b5cf6d158055b29b0e71dbd082a5fb3585dba665" 2024-08-07T17:47:15.1173062Z }, 2024-08-07T17:47:15.1173306Z { 2024-08-07T17:47:15.1173727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1174255Z "size": 563, 2024-08-07T17:47:15.1174755Z "digest": "sha256:2b9731d295d60278e361d491fc274e5d558a89ec5adeb9fdf342d7a8cc17bfec" 2024-08-07T17:47:15.1175354Z }, 2024-08-07T17:47:15.1175618Z { 2024-08-07T17:47:15.1176027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1176569Z "size": 35807525, 2024-08-07T17:47:15.1177099Z "digest": "sha256:19da43312ebac2831028913760327e53d9812e80ab6b806cd7a920c549a3b554" 2024-08-07T17:47:15.1177655Z }, 2024-08-07T17:47:15.1177919Z { 2024-08-07T17:47:15.1178349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1178865Z "size": 104, 2024-08-07T17:47:15.1179393Z "digest": "sha256:b846263d066405189d21552b7288e3efbc47e6cd2f236aa39f39d88ca576a33a" 2024-08-07T17:47:15.1179980Z }, 2024-08-07T17:47:15.1180221Z { 2024-08-07T17:47:15.1180649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1181188Z "size": 424, 2024-08-07T17:47:15.1181701Z "digest": "sha256:3886e8f57dc887a87bb710fbda080ef4fd521f1ed2e560a895b000c05f6e9b0a" 2024-08-07T17:47:15.1182452Z }, 2024-08-07T17:47:15.1182718Z { 2024-08-07T17:47:15.1183128Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1183662Z "size": 20262048, 2024-08-07T17:47:15.1184201Z "digest": "sha256:58e13e7b6d7b78400d84805c843f08fb8f1d6583bccdb3373c8122b317e4d15a" 2024-08-07T17:47:15.1184771Z }, 2024-08-07T17:47:15.1185042Z { 2024-08-07T17:47:15.1185474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1185983Z "size": 438, 2024-08-07T17:47:15.1186510Z "digest": "sha256:0eb80a0ac0006a45aa6b4581263cc1613ddf9545bd4a1d5a544310efe898ad81" 2024-08-07T17:47:15.1187101Z }, 2024-08-07T17:47:15.1187343Z { 2024-08-07T17:47:15.1187766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1188294Z "size": 699, 2024-08-07T17:47:15.1188801Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:47:15.1189404Z }, 2024-08-07T17:47:15.1189648Z { 2024-08-07T17:47:15.1190073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1190602Z "size": 141, 2024-08-07T17:47:15.1191107Z "digest": "sha256:a9f11b2b3622f5530c5cfefe973e96e028323fab1ff02e7a6337eb86fadebeb5" 2024-08-07T17:47:15.1191699Z }, 2024-08-07T17:47:15.1191962Z { 2024-08-07T17:47:15.1192370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1192904Z "size": 135, 2024-08-07T17:47:15.1193890Z "digest": "sha256:d7e94279f8b2fe549e3f23637275d9dca8c831563ce84f1d44117eb297ddc871" 2024-08-07T17:47:15.1194488Z }, 2024-08-07T17:47:15.1194750Z { 2024-08-07T17:47:15.1195174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1195686Z "size": 32, 2024-08-07T17:47:15.1196212Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1196808Z }, 2024-08-07T17:47:15.1197063Z { 2024-08-07T17:47:15.1197546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1198080Z "size": 190, 2024-08-07T17:47:15.1198589Z "digest": "sha256:0a3560fcd26c387cb3ff0638718fa9da02d3b08ad75b23c434b06d6a6d21cb55" 2024-08-07T17:47:15.1199185Z }, 2024-08-07T17:47:15.1199453Z { 2024-08-07T17:47:15.1199866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1200400Z "size": 562, 2024-08-07T17:47:15.1200929Z "digest": "sha256:adf9d181cfbca5b0544f23797995d4ea6500cc4a635db9f7e18954c9044e9f4a" 2024-08-07T17:47:15.1201506Z }, 2024-08-07T17:47:15.1201778Z { 2024-08-07T17:47:15.1202209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1202720Z "size": 43163050, 2024-08-07T17:47:15.1203249Z "digest": "sha256:bc18dd8315979255e24cf91e83757510baeb22f29a64d7579b02aca642215b58" 2024-08-07T17:47:15.1203836Z }, 2024-08-07T17:47:15.1204083Z { 2024-08-07T17:47:15.1204511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1205051Z "size": 106, 2024-08-07T17:47:15.1205559Z "digest": "sha256:d62f2a6086195c833ef952a43eed18216c6cde79b07f8deeb4fb3aeea648f868" 2024-08-07T17:47:15.1206158Z }, 2024-08-07T17:47:15.1206423Z { 2024-08-07T17:47:15.1206836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1207367Z "size": 1210, 2024-08-07T17:47:15.1207882Z "digest": "sha256:719a1c2abcfd7e4113aa2f03adeb54350a5e381c720688ea53bc0e18401ad33d" 2024-08-07T17:47:15.1208473Z }, 2024-08-07T17:47:15.1208734Z { 2024-08-07T17:47:15.1209147Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1209722Z "size": 699, 2024-08-07T17:47:15.1210255Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:47:15.1210832Z }, 2024-08-07T17:47:15.1211089Z { 2024-08-07T17:47:15.1211515Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1212186Z "size": 138, 2024-08-07T17:47:15.1212699Z "digest": "sha256:8e1355d516c5e0527c25402c664fc648a8a2b814090e6ee0fdd60f49a8826ac1" 2024-08-07T17:47:15.1213283Z }, 2024-08-07T17:47:15.1213530Z { 2024-08-07T17:47:15.1213952Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1214491Z "size": 117, 2024-08-07T17:47:15.1214988Z "digest": "sha256:701898e66f54fcf3b57d837ab651a876666391d9f9b51b9969ec842a23fd86f0" 2024-08-07T17:47:15.1215569Z }, 2024-08-07T17:47:15.1215834Z { 2024-08-07T17:47:15.1216249Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1216783Z "size": 1916643487, 2024-08-07T17:47:15.1217328Z "digest": "sha256:129b38cfed12360bb7c4d882141b5fb7f7337791383b9f75e4ffed521548f060" 2024-08-07T17:47:15.1217898Z }, 2024-08-07T17:47:15.1218159Z { 2024-08-07T17:47:15.1218586Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1219109Z "size": 172, 2024-08-07T17:47:15.1219618Z "digest": "sha256:1751651a072f518b34289de9dfa631c4e82c7958a5331061379684f9b9b11720" 2024-08-07T17:47:15.1220193Z }, 2024-08-07T17:47:15.1220440Z { 2024-08-07T17:47:15.1220864Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1221397Z "size": 905, 2024-08-07T17:47:15.1221918Z "digest": "sha256:fc956bf4e5b49119cab7cc7bda867c61bed76c76c1e113dfa7c7b184251f8173" 2024-08-07T17:47:15.1222514Z }, 2024-08-07T17:47:15.1222776Z { 2024-08-07T17:47:15.1223184Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1223804Z "size": 699, 2024-08-07T17:47:15.1224359Z "digest": "sha256:03fb78f207e3cbcfedfc7619a1d60fa576d3293d508a7e97dc09a5e7a9c65ab0" 2024-08-07T17:47:15.1224941Z }, 2024-08-07T17:47:15.1225201Z { 2024-08-07T17:47:15.1225631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1226145Z "size": 134, 2024-08-07T17:47:15.1226670Z "digest": "sha256:874491333fb5b81d9cbbd71520163a6baf07e6f011dc148b1634b94e301972b8" 2024-08-07T17:47:15.1227252Z }, 2024-08-07T17:47:15.1227499Z { 2024-08-07T17:47:15.1227922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1228434Z "size": 32, 2024-08-07T17:47:15.1228958Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1229546Z }, 2024-08-07T17:47:15.1229790Z { 2024-08-07T17:47:15.1230210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1230739Z "size": 156, 2024-08-07T17:47:15.1231266Z "digest": "sha256:365b4a0eec21fb9dfc58baf7b43a229ae3f8b638b17e9b4db5ee1cfaecb82de4" 2024-08-07T17:47:15.1231868Z }, 2024-08-07T17:47:15.1232128Z { 2024-08-07T17:47:15.1232535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1233068Z "size": 1839, 2024-08-07T17:47:15.1233960Z "digest": "sha256:c1d4cac78640fc7d364011da807a363d6b248a2c46ccf99a6e1488c4974c17e3" 2024-08-07T17:47:15.1234545Z }, 2024-08-07T17:47:15.1234808Z { 2024-08-07T17:47:15.1235233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1235749Z "size": 7529788, 2024-08-07T17:47:15.1236279Z "digest": "sha256:29afe2181ca6174db390fd0d9e8c4b8675fb99bb961c93936d4080ae92c8af57" 2024-08-07T17:47:15.1236871Z }, 2024-08-07T17:47:15.1237116Z { 2024-08-07T17:47:15.1237543Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1238075Z "size": 161, 2024-08-07T17:47:15.1238584Z "digest": "sha256:190dccfa687dc1e698f27965d8065a183168d7c66361505f368765456ff9f4bc" 2024-08-07T17:47:15.1239168Z }, 2024-08-07T17:47:15.1239434Z { 2024-08-07T17:47:15.1239845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1240376Z "size": 7942, 2024-08-07T17:47:15.1240907Z "digest": "sha256:b01bea4b6634f47c51ede2b4a95ceade4e61356ca29551168b493ae6d4492f44" 2024-08-07T17:47:15.1241634Z }, 2024-08-07T17:47:15.1241904Z { 2024-08-07T17:47:15.1242332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1242845Z "size": 8066, 2024-08-07T17:47:15.1243376Z "digest": "sha256:f13a9cc5968861df73c96e4ad67eef7f58576eeba8cdbee3d39e42ed3d6fe96d" 2024-08-07T17:47:15.1243967Z }, 2024-08-07T17:47:15.1244212Z { 2024-08-07T17:47:15.1244631Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1245139Z "size": 302, 2024-08-07T17:47:15.1245656Z "digest": "sha256:96d6dcd01b9f6c5557ba405d028442d808dd329f4f93e75f2c3ba0554e703a52" 2024-08-07T17:47:15.1246236Z }, 2024-08-07T17:47:15.1246472Z { 2024-08-07T17:47:15.1246893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1247416Z "size": 7629762, 2024-08-07T17:47:15.1247916Z "digest": "sha256:fe134554f1b8a0509798c1f49e6f973f9b8b7f954eb1fda146f35a37c9c71599" 2024-08-07T17:47:15.1248494Z }, 2024-08-07T17:47:15.1248761Z { 2024-08-07T17:47:15.1249169Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1249688Z "size": 108, 2024-08-07T17:47:15.1250208Z "digest": "sha256:8068d364963fe87857fbb7c3aad42ddee962351a57eddc01b4282be12c69a074" 2024-08-07T17:47:15.1250772Z }, 2024-08-07T17:47:15.1251032Z { 2024-08-07T17:47:15.1251460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1251971Z "size": 54145777, 2024-08-07T17:47:15.1252506Z "digest": "sha256:f307f545c1a5e086f18e23247c890ddd5b135b1cb7157807aeeaef38ceb3b7dc" 2024-08-07T17:47:15.1253204Z }, 2024-08-07T17:47:15.1253472Z { 2024-08-07T17:47:15.1253898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1254423Z "size": 471, 2024-08-07T17:47:15.1254917Z "digest": "sha256:51c0b7eab50584e37e08f303a4bf3ba3141998800971452b2024a8b897858877" 2024-08-07T17:47:15.1255495Z }, 2024-08-07T17:47:15.1255760Z { 2024-08-07T17:47:15.1256182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1256714Z "size": 1373229322, 2024-08-07T17:47:15.1257255Z "digest": "sha256:60aba96f31c695ec1e1433578783bab1257233794a6e8b5cc5c6365fdf68669e" 2024-08-07T17:47:15.1257826Z }, 2024-08-07T17:47:15.1258088Z { 2024-08-07T17:47:15.1258513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1259023Z "size": 106, 2024-08-07T17:47:15.1259547Z "digest": "sha256:46db894a106ccd6c28198b9236ee84ac2973b37ae608caeb380dfaef91f13961" 2024-08-07T17:47:15.1260133Z }, 2024-08-07T17:47:15.1260385Z { 2024-08-07T17:47:15.1260811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1261336Z "size": 557, 2024-08-07T17:47:15.1261825Z "digest": "sha256:0052583728e40920c255902250fe4a615b10ff00a5c0e7ed51a0075121d42fad" 2024-08-07T17:47:15.1262392Z }, 2024-08-07T17:47:15.1262635Z { 2024-08-07T17:47:15.1263063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1263607Z "size": 36532090, 2024-08-07T17:47:15.1264134Z "digest": "sha256:8c6d352a3b51edbb546310cc3d97ded6aa8fd29a7d610e4d13fc54b5828fdd05" 2024-08-07T17:47:15.1264731Z }, 2024-08-07T17:47:15.1264994Z { 2024-08-07T17:47:15.1265408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1265942Z "size": 111, 2024-08-07T17:47:15.1266469Z "digest": "sha256:380c4e16278f4e3bf0eb6aee3ff4a730e3a1adb82aeb1cfb8e4114f1759a2bcf" 2024-08-07T17:47:15.1267049Z }, 2024-08-07T17:47:15.1267314Z { 2024-08-07T17:47:15.1267748Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1268255Z "size": 32, 2024-08-07T17:47:15.1268780Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1269370Z }, 2024-08-07T17:47:15.1269615Z { 2024-08-07T17:47:15.1270046Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1270678Z "size": 32, 2024-08-07T17:47:15.1271180Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1271777Z }, 2024-08-07T17:47:15.1272038Z { 2024-08-07T17:47:15.1272449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1272982Z "size": 32, 2024-08-07T17:47:15.1273843Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1274422Z }, 2024-08-07T17:47:15.1274685Z { 2024-08-07T17:47:15.1275121Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-07T17:47:15.1275634Z "size": 32, 2024-08-07T17:47:15.1276168Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-07T17:47:15.1276754Z } 2024-08-07T17:47:15.1276996Z ] 2024-08-07T17:47:15.1277256Z } 2024-08-07T17:47:15.1354640Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:47:15.1355150Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-07T17:47:15.1355684Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-07T17:47:15.1364026Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:15.1364498Z env: 2024-08-07T17:47:15.1364792Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:15.1365746Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.1366721Z ##[endgroup] 2024-08-07T17:47:15.1393507Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.1451217Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-07T17:47:15.1451821Z with: 2024-08-07T17:47:15.1452694Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.1453801Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:15.1454285Z env: 2024-08-07T17:47:15.1454574Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:15.1454923Z ##[endgroup] 2024-08-07T17:47:15.1486794Z ##[group]Run set -x 2024-08-07T17:47:15.1487171Z set -x 2024-08-07T17:47:15.1487467Z set +e 2024-08-07T17:47:15.1487771Z  2024-08-07T17:47:15.1488065Z login() { 2024-08-07T17:47:15.1488657Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-07T17:47:15.1489315Z } 2024-08-07T17:47:15.1489611Z  2024-08-07T17:47:15.1489956Z retry () { 2024-08-07T17:47:15.1490312Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-07T17:47:15.1490729Z } 2024-08-07T17:47:15.1491015Z  2024-08-07T17:47:15.1491313Z retry login "${DOCKER_REGISTRY}" 2024-08-07T17:47:15.1491714Z  2024-08-07T17:47:15.1492005Z set -e 2024-08-07T17:47:15.1492446Z # ignore output since only exit code is used for conditional 2024-08-07T17:47:15.1493072Z # only pull docker image if it's not available locally 2024-08-07T17:47:15.1493767Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-07T17:47:15.1494377Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-07T17:47:15.1494798Z fi 2024-08-07T17:47:15.1502358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:47:15.1502855Z env: 2024-08-07T17:47:15.1503155Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:47:15.1504087Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.1505174Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:15.1505677Z ##[endgroup] 2024-08-07T17:47:15.1529332Z + set +e 2024-08-07T17:47:15.1529756Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:15.1530519Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:15.1532845Z + aws ecr get-login-password --region us-east-1 2024-08-07T17:47:15.1534022Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-07T17:47:15.6222192Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-07T17:47:15.6222953Z Configure a credential helper to remove this warning. See 2024-08-07T17:47:15.6223666Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-07T17:47:15.6224144Z 2024-08-07T17:47:15.6224317Z Login Succeeded 2024-08-07T17:47:15.6235092Z + set -e 2024-08-07T17:47:15.6236055Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.6413876Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.6415467Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:47:15.9022301Z 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-08-07T17:47:15.9023431Z 7a2c55901189: Pulling fs layer 2024-08-07T17:47:15.9023953Z 09d415c238d7: Pulling fs layer 2024-08-07T17:47:15.9024339Z 9fe6e2e61518: Pulling fs layer 2024-08-07T17:47:15.9024925Z 41f16248e682: Pulling fs layer 2024-08-07T17:47:15.9025742Z 95d7b7817039: Pulling fs layer 2024-08-07T17:47:15.9026167Z 8f6c90485347: Pulling fs layer 2024-08-07T17:47:15.9026845Z ab17245097e4: Pulling fs layer 2024-08-07T17:47:15.9027528Z dfecd7e9912b: Pulling fs layer 2024-08-07T17:47:15.9027934Z 464a8f745445: Pulling fs layer 2024-08-07T17:47:15.9028309Z d67c111fa588: Pulling fs layer 2024-08-07T17:47:15.9028686Z 0723abde19c0: Pulling fs layer 2024-08-07T17:47:15.9029104Z ceceb076946e: Pulling fs layer 2024-08-07T17:47:15.9029481Z 889eb8142c51: Pulling fs layer 2024-08-07T17:47:15.9029849Z af8081143f00: Pulling fs layer 2024-08-07T17:47:15.9030199Z cc64a79090ad: Pulling fs layer 2024-08-07T17:47:15.9030572Z ad26542e0d01: Pulling fs layer 2024-08-07T17:47:15.9030927Z 7b42d8ae2367: Pulling fs layer 2024-08-07T17:47:15.9031303Z 6d46e3b97d9f: Pulling fs layer 2024-08-07T17:47:15.9031677Z 91f9dddffbee: Pulling fs layer 2024-08-07T17:47:15.9032032Z 03fb78f207e3: Pulling fs layer 2024-08-07T17:47:15.9032408Z eeedca475f8c: Pulling fs layer 2024-08-07T17:47:15.9032795Z 202c71391ade: Pulling fs layer 2024-08-07T17:47:15.9033426Z 5af882ef27a2: Pulling fs layer 2024-08-07T17:47:15.9033846Z bac001ac2fee: Pulling fs layer 2024-08-07T17:47:15.9034224Z 05896f1d52f7: Pulling fs layer 2024-08-07T17:47:15.9034578Z 49a0e418a175: Pulling fs layer 2024-08-07T17:47:15.9034951Z b5632121ff7a: Pulling fs layer 2024-08-07T17:47:15.9035332Z d67c111fa588: Waiting 2024-08-07T17:47:15.9035656Z deceb46806b5: Pulling fs layer 2024-08-07T17:47:15.9036032Z a2630e5fe166: Pulling fs layer 2024-08-07T17:47:15.9036406Z 603261fb379f: Pulling fs layer 2024-08-07T17:47:15.9036751Z 0723abde19c0: Waiting 2024-08-07T17:47:15.9037091Z d8ac71337d62: Pulling fs layer 2024-08-07T17:47:15.9037458Z 2c2e35b979f5: Pulling fs layer 2024-08-07T17:47:15.9037796Z ceceb076946e: Waiting 2024-08-07T17:47:15.9038130Z e559222c0ec6: Pulling fs layer 2024-08-07T17:47:15.9038494Z 41f16248e682: Waiting 2024-08-07T17:47:15.9038816Z c4dcb7e9d189: Pulling fs layer 2024-08-07T17:47:15.9039173Z 95d7b7817039: Waiting 2024-08-07T17:47:15.9039496Z 2b9731d295d6: Pulling fs layer 2024-08-07T17:47:15.9039857Z 889eb8142c51: Waiting 2024-08-07T17:47:15.9040179Z dfecd7e9912b: Waiting 2024-08-07T17:47:15.9040483Z af8081143f00: Waiting 2024-08-07T17:47:15.9040811Z 8f6c90485347: Waiting 2024-08-07T17:47:15.9041123Z 464a8f745445: Waiting 2024-08-07T17:47:15.9041424Z ab17245097e4: Waiting 2024-08-07T17:47:15.9041945Z 03fb78f207e3: Waiting 2024-08-07T17:47:15.9042261Z 05896f1d52f7: Waiting 2024-08-07T17:47:15.9042560Z cc64a79090ad: Waiting 2024-08-07T17:47:15.9042887Z eeedca475f8c: Waiting 2024-08-07T17:47:15.9043211Z 49a0e418a175: Waiting 2024-08-07T17:47:15.9043509Z 202c71391ade: Waiting 2024-08-07T17:47:15.9043845Z 19da43312eba: Pulling fs layer 2024-08-07T17:47:15.9044207Z 5af882ef27a2: Waiting 2024-08-07T17:47:15.9044510Z 603261fb379f: Waiting 2024-08-07T17:47:15.9044828Z b5632121ff7a: Waiting 2024-08-07T17:47:15.9045156Z b846263d0664: Pulling fs layer 2024-08-07T17:47:15.9045496Z ad26542e0d01: Waiting 2024-08-07T17:47:15.9045829Z bac001ac2fee: Waiting 2024-08-07T17:47:15.9046138Z d8ac71337d62: Waiting 2024-08-07T17:47:15.9046461Z deceb46806b5: Waiting 2024-08-07T17:47:15.9046803Z 3886e8f57dc8: Pulling fs layer 2024-08-07T17:47:15.9047146Z a2630e5fe166: Waiting 2024-08-07T17:47:15.9047468Z 7b42d8ae2367: Waiting 2024-08-07T17:47:15.9047828Z 58e13e7b6d7b: Pulling fs layer 2024-08-07T17:47:15.9048189Z c4dcb7e9d189: Waiting 2024-08-07T17:47:15.9048512Z 2b9731d295d6: Waiting 2024-08-07T17:47:15.9048845Z 0eb80a0ac000: Pulling fs layer 2024-08-07T17:47:15.9049194Z 91f9dddffbee: Waiting 2024-08-07T17:47:15.9049520Z 2c2e35b979f5: Waiting 2024-08-07T17:47:15.9049838Z 6d46e3b97d9f: Waiting 2024-08-07T17:47:15.9050141Z e559222c0ec6: Waiting 2024-08-07T17:47:15.9050495Z 19da43312eba: Waiting 2024-08-07T17:47:15.9050817Z b846263d0664: Waiting 2024-08-07T17:47:15.9051130Z a9f11b2b3622: Pulling fs layer 2024-08-07T17:47:15.9051490Z 0eb80a0ac000: Waiting 2024-08-07T17:47:15.9051827Z d7e94279f8b2: Pulling fs layer 2024-08-07T17:47:15.9052172Z 58e13e7b6d7b: Waiting 2024-08-07T17:47:15.9052641Z 4f4fb700ef54: Pulling fs layer 2024-08-07T17:47:15.9053038Z 0a3560fcd26c: Pulling fs layer 2024-08-07T17:47:15.9053401Z adf9d181cfbc: Pulling fs layer 2024-08-07T17:47:15.9053771Z a9f11b2b3622: Waiting 2024-08-07T17:47:15.9054111Z bc18dd831597: Pulling fs layer 2024-08-07T17:47:15.9054469Z d62f2a608619: Pulling fs layer 2024-08-07T17:47:15.9054851Z 719a1c2abcfd: Pulling fs layer 2024-08-07T17:47:15.9055207Z 8e1355d516c5: Pulling fs layer 2024-08-07T17:47:15.9055573Z adf9d181cfbc: Waiting 2024-08-07T17:47:15.9055914Z 701898e66f54: Pulling fs layer 2024-08-07T17:47:15.9056251Z bc18dd831597: Waiting 2024-08-07T17:47:15.9056573Z d62f2a608619: Waiting 2024-08-07T17:47:15.9056891Z d7e94279f8b2: Waiting 2024-08-07T17:47:15.9057191Z 0a3560fcd26c: Waiting 2024-08-07T17:47:15.9057510Z 4f4fb700ef54: Waiting 2024-08-07T17:47:15.9057844Z 129b38cfed12: Pulling fs layer 2024-08-07T17:47:15.9058184Z 701898e66f54: Waiting 2024-08-07T17:47:15.9058516Z 1751651a072f: Pulling fs layer 2024-08-07T17:47:15.9058883Z 719a1c2abcfd: Waiting 2024-08-07T17:47:15.9059193Z 8e1355d516c5: Waiting 2024-08-07T17:47:15.9059530Z fc956bf4e5b4: Pulling fs layer 2024-08-07T17:47:15.9059890Z 129b38cfed12: Waiting 2024-08-07T17:47:15.9060191Z 1751651a072f: Waiting 2024-08-07T17:47:15.9060520Z 874491333fb5: Pulling fs layer 2024-08-07T17:47:15.9060886Z 365b4a0eec21: Pulling fs layer 2024-08-07T17:47:15.9061260Z c1d4cac78640: Pulling fs layer 2024-08-07T17:47:15.9061632Z 29afe2181ca6: Pulling fs layer 2024-08-07T17:47:15.9062000Z 190dccfa687d: Pulling fs layer 2024-08-07T17:47:15.9062357Z b01bea4b6634: Pulling fs layer 2024-08-07T17:47:15.9062727Z f13a9cc59688: Pulling fs layer 2024-08-07T17:47:15.9063078Z 96d6dcd01b9f: Pulling fs layer 2024-08-07T17:47:15.9063449Z fe134554f1b8: Pulling fs layer 2024-08-07T17:47:15.9063815Z 8068d364963f: Pulling fs layer 2024-08-07T17:47:15.9064157Z 29afe2181ca6: Waiting 2024-08-07T17:47:15.9064479Z f13a9cc59688: Waiting 2024-08-07T17:47:15.9064802Z 96d6dcd01b9f: Waiting 2024-08-07T17:47:15.9065119Z fe134554f1b8: Waiting 2024-08-07T17:47:15.9065447Z b01bea4b6634: Waiting 2024-08-07T17:47:15.9065787Z f307f545c1a5: Pulling fs layer 2024-08-07T17:47:15.9066137Z c1d4cac78640: Waiting 2024-08-07T17:47:15.9066476Z 51c0b7eab505: Pulling fs layer 2024-08-07T17:47:15.9066838Z 365b4a0eec21: Waiting 2024-08-07T17:47:15.9067159Z 60aba96f31c6: Pulling fs layer 2024-08-07T17:47:15.9067630Z 874491333fb5: Waiting 2024-08-07T17:47:15.9067949Z f307f545c1a5: Waiting 2024-08-07T17:47:15.9068253Z 51c0b7eab505: Waiting 2024-08-07T17:47:15.9068589Z 46db894a106c: Pulling fs layer 2024-08-07T17:47:15.9068965Z 0052583728e4: Pulling fs layer 2024-08-07T17:47:15.9069303Z 60aba96f31c6: Waiting 2024-08-07T17:47:15.9069643Z 8c6d352a3b51: Pulling fs layer 2024-08-07T17:47:15.9069989Z 46db894a106c: Waiting 2024-08-07T17:47:15.9070324Z 380c4e16278f: Pulling fs layer 2024-08-07T17:47:15.9070682Z 8c6d352a3b51: Waiting 2024-08-07T17:47:15.9070988Z 0052583728e4: Waiting 2024-08-07T17:47:15.9071304Z 380c4e16278f: Waiting 2024-08-07T17:47:16.0942611Z 09d415c238d7: Verifying Checksum 2024-08-07T17:47:16.0943054Z 09d415c238d7: Download complete 2024-08-07T17:47:16.1706915Z 41f16248e682: Verifying Checksum 2024-08-07T17:47:16.1707331Z 41f16248e682: Download complete 2024-08-07T17:47:16.2616227Z 95d7b7817039: Download complete 2024-08-07T17:47:16.3003675Z 7a2c55901189: Verifying Checksum 2024-08-07T17:47:16.3004113Z 7a2c55901189: Download complete 2024-08-07T17:47:16.4445569Z ab17245097e4: Verifying Checksum 2024-08-07T17:47:16.4446051Z ab17245097e4: Download complete 2024-08-07T17:47:16.5181296Z dfecd7e9912b: Verifying Checksum 2024-08-07T17:47:16.5181745Z dfecd7e9912b: Download complete 2024-08-07T17:47:16.5987823Z 464a8f745445: Verifying Checksum 2024-08-07T17:47:16.6326402Z 9fe6e2e61518: Verifying Checksum 2024-08-07T17:47:16.6326828Z 9fe6e2e61518: Download complete 2024-08-07T17:47:16.7106743Z 0723abde19c0: Download complete 2024-08-07T17:47:16.7958402Z ceceb076946e: Download complete 2024-08-07T17:47:17.6083015Z 7a2c55901189: Pull complete 2024-08-07T17:47:17.9661624Z 09d415c238d7: Pull complete 2024-08-07T17:47:19.1207489Z 9fe6e2e61518: Pull complete 2024-08-07T17:47:19.1710076Z 41f16248e682: Pull complete 2024-08-07T17:47:19.2178388Z 95d7b7817039: Pull complete 2024-08-07T17:47:23.4139185Z 889eb8142c51: Verifying Checksum 2024-08-07T17:47:23.4139657Z 889eb8142c51: Download complete 2024-08-07T17:47:23.4913295Z af8081143f00: Verifying Checksum 2024-08-07T17:47:23.4913996Z af8081143f00: Download complete 2024-08-07T17:47:23.5746570Z cc64a79090ad: Verifying Checksum 2024-08-07T17:47:23.5746976Z cc64a79090ad: Download complete 2024-08-07T17:47:23.6438565Z ad26542e0d01: Verifying Checksum 2024-08-07T17:47:23.6438988Z ad26542e0d01: Download complete 2024-08-07T17:47:24.6379120Z 7b42d8ae2367: Verifying Checksum 2024-08-07T17:47:24.6379738Z 7b42d8ae2367: Download complete 2024-08-07T17:47:24.7098671Z 6d46e3b97d9f: Verifying Checksum 2024-08-07T17:47:24.7099082Z 6d46e3b97d9f: Download complete 2024-08-07T17:47:24.7864883Z 91f9dddffbee: Verifying Checksum 2024-08-07T17:47:24.7865323Z 91f9dddffbee: Download complete 2024-08-07T17:47:24.9324424Z 03fb78f207e3: Verifying Checksum 2024-08-07T17:47:24.9324861Z 03fb78f207e3: Download complete 2024-08-07T17:47:30.1562514Z 8f6c90485347: Verifying Checksum 2024-08-07T17:47:30.1562957Z 8f6c90485347: Download complete 2024-08-07T17:47:30.2533902Z 202c71391ade: Verifying Checksum 2024-08-07T17:47:30.2534387Z 202c71391ade: Download complete 2024-08-07T17:47:30.3324592Z 5af882ef27a2: Verifying Checksum 2024-08-07T17:47:30.3325059Z 5af882ef27a2: Download complete 2024-08-07T17:47:30.4195810Z bac001ac2fee: Verifying Checksum 2024-08-07T17:47:30.4196267Z bac001ac2fee: Download complete 2024-08-07T17:47:30.5721565Z 05896f1d52f7: Verifying Checksum 2024-08-07T17:47:30.5722064Z 05896f1d52f7: Download complete 2024-08-07T17:47:30.6706629Z 49a0e418a175: Download complete 2024-08-07T17:47:32.1777001Z b5632121ff7a: Verifying Checksum 2024-08-07T17:47:32.1777473Z b5632121ff7a: Download complete 2024-08-07T17:47:32.2634426Z deceb46806b5: Verifying Checksum 2024-08-07T17:47:32.2635067Z deceb46806b5: Download complete 2024-08-07T17:47:32.3523736Z a2630e5fe166: Verifying Checksum 2024-08-07T17:47:32.3524187Z a2630e5fe166: Download complete 2024-08-07T17:47:32.4406646Z 603261fb379f: Verifying Checksum 2024-08-07T17:47:32.5279304Z 603261fb379f: Download complete 2024-08-07T17:47:32.5280148Z d8ac71337d62: Verifying Checksum 2024-08-07T17:47:32.5280575Z d8ac71337d62: Download complete 2024-08-07T17:47:32.6156568Z 2c2e35b979f5: Verifying Checksum 2024-08-07T17:47:32.6157031Z 2c2e35b979f5: Download complete 2024-08-07T17:47:37.1837435Z e559222c0ec6: Download complete 2024-08-07T17:47:37.2788227Z c4dcb7e9d189: Verifying Checksum 2024-08-07T17:47:37.2788693Z c4dcb7e9d189: Download complete 2024-08-07T17:47:37.3709013Z 2b9731d295d6: Verifying Checksum 2024-08-07T17:47:37.3709446Z 2b9731d295d6: Download complete 2024-08-07T17:47:37.7945033Z 19da43312eba: Verifying Checksum 2024-08-07T17:47:37.7945801Z 19da43312eba: Download complete 2024-08-07T17:47:37.8789105Z b846263d0664: Verifying Checksum 2024-08-07T17:47:37.8789557Z b846263d0664: Download complete 2024-08-07T17:47:38.0154409Z 3886e8f57dc8: Verifying Checksum 2024-08-07T17:47:38.0154848Z 3886e8f57dc8: Download complete 2024-08-07T17:47:38.2858795Z 58e13e7b6d7b: Verifying Checksum 2024-08-07T17:47:38.2859267Z 58e13e7b6d7b: Download complete 2024-08-07T17:47:38.3619291Z 0eb80a0ac000: Verifying Checksum 2024-08-07T17:47:38.3620077Z 0eb80a0ac000: Download complete 2024-08-07T17:47:38.4322424Z a9f11b2b3622: Verifying Checksum 2024-08-07T17:47:38.4322871Z a9f11b2b3622: Download complete 2024-08-07T17:47:38.5533466Z d7e94279f8b2: Download complete 2024-08-07T17:47:38.5629651Z 4f4fb700ef54: Verifying Checksum 2024-08-07T17:47:38.6369812Z 0a3560fcd26c: Verifying Checksum 2024-08-07T17:47:38.6370227Z 0a3560fcd26c: Download complete 2024-08-07T17:47:38.7178487Z adf9d181cfbc: Verifying Checksum 2024-08-07T17:47:38.7178924Z adf9d181cfbc: Download complete 2024-08-07T17:47:39.5058905Z bc18dd831597: Verifying Checksum 2024-08-07T17:47:39.5059869Z bc18dd831597: Download complete 2024-08-07T17:47:39.6200423Z d62f2a608619: Verifying Checksum 2024-08-07T17:47:39.6200888Z d62f2a608619: Download complete 2024-08-07T17:47:39.7161805Z 719a1c2abcfd: Verifying Checksum 2024-08-07T17:47:39.7162613Z 719a1c2abcfd: Download complete 2024-08-07T17:47:39.8028875Z 8e1355d516c5: Verifying Checksum 2024-08-07T17:47:39.8029396Z 8e1355d516c5: Download complete 2024-08-07T17:47:39.8764658Z 701898e66f54: Verifying Checksum 2024-08-07T17:47:39.8765114Z 701898e66f54: Download complete 2024-08-07T17:47:41.1226254Z d67c111fa588: Download complete 2024-08-07T17:47:41.2149621Z 1751651a072f: Verifying Checksum 2024-08-07T17:47:41.2150494Z 1751651a072f: Download complete 2024-08-07T17:47:41.2908240Z fc956bf4e5b4: Verifying Checksum 2024-08-07T17:47:41.2909026Z fc956bf4e5b4: Download complete 2024-08-07T17:47:41.3804429Z 874491333fb5: Verifying Checksum 2024-08-07T17:47:41.3804869Z 874491333fb5: Download complete 2024-08-07T17:47:41.4907013Z 365b4a0eec21: Download complete 2024-08-07T17:47:41.5687595Z c1d4cac78640: Verifying Checksum 2024-08-07T17:47:41.5688440Z c1d4cac78640: Download complete 2024-08-07T17:47:41.7639259Z 29afe2181ca6: Verifying Checksum 2024-08-07T17:47:41.7639736Z 29afe2181ca6: Download complete 2024-08-07T17:47:41.8602190Z 190dccfa687d: Verifying Checksum 2024-08-07T17:47:41.8603072Z 190dccfa687d: Download complete 2024-08-07T17:47:41.9472700Z b01bea4b6634: Verifying Checksum 2024-08-07T17:47:41.9473459Z b01bea4b6634: Download complete 2024-08-07T17:47:42.0413874Z f13a9cc59688: Verifying Checksum 2024-08-07T17:47:42.0414290Z f13a9cc59688: Download complete 2024-08-07T17:47:42.2040294Z 96d6dcd01b9f: Verifying Checksum 2024-08-07T17:47:42.2040794Z 96d6dcd01b9f: Download complete 2024-08-07T17:47:42.3856703Z fe134554f1b8: Verifying Checksum 2024-08-07T17:47:42.3857157Z fe134554f1b8: Download complete 2024-08-07T17:47:42.4581726Z 8068d364963f: Download complete 2024-08-07T17:47:43.1206075Z f307f545c1a5: Verifying Checksum 2024-08-07T17:47:43.1206579Z f307f545c1a5: Download complete 2024-08-07T17:47:43.1974546Z 51c0b7eab505: Download complete 2024-08-07T17:47:49.0170153Z 8f6c90485347: Pull complete 2024-08-07T17:47:49.0641990Z ab17245097e4: Pull complete 2024-08-07T17:47:49.1172776Z dfecd7e9912b: Pull complete 2024-08-07T17:47:49.1837007Z 464a8f745445: Pull complete 2024-08-07T17:47:53.1917340Z eeedca475f8c: Verifying Checksum 2024-08-07T17:47:53.1918187Z eeedca475f8c: Download complete 2024-08-07T17:47:53.2951366Z 46db894a106c: Verifying Checksum 2024-08-07T17:47:53.2951846Z 46db894a106c: Download complete 2024-08-07T17:47:53.3904532Z 0052583728e4: Download complete 2024-08-07T17:47:53.9080377Z 8c6d352a3b51: Verifying Checksum 2024-08-07T17:47:53.9080829Z 8c6d352a3b51: Download complete 2024-08-07T17:47:53.9936806Z 380c4e16278f: Verifying Checksum 2024-08-07T17:47:53.9937327Z 380c4e16278f: Download complete 2024-08-07T17:47:57.0588229Z 60aba96f31c6: Download complete 2024-08-07T17:47:59.1282122Z 129b38cfed12: Verifying Checksum 2024-08-07T17:47:59.1282666Z 129b38cfed12: Download complete 2024-08-07T17:48:27.5378340Z d67c111fa588: Pull complete 2024-08-07T17:48:29.9810542Z 0723abde19c0: Pull complete 2024-08-07T17:48:32.6116114Z ceceb076946e: Pull complete 2024-08-07T17:48:49.0984257Z 889eb8142c51: Pull complete 2024-08-07T17:48:51.5716261Z af8081143f00: Pull complete 2024-08-07T17:48:54.0122365Z cc64a79090ad: Pull complete 2024-08-07T17:48:56.3212136Z ad26542e0d01: Pull complete 2024-08-07T17:49:02.0892035Z 7b42d8ae2367: Pull complete 2024-08-07T17:49:04.5910648Z 6d46e3b97d9f: Pull complete 2024-08-07T17:49:07.0972200Z 91f9dddffbee: Pull complete 2024-08-07T17:49:09.8382124Z 03fb78f207e3: Pull complete 2024-08-07T17:50:15.9181328Z eeedca475f8c: Pull complete 2024-08-07T17:50:20.0420745Z 202c71391ade: Pull complete 2024-08-07T17:50:24.3324959Z 5af882ef27a2: Pull complete 2024-08-07T17:50:27.3039605Z bac001ac2fee: Pull complete 2024-08-07T17:50:29.8064671Z 05896f1d52f7: Pull complete 2024-08-07T17:50:31.6164475Z 49a0e418a175: Pull complete 2024-08-07T17:50:35.8827120Z b5632121ff7a: Pull complete 2024-08-07T17:50:35.9271975Z deceb46806b5: Pull complete 2024-08-07T17:50:35.9779274Z a2630e5fe166: Pull complete 2024-08-07T17:50:36.0427480Z 603261fb379f: Pull complete 2024-08-07T17:50:36.0861355Z d8ac71337d62: Pull complete 2024-08-07T17:50:36.1295777Z 2c2e35b979f5: Pull complete 2024-08-07T17:50:48.1667724Z e559222c0ec6: Pull complete 2024-08-07T17:50:49.7410597Z c4dcb7e9d189: Pull complete 2024-08-07T17:50:49.7804396Z 2b9731d295d6: Pull complete 2024-08-07T17:50:50.7945761Z 19da43312eba: Pull complete 2024-08-07T17:50:50.8329188Z b846263d0664: Pull complete 2024-08-07T17:50:50.8732959Z 3886e8f57dc8: Pull complete 2024-08-07T17:50:51.2563438Z 58e13e7b6d7b: Pull complete 2024-08-07T17:50:51.3121368Z 0eb80a0ac000: Pull complete 2024-08-07T17:50:51.4011919Z a9f11b2b3622: Pull complete 2024-08-07T17:50:51.4571968Z d7e94279f8b2: Pull complete 2024-08-07T17:50:51.4941416Z 4f4fb700ef54: Pull complete 2024-08-07T17:50:51.5460570Z 0a3560fcd26c: Pull complete 2024-08-07T17:50:51.5863052Z adf9d181cfbc: Pull complete 2024-08-07T17:50:54.0956515Z bc18dd831597: Pull complete 2024-08-07T17:50:54.1291523Z d62f2a608619: Pull complete 2024-08-07T17:50:54.1777339Z 719a1c2abcfd: Pull complete 2024-08-07T17:50:54.2938469Z 8e1355d516c5: Pull complete 2024-08-07T17:50:54.3470667Z 701898e66f54: Pull complete 2024-08-07T17:51:37.3472586Z 129b38cfed12: Pull complete 2024-08-07T17:51:37.3896837Z 1751651a072f: Pull complete 2024-08-07T17:51:37.4375522Z fc956bf4e5b4: Pull complete 2024-08-07T17:51:37.5623877Z 874491333fb5: Pull complete 2024-08-07T17:51:37.6809618Z 365b4a0eec21: Pull complete 2024-08-07T17:51:37.7139562Z c1d4cac78640: Pull complete 2024-08-07T17:51:37.9476391Z 29afe2181ca6: Pull complete 2024-08-07T17:51:37.9906541Z 190dccfa687d: Pull complete 2024-08-07T17:51:38.0381690Z b01bea4b6634: Pull complete 2024-08-07T17:51:38.2157168Z f13a9cc59688: Pull complete 2024-08-07T17:51:40.0113083Z 96d6dcd01b9f: Pull complete 2024-08-07T17:51:44.1154631Z fe134554f1b8: Pull complete 2024-08-07T17:51:46.5872046Z 8068d364963f: Pull complete 2024-08-07T17:51:51.7117969Z f307f545c1a5: Pull complete 2024-08-07T17:51:54.2168789Z 51c0b7eab505: Pull complete 2024-08-07T17:52:12.5665004Z 60aba96f31c6: Pull complete 2024-08-07T17:52:12.6141133Z 46db894a106c: Pull complete 2024-08-07T17:52:12.6554809Z 0052583728e4: Pull complete 2024-08-07T17:52:13.3104077Z 8c6d352a3b51: Pull complete 2024-08-07T17:52:13.3495187Z 380c4e16278f: Pull complete 2024-08-07T17:52:13.6244775Z Digest: sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T17:52:13.6288406Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:52:13.6316436Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:52:13.6365974Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:52:13.6366871Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-07T17:52:13.6376005Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:13.6376501Z env: 2024-08-07T17:52:13.6376780Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:13.6377153Z ##[endgroup] 2024-08-07T17:52:13.6646660Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-07T17:52:13.6647228Z with: 2024-08-07T17:52:13.6647511Z driver-version: 550.54.15 2024-08-07T17:52:13.6647862Z env: 2024-08-07T17:52:13.6648155Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:13.6648485Z ##[endgroup] 2024-08-07T17:52:13.6699610Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:52:13.6700142Z with: 2024-08-07T17:52:13.6700436Z timeout_minutes: 10 2024-08-07T17:52:13.6700750Z max_attempts: 3 2024-08-07T17:52:13.6730771Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-07T17:52:13.6761050Z retry_wait_seconds: 10 2024-08-07T17:52:13.6761421Z polling_interval_seconds: 1 2024-08-07T17:52:13.6761778Z warning_on_retry: true 2024-08-07T17:52:13.6762127Z continue_on_error: false 2024-08-07T17:52:13.6762469Z env: 2024-08-07T17:52:13.6762740Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:13.6763094Z DRIVER_VERSION: 550.54.15 2024-08-07T17:52:13.6763419Z ##[endgroup] 2024-08-07T17:52:13.7525040Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-07T17:52:13.7526976Z + pre_install_nvidia_driver_amzn2 2024-08-07T17:52:13.7529282Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-07T17:52:14.2617065Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:14.3227164Z No Match for argument: nvidia-driver-latest-dkms 2024-08-07T17:52:14.3617337Z No Packages marked for removal 2024-08-07T17:52:14.3778291Z + install_nvidia_driver_common 2024-08-07T17:52:14.3782549Z + echo 'Before installing NVIDIA driver' 2024-08-07T17:52:14.3784136Z + lspci 2024-08-07T17:52:14.3786708Z Before installing NVIDIA driver 2024-08-07T17:52:14.3963246Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:52:14.3963957Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:52:14.3964658Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:52:14.3965329Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:52:14.3965888Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:52:14.3966509Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:52:14.3967257Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:14.3968000Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:14.3968733Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:52:14.3969524Z + lsmod 2024-08-07T17:52:14.3987168Z Module Size Used by 2024-08-07T17:52:14.3987620Z ib_core 266240 0 2024-08-07T17:52:14.3987989Z veth 16384 0 2024-08-07T17:52:14.3988354Z nvidia_modeset 1339392 0 2024-08-07T17:52:14.3988705Z nvidia_uvm 4599808 0 2024-08-07T17:52:14.3989128Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:52:14.3989587Z drm 421888 1 nvidia 2024-08-07T17:52:14.3989973Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:52:14.3990420Z backlight 16384 1 nvidia_modeset 2024-08-07T17:52:14.3990845Z xt_conntrack 16384 1 2024-08-07T17:52:14.3991201Z ipt_MASQUERADE 16384 1 2024-08-07T17:52:14.3991618Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:52:14.3992066Z nf_conntrack_netlink 49152 0 2024-08-07T17:52:14.3992476Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:52:14.3992949Z xfrm_user 45056 1 2024-08-07T17:52:14.3993910Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:52:14.3994308Z iptable_nat 16384 1 2024-08-07T17:52:14.3994686Z nf_conntrack_ipv4 16384 3 2024-08-07T17:52:14.3995075Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:52:14.3995526Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:52:14.3996005Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:52:14.3996877Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:52:14.3997652Z xt_addrtype 16384 2 2024-08-07T17:52:14.3998027Z iptable_filter 16384 1 2024-08-07T17:52:14.3998397Z br_netfilter 24576 0 2024-08-07T17:52:14.3998768Z bridge 172032 1 br_netfilter 2024-08-07T17:52:14.3999187Z stp 16384 1 bridge 2024-08-07T17:52:14.3999618Z llc 16384 2 bridge,stp 2024-08-07T17:52:14.4000007Z overlay 86016 0 2024-08-07T17:52:14.4000367Z sunrpc 393216 1 2024-08-07T17:52:14.4000705Z dm_mirror 28672 0 2024-08-07T17:52:14.4001086Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:52:14.4001539Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:52:14.4001986Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:52:14.4002410Z dax 69632 1 dm_mod 2024-08-07T17:52:14.4002797Z sb_edac 24576 0 2024-08-07T17:52:14.4003145Z crc32_pclmul 16384 0 2024-08-07T17:52:14.4003516Z ghash_clmulni_intel 16384 0 2024-08-07T17:52:14.4003883Z pcbc 16384 0 2024-08-07T17:52:14.4004227Z aesni_intel 188416 0 2024-08-07T17:52:14.4004605Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:52:14.4005028Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:52:14.4005438Z glue_helper 16384 1 aesni_intel 2024-08-07T17:52:14.4006058Z mousedev 24576 0 2024-08-07T17:52:14.4006539Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:52:14.4007041Z pcc_cpufreq 16384 0 2024-08-07T17:52:14.4007418Z psmouse 32768 0 2024-08-07T17:52:14.4007780Z evdev 20480 3 2024-08-07T17:52:14.4008119Z button 16384 0 2024-08-07T17:52:14.4008479Z ena 139264 0 2024-08-07T17:52:14.4008842Z ptp 20480 1 ena 2024-08-07T17:52:14.4009217Z pps_core 20480 1 ptp 2024-08-07T17:52:14.4009603Z ata_piix 36864 0 2024-08-07T17:52:14.4009978Z libata 266240 1 ata_piix 2024-08-07T17:52:14.4010358Z xen_blkfront 49152 2 2024-08-07T17:52:14.4010727Z crc32c_intel 24576 0 2024-08-07T17:52:14.4011093Z scsi_mod 245760 1 libata 2024-08-07T17:52:14.4011464Z autofs4 49152 2 2024-08-07T17:52:14.4011837Z + modinfo nvidia 2024-08-07T17:52:14.4012524Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:52:14.4013147Z alias: char-major-195-* 2024-08-07T17:52:14.4013543Z version: 550.54.15 2024-08-07T17:52:14.4013895Z supported: external 2024-08-07T17:52:14.4014247Z license: NVIDIA 2024-08-07T17:52:14.4014606Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:52:14.4015076Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:52:14.4015523Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:52:14.4015948Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:52:14.4016421Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:52:14.4016890Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:52:14.4017306Z depends: i2c-core,drm 2024-08-07T17:52:14.4017673Z retpoline: Y 2024-08-07T17:52:14.4017990Z name: nvidia 2024-08-07T17:52:14.4018433Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:52:14.4019045Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:52:14.4019659Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:52:14.4020343Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:52:14.4020780Z parm: NVreg_RmLogonRC:int 2024-08-07T17:52:14.4021207Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:52:14.4021636Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:52:14.4022212Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:52:14.4022648Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:52:14.4023133Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:52:14.4023670Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:52:14.4024136Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:52:14.4024541Z parm: NVreg_EnableMSI:int 2024-08-07T17:52:14.4024958Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:52:14.4025423Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:52:14.4025910Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:52:14.4026455Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:52:14.4026976Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:52:14.4027520Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:14.4028077Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:52:14.4028658Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:14.4029200Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:52:14.4029671Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:52:14.4030175Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:52:14.4030668Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:52:14.4031141Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:52:14.4031722Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:52:14.4032166Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:52:14.4032616Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:52:14.4033052Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:52:14.4033799Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:52:14.4034302Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:52:14.4034758Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:52:14.4035209Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:52:14.4035689Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:52:14.4036153Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:52:14.4036614Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:52:14.4037076Z parm: NVreg_RmMsg:charp 2024-08-07T17:52:14.4037489Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:52:14.4037925Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:52:14.4038549Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:52:14.4039026Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:52:14.4039471Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:52:14.4039934Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:52:14.4040373Z parm: rm_firmware_active:charp 2024-08-07T17:52:14.4040762Z + HAS_NVIDIA_DRIVER=0 2024-08-07T17:52:14.4041113Z ++ command -v nvidia-smi 2024-08-07T17:52:14.4041466Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-07T17:52:14.4041840Z + set +e 2024-08-07T17:52:14.4042279Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-07T17:52:14.4268204Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-07T17:52:14.4268634Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:52:14.4268972Z + '[' 0 -ne 0 ']' 2024-08-07T17:52:14.4269269Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-07T17:52:14.4269643Z + HAS_NVIDIA_DRIVER=1 2024-08-07T17:52:14.4270229Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-07T17:52:14.4270881Z + set -e 2024-08-07T17:52:14.4271176Z + '[' 1 -eq 0 ']' 2024-08-07T17:52:14.4271707Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-07T17:52:14.4272319Z + post_install_nvidia_driver_common 2024-08-07T17:52:14.4275949Z + sudo modprobe nvidia 2024-08-07T17:52:14.4394291Z + echo 'After installing NVIDIA driver' 2024-08-07T17:52:14.4394735Z + lspci 2024-08-07T17:52:14.4396380Z After installing NVIDIA driver 2024-08-07T17:52:14.4565785Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-08-07T17:52:14.4566517Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-07T17:52:14.4567200Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-08-07T17:52:14.4567866Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-08-07T17:52:14.4568444Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-08-07T17:52:14.4569077Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-07T17:52:14.4569817Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:14.4570588Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-08-07T17:52:14.4571306Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-08-07T17:52:14.4571856Z + lsmod 2024-08-07T17:52:14.4587797Z Module Size Used by 2024-08-07T17:52:14.4588203Z ib_core 266240 0 2024-08-07T17:52:14.4588566Z veth 16384 0 2024-08-07T17:52:14.4588933Z nvidia_modeset 1339392 0 2024-08-07T17:52:14.4589303Z nvidia_uvm 4599808 0 2024-08-07T17:52:14.4589713Z nvidia 53989376 11 nvidia_uvm,nvidia_modeset 2024-08-07T17:52:14.4590178Z drm 421888 1 nvidia 2024-08-07T17:52:14.4590584Z i2c_core 77824 2 nvidia,drm 2024-08-07T17:52:14.4591274Z backlight 16384 1 nvidia_modeset 2024-08-07T17:52:14.4591716Z xt_conntrack 16384 1 2024-08-07T17:52:14.4592099Z ipt_MASQUERADE 16384 1 2024-08-07T17:52:14.4592501Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-08-07T17:52:14.4592956Z nf_conntrack_netlink 49152 0 2024-08-07T17:52:14.4593943Z nfnetlink 16384 2 nf_conntrack_netlink 2024-08-07T17:52:14.4594381Z xfrm_user 45056 1 2024-08-07T17:52:14.4594766Z xfrm_algo 16384 1 xfrm_user 2024-08-07T17:52:14.4595173Z iptable_nat 16384 1 2024-08-07T17:52:14.4595525Z nf_conntrack_ipv4 16384 3 2024-08-07T17:52:14.4595940Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-08-07T17:52:14.4596402Z nf_nat_ipv4 16384 1 iptable_nat 2024-08-07T17:52:14.4596870Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-08-07T17:52:14.4597906Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-08-07T17:52:14.4598740Z xt_addrtype 16384 2 2024-08-07T17:52:14.4599103Z iptable_filter 16384 1 2024-08-07T17:52:14.4599488Z br_netfilter 24576 0 2024-08-07T17:52:14.4599882Z bridge 172032 1 br_netfilter 2024-08-07T17:52:14.4600283Z stp 16384 1 bridge 2024-08-07T17:52:14.4600696Z llc 16384 2 bridge,stp 2024-08-07T17:52:14.4601101Z overlay 86016 0 2024-08-07T17:52:14.4601514Z sunrpc 393216 1 2024-08-07T17:52:14.4601858Z dm_mirror 28672 0 2024-08-07T17:52:14.4602240Z dm_region_hash 20480 1 dm_mirror 2024-08-07T17:52:14.4602697Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-08-07T17:52:14.4603145Z dm_mod 143360 2 dm_log,dm_mirror 2024-08-07T17:52:14.4603569Z dax 69632 1 dm_mod 2024-08-07T17:52:14.4603955Z sb_edac 24576 0 2024-08-07T17:52:14.4604316Z crc32_pclmul 16384 0 2024-08-07T17:52:14.4604696Z ghash_clmulni_intel 16384 0 2024-08-07T17:52:14.4605051Z pcbc 16384 0 2024-08-07T17:52:14.4605418Z aesni_intel 188416 0 2024-08-07T17:52:14.4605798Z aes_x86_64 20480 1 aesni_intel 2024-08-07T17:52:14.4606202Z crypto_simd 16384 1 aesni_intel 2024-08-07T17:52:14.4606634Z glue_helper 16384 1 aesni_intel 2024-08-07T17:52:14.4607036Z mousedev 24576 0 2024-08-07T17:52:14.4607490Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-08-07T17:52:14.4608014Z pcc_cpufreq 16384 0 2024-08-07T17:52:14.4608382Z psmouse 32768 0 2024-08-07T17:52:14.4608724Z evdev 20480 3 2024-08-07T17:52:14.4609081Z button 16384 0 2024-08-07T17:52:14.4609432Z ena 139264 0 2024-08-07T17:52:14.4609775Z ptp 20480 1 ena 2024-08-07T17:52:14.4610159Z pps_core 20480 1 ptp 2024-08-07T17:52:14.4610560Z ata_piix 36864 0 2024-08-07T17:52:14.4610920Z libata 266240 1 ata_piix 2024-08-07T17:52:14.4611322Z xen_blkfront 49152 2 2024-08-07T17:52:14.4611691Z crc32c_intel 24576 0 2024-08-07T17:52:14.4612041Z scsi_mod 245760 1 libata 2024-08-07T17:52:14.4612548Z autofs4 49152 2 2024-08-07T17:52:14.4612920Z + modinfo nvidia 2024-08-07T17:52:14.4613418Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-08-07T17:52:14.4614034Z alias: char-major-195-* 2024-08-07T17:52:14.4614563Z version: 550.54.15 2024-08-07T17:52:14.4614902Z supported: external 2024-08-07T17:52:14.4615249Z license: NVIDIA 2024-08-07T17:52:14.4615624Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-07T17:52:14.4616076Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-07T17:52:14.4616520Z srcversion: 833721318DA517F0C2FEC97 2024-08-07T17:52:14.4617154Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-07T17:52:14.4617613Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-07T17:52:14.4618082Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-07T17:52:14.4618520Z depends: i2c-core,drm 2024-08-07T17:52:14.4618873Z retpoline: Y 2024-08-07T17:52:14.4619196Z name: nvidia 2024-08-07T17:52:14.4619652Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-08-07T17:52:14.4620233Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-07T17:52:14.4620843Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-07T17:52:14.4621419Z parm: NVreg_ResmanDebugLevel:int 2024-08-07T17:52:14.4621833Z parm: NVreg_RmLogonRC:int 2024-08-07T17:52:14.4622266Z parm: NVreg_ModifyDeviceFiles:int 2024-08-07T17:52:14.4622712Z parm: NVreg_DeviceFileUID:int 2024-08-07T17:52:14.4623118Z parm: NVreg_DeviceFileGID:int 2024-08-07T17:52:14.4623651Z parm: NVreg_DeviceFileMode:int 2024-08-07T17:52:14.4624163Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-07T17:52:14.4624677Z parm: NVreg_UsePageAttributeTable:int 2024-08-07T17:52:14.4625139Z parm: NVreg_EnablePCIeGen3:int 2024-08-07T17:52:14.4625562Z parm: NVreg_EnableMSI:int 2024-08-07T17:52:14.4625949Z parm: NVreg_TCEBypassMode:int 2024-08-07T17:52:14.4626393Z parm: NVreg_EnableStreamMemOPs:int 2024-08-07T17:52:14.4626889Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-07T17:52:14.4627415Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-07T17:52:14.4627942Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-07T17:52:14.4628504Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:14.4629050Z parm: NVreg_DynamicPowerManagement:int 2024-08-07T17:52:14.4629629Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-07T17:52:14.4630209Z parm: NVreg_EnableGpuFirmware:int 2024-08-07T17:52:14.4630665Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-07T17:52:14.4631171Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-07T17:52:14.4631686Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-07T17:52:14.4632145Z parm: NVreg_MemoryPoolSize:int 2024-08-07T17:52:14.4632599Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-07T17:52:14.4633064Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-07T17:52:14.4634039Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-07T17:52:14.4634478Z parm: NVreg_NvLinkDisable:int 2024-08-07T17:52:14.4634969Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-07T17:52:14.4635457Z parm: NVreg_RegisterPCIDriver:int 2024-08-07T17:52:14.4635916Z parm: NVreg_EnableResizableBar:int 2024-08-07T17:52:14.4636386Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-07T17:52:14.4636846Z parm: NVreg_EnableNonblockingOpen:int 2024-08-07T17:52:14.4637339Z parm: NVreg_RegistryDwords:charp 2024-08-07T17:52:14.4637814Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-07T17:52:14.4638260Z parm: NVreg_RmMsg:charp 2024-08-07T17:52:14.4638670Z parm: NVreg_GpuBlacklist:charp 2024-08-07T17:52:14.4639125Z parm: NVreg_TemporaryFilePath:charp 2024-08-07T17:52:14.4639562Z parm: NVreg_ExcludedGpus:charp 2024-08-07T17:52:14.4640003Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-07T17:52:14.4640460Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-07T17:52:14.4640899Z parm: NVreg_ImexChannelCount:int 2024-08-07T17:52:14.4641337Z parm: rm_firmware_active:charp 2024-08-07T17:52:14.4641749Z + set +e 2024-08-07T17:52:14.4642017Z + nvidia-smi 2024-08-07T17:52:14.4850868Z Wed Aug 7 17:52:14 2024 2024-08-07T17:52:14.4851388Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:14.4852267Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:52:14.4852940Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:14.4853617Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:52:14.4854321Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:52:14.4854925Z | | | MIG M. | 2024-08-07T17:52:14.4855404Z |=========================================+========================+======================| 2024-08-07T17:52:14.4936391Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:52:14.4937007Z | N/A 35C P8 16W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:14.4937719Z | | | N/A | 2024-08-07T17:52:14.4938313Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:14.5003984Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:52:14.5004593Z | N/A 35C P8 14W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:14.5005131Z | | | N/A | 2024-08-07T17:52:14.5005684Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:14.5006210Z 2024-08-07T17:52:14.5006753Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:14.5007343Z | Processes: | 2024-08-07T17:52:14.5007965Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:52:14.5008557Z | ID ID Usage | 2024-08-07T17:52:14.5009049Z |=========================================================================================| 2024-08-07T17:52:14.5010282Z | No running processes found | 2024-08-07T17:52:14.5010940Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:14.6132747Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-07T17:52:14.6307732Z Tesla M60 2024-08-07T17:52:14.6372186Z + NVIDIA_SMI_STATUS=0 2024-08-07T17:52:14.6372539Z + '[' 0 -eq 0 ']' 2024-08-07T17:52:14.6372882Z + echo 'INFO: Ignoring allowed status 0' 2024-08-07T17:52:14.6373270Z + set -e 2024-08-07T17:52:14.6373955Z INFO: Ignoring allowed status 0 2024-08-07T17:52:14.6379011Z == Installing nvidia container toolkit for amzn2 == 2024-08-07T17:52:14.6382471Z + sudo yum install -y yum-utils 2024-08-07T17:52:15.1268782Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:16.8754637Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-08-07T17:52:16.8755284Z Nothing to do 2024-08-07T17:52:17.0211676Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-08-07T17:52:17.0212324Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:17.0213249Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:17.5339522Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:17.5672704Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-08-07T17:52:17.5675383Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:52:17.5677760Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-08-07T17:52:17.5822626Z + sudo yum install -y nvidia-docker2 2024-08-07T17:52:18.0719439Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-08-07T17:52:19.7310281Z Package nvidia-docker2-2.14.0-1.noarch already installed and latest version 2024-08-07T17:52:19.7310915Z Nothing to do 2024-08-07T17:52:19.8718480Z + sudo systemctl restart docker 2024-08-07T17:52:38.6789399Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-07T17:52:38.7247046Z Wed Aug 7 17:52:38 2024 2024-08-07T17:52:38.7247605Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:38.7248290Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-07T17:52:38.7249001Z |-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:38.7249964Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-07T17:52:38.7250732Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-07T17:52:38.7251336Z | | | MIG M. | 2024-08-07T17:52:38.7251819Z |=========================================+========================+======================| 2024-08-07T17:52:38.7336724Z | 0 Tesla M60 On | 00000000:00:1D.0 Off | 0 | 2024-08-07T17:52:38.7337329Z | N/A 35C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:38.7337871Z | | | N/A | 2024-08-07T17:52:38.7338422Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:38.7404207Z | 1 Tesla M60 On | 00000000:00:1E.0 Off | 0 | 2024-08-07T17:52:38.7404847Z | N/A 35C P8 15W / 150W | 0MiB / 7680MiB | 0% Default | 2024-08-07T17:52:38.7405388Z | | | N/A | 2024-08-07T17:52:38.7405938Z +-----------------------------------------+------------------------+----------------------+ 2024-08-07T17:52:38.7406474Z 2024-08-07T17:52:38.7407021Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:38.7407612Z | Processes: | 2024-08-07T17:52:38.7408218Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-07T17:52:38.7408811Z | ID ID Usage | 2024-08-07T17:52:38.7409327Z |=========================================================================================| 2024-08-07T17:52:38.7410686Z | No running processes found | 2024-08-07T17:52:38.7411323Z +-----------------------------------------------------------------------------------------+ 2024-08-07T17:52:39.7781222Z Command completed after 1 attempt(s). 2024-08-07T17:52:39.7851402Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:39.7852159Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-07T17:52:39.7852802Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-07T17:52:39.7853410Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-07T17:52:39.7861537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:39.7862023Z env: 2024-08-07T17:52:39.7862318Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:39.7862735Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:39.7863421Z ##[endgroup] 2024-08-07T17:52:40.0840033Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:52:40.1053416Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-08-07T17:52:40.1181477Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-08-07T17:52:40.2365535Z Prepare all required actions 2024-08-07T17:52:40.2366665Z Getting action download info 2024-08-07T17:52:40.3359142Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-07T17:52:40.5544359Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-07T17:52:40.7056100Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-07T17:52:40.7056620Z with: 2024-08-07T17:52:40.7056985Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:40.7057464Z s3-bucket: gha-artifacts 2024-08-07T17:52:40.7057826Z env: 2024-08-07T17:52:40.7058123Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:40.7058542Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:40.7059011Z ##[endgroup] 2024-08-07T17:52:40.7104713Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:52:40.7105155Z with: 2024-08-07T17:52:40.7105532Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:52:40.7105947Z s3-bucket: gha-artifacts 2024-08-07T17:52:40.7106302Z region: us-east-1 2024-08-07T17:52:40.7106604Z env: 2024-08-07T17:52:40.7106874Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:40.7107307Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:40.7107773Z ##[endgroup] 2024-08-07T17:52:41.3235258Z (node:75898) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:52:41.3235885Z 2024-08-07T17:52:41.3236143Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:52:41.3236845Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:52:41.3237530Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:52:41.4234440Z Found 1 objects with prefix pytorch/pytorch/10288745067/linux-focal-cuda11.8-py3.10-gcc9/ 2024-08-07T17:52:41.4235378Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:49.0577518Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-07T17:52:49.0589314Z Artifact download has finished successfully 2024-08-07T17:52:49.0805423Z ##[group]Run unzip -o artifacts.zip 2024-08-07T17:52:49.0805863Z unzip -o artifacts.zip 2024-08-07T17:52:49.0814143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:52:49.0814632Z env: 2024-08-07T17:52:49.0814925Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:52:49.0815349Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:52:49.0815840Z ##[endgroup] 2024-08-07T17:52:49.0852991Z Archive: artifacts.zip 2024-08-07T17:52:49.0854959Z creating: dist/ 2024-08-07T17:52:51.5435573Z inflating: dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:52:51.5436222Z creating: build/custom_test_artifacts/ 2024-08-07T17:52:51.5436735Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-07T17:52:51.5437419Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-07T17:52:51.5438141Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:51.5445737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:51.5446581Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:51.5447392Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:51.5448597Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:51.5449460Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:51.5450478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:51.5453031Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:51.5453961Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:51.5454844Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:51.5456110Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:51.5458013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:51.5460392Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:51.5461424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:51.5462974Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:51.5464044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:51.5464981Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:51.5465880Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:51.5524093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:51.5525457Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:51.5526774Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:51.5528083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:51.5529398Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:51.5530644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:51.5531869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:51.5533131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:51.5534371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:51.5580756Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:51.5630795Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:51.5632066Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:51.5633473Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:51.5634617Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:51.5635708Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:51.5636820Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:51.5638167Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:51.5639259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:51.5714432Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:51.5790216Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:51.5791290Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:51.5792189Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:51.5792962Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:51.5793997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:51.5794846Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-07T17:52:51.5795748Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:51.5796781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-07T17:52:51.5797790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-07T17:52:51.5798714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-07T17:52:51.5799688Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:51.5800659Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-07T17:52:51.5801631Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:51.5802583Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-07T17:52:51.5803535Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-07T17:52:51.5825743Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-07T17:52:51.6005590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-07T17:52:51.6006468Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-07T17:52:51.6007441Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-07T17:52:51.6008538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-07T17:52:51.6009567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-07T17:52:51.6010560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-07T17:52:51.6011580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-07T17:52:51.6012609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-07T17:52:51.6013610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-07T17:52:51.6014633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-07T17:52:51.6015639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-07T17:52:51.6038964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-07T17:52:51.6143225Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-07T17:52:51.6144308Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:51.6145276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:51.6146330Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-07T17:52:51.6147182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-07T17:52:51.6147992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:51.6148809Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-07T17:52:51.6151286Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-07T17:52:51.6152005Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-07T17:52:51.6152778Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-07T17:52:51.6302712Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-07T17:52:51.6381466Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-07T17:52:51.6382131Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-07T17:52:51.6382752Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-07T17:52:51.6383458Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:51.6391371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:51.6392205Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:51.6393004Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:51.6394140Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:51.6395011Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:51.6396275Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:51.6397755Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:51.6398661Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:51.6399520Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:51.6401623Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:51.6403219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:51.6405569Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:51.6406585Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:51.6408099Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:51.6409160Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:51.6410089Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:51.6410959Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:51.6468500Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:51.6469752Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:51.6471220Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:51.6472519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:51.6474041Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:51.6475412Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:51.6476650Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:51.6477882Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:51.6479104Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:51.6525130Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:51.6575266Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:51.6576934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:51.6578136Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:51.6579241Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:51.6580696Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:51.6581794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:51.6582864Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:51.6583912Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:51.6658719Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:51.6734601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:51.6735651Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:51.6736514Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:51.6737287Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:51.6738081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:51.6738915Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-07T17:52:51.6739870Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-07T17:52:51.6740934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-07T17:52:51.6741968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-07T17:52:51.6742902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-07T17:52:51.6743898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-07T17:52:51.6744884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-07T17:52:51.6745857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-07T17:52:51.6747100Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-07T17:52:51.6748070Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-07T17:52:51.6770605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-07T17:52:51.6851189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-07T17:52:51.6852413Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:51.6853621Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:51.6854499Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-07T17:52:51.6855295Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-07T17:52:51.6856101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:51.6856892Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-07T17:52:51.6859008Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-07T17:52:51.6859708Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-07T17:52:51.6860608Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-07T17:52:51.6922886Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-07T17:52:51.6923540Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-07T17:52:51.6924205Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-07T17:52:51.6924985Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-07T17:52:51.6933096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-07T17:52:51.6934008Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-07T17:52:51.6934874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-07T17:52:51.6935784Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-07T17:52:51.6936710Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-07T17:52:51.6937751Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-07T17:52:51.6939156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-07T17:52:51.6940087Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-07T17:52:51.6941031Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-07T17:52:51.6942816Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-07T17:52:51.6945048Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-07T17:52:51.6946705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-07T17:52:51.6947796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-07T17:52:51.6949584Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-07T17:52:51.6950720Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-07T17:52:51.6951726Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-07T17:52:51.6952671Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-07T17:52:51.7010537Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-07T17:52:51.7011871Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-07T17:52:51.7013408Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-07T17:52:51.7014824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-07T17:52:51.7016184Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-07T17:52:51.7017486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-07T17:52:51.7018785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-07T17:52:51.7020097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-07T17:52:51.7021409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-07T17:52:51.7082705Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-07T17:52:51.7132653Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-07T17:52:51.7133961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-07T17:52:51.7135176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-07T17:52:51.7136376Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-07T17:52:51.7137520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-07T17:52:51.7138695Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-07T17:52:51.7139838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-07T17:52:51.7140972Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-07T17:52:51.7217131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-07T17:52:51.7292864Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-07T17:52:51.7294005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-07T17:52:51.7294955Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-07T17:52:51.7295761Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-07T17:52:51.7296628Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-07T17:52:51.7297539Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-07T17:52:51.7298542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.7299686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.7300801Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-07T17:52:51.7302037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-07T17:52:51.7303077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.7304295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-07T17:52:51.7305385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.7306460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-07T17:52:51.7307500Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-07T17:52:51.7308647Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-07T17:52:51.7465043Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-07T17:52:51.7466108Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-07T17:52:51.7467162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-07T17:52:51.7468362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-07T17:52:51.7469523Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-07T17:52:51.7470591Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-07T17:52:51.7471728Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-07T17:52:51.7472855Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-07T17:52:51.7474891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-07T17:52:51.7476005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-07T17:52:51.7477116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-07T17:52:51.7498151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-07T17:52:51.7567916Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-07T17:52:51.7569072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-07T17:52:51.7570113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-07T17:52:51.7571050Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-07T17:52:51.7571885Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-07T17:52:51.7572742Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-07T17:52:51.7573608Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-07T17:52:51.7576004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-07T17:52:51.7576745Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-07T17:52:51.7577536Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-07T17:52:51.7706888Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-07T17:52:51.7760615Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-07T17:52:51.7761224Z creating: build/lib/ 2024-08-07T17:52:51.7772755Z inflating: build/lib/libpthreadpool.a 2024-08-07T17:52:51.7782496Z inflating: build/lib/libcpuinfo_internals.a 2024-08-07T17:52:51.7792536Z inflating: build/lib/libcpuinfo.a 2024-08-07T17:52:51.7794012Z inflating: build/lib/libclog.a 2024-08-07T17:52:51.7904551Z inflating: build/lib/libprotobuf-lite.a 2024-08-07T17:52:51.7907042Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-07T17:52:51.8474343Z inflating: build/lib/libprotobuf.a 2024-08-07T17:52:51.8555776Z inflating: build/lib/libgtest.a 2024-08-07T17:52:51.8648841Z inflating: build/lib/libbenchmark.a 2024-08-07T17:52:51.8671319Z inflating: build/lib/libnnpack.a 2024-08-07T17:52:51.8750576Z inflating: build/lib/libasmjit.a 2024-08-07T17:52:51.8760307Z inflating: build/lib/libittnotify.a 2024-08-07T17:52:51.8794845Z inflating: build/lib/libtensorpipe_uv.a 2024-08-07T17:52:51.8952952Z inflating: build/lib/libgloo.a 2024-08-07T17:52:51.8979052Z inflating: build/lib/libfmt.a 2024-08-07T17:52:51.9097174Z inflating: build/lib/libc10.so 2024-08-07T17:52:51.9099239Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-07T17:52:51.9099779Z inflating: build/lib/libfoxi_loader.a 2024-08-07T17:52:51.9101996Z inflating: build/lib/libtorch_global_deps.so 2024-08-07T17:52:51.9126194Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-07T17:52:51.9148603Z inflating: build/lib/libgmock.a 2024-08-07T17:52:51.9775562Z inflating: build/lib/libprotoc.a 2024-08-07T17:52:51.9776051Z inflating: build/lib/libgtest_main.a 2024-08-07T17:52:51.9777110Z inflating: build/lib/libbenchmark_main.a 2024-08-07T17:52:53.2401794Z inflating: build/lib/libdnnl.a 2024-08-07T17:52:53.3114657Z inflating: build/lib/libtensorpipe.a 2024-08-07T17:52:53.3186357Z inflating: build/lib/libc10_cuda.so 2024-08-07T17:52:53.3187143Z inflating: build/lib/libgmock_main.a 2024-08-07T17:52:53.4745047Z inflating: build/lib/libfbgemm.a 2024-08-07T17:52:53.5373936Z inflating: build/lib/libkineto.a 2024-08-07T17:52:53.5691716Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-07T17:52:53.6175433Z inflating: build/lib/libgloo_cuda.a 2024-08-07T17:52:53.6404661Z inflating: build/lib/libXNNPACK.a 2024-08-07T17:52:53.6457560Z inflating: build/lib/libonnx_proto.a 2024-08-07T17:52:53.7318136Z inflating: build/lib/libonnx.a 2024-08-07T17:52:56.7452021Z inflating: build/lib/libtorch_cpu.so 2024-08-07T17:52:56.7456795Z inflating: build/lib/libshm.so 2024-08-07T17:52:56.7462736Z inflating: build/lib/libunbox_lib.a 2024-08-07T17:52:59.2182146Z inflating: build/lib/libtorch_cuda.so 2024-08-07T17:52:59.2182983Z inflating: build/lib/libtorch.so 2024-08-07T17:52:59.2186985Z inflating: build/lib/libc10d_cuda_test.so 2024-08-07T17:53:00.3570338Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-07T17:53:00.3596108Z inflating: build/lib/libjitbackend_test.so 2024-08-07T17:53:00.3685533Z inflating: build/lib/libtorchbind_test.so 2024-08-07T17:53:00.3718203Z inflating: build/lib/libaoti_custom_ops.so 2024-08-07T17:53:00.3749647Z inflating: build/lib/libbackend_with_compiler.so 2024-08-07T17:53:00.6232160Z inflating: build/lib/libtorch_python.so 2024-08-07T17:53:00.6275406Z inflating: build/lib/libnnapi_backend.so 2024-08-07T17:53:00.6275843Z creating: build/bin/ 2024-08-07T17:53:00.6338149Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-07T17:53:00.6402535Z inflating: build/bin/c10_DeviceGuard_test 2024-08-07T17:53:00.6465245Z inflating: build/bin/c10_Device_test 2024-08-07T17:53:00.6538987Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-07T17:53:00.6606165Z inflating: build/bin/c10_Scalar_test 2024-08-07T17:53:00.6667215Z inflating: build/bin/c10_StreamGuard_test 2024-08-07T17:53:00.6731170Z inflating: build/bin/c10_SymInt_test 2024-08-07T17:53:00.6799349Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-07T17:53:00.6868759Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-07T17:53:00.6939253Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-07T17:53:00.7028200Z inflating: build/bin/c10_cow_test 2024-08-07T17:53:00.7094785Z inflating: build/bin/c10_Bitset_test 2024-08-07T17:53:00.7156403Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-07T17:53:00.7218730Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-07T17:53:00.7282333Z inflating: build/bin/c10_Half_test 2024-08-07T17:53:00.7352923Z inflating: build/bin/c10_LeftRight_test 2024-08-07T17:53:00.7421845Z inflating: build/bin/c10_Metaprogramming_test 2024-08-07T17:53:00.7484375Z inflating: build/bin/c10_Synchronized_test 2024-08-07T17:53:00.7553816Z inflating: build/bin/c10_ThreadLocal_test 2024-08-07T17:53:00.7618377Z inflating: build/bin/c10_TypeIndex_test 2024-08-07T17:53:00.7682292Z inflating: build/bin/c10_TypeList_test 2024-08-07T17:53:00.7743254Z inflating: build/bin/c10_TypeTraits_test 2024-08-07T17:53:00.7807976Z inflating: build/bin/c10_accumulate_test 2024-08-07T17:53:00.7877506Z inflating: build/bin/c10_bfloat16_test 2024-08-07T17:53:00.7940603Z inflating: build/bin/c10_bit_cast_test 2024-08-07T17:53:00.8011009Z inflating: build/bin/c10_complex_math_test 2024-08-07T17:53:00.8077177Z inflating: build/bin/c10_exception_test 2024-08-07T17:53:00.8146609Z inflating: build/bin/c10_complex_test 2024-08-07T17:53:00.8209754Z inflating: build/bin/c10_flags_test 2024-08-07T17:53:00.8272219Z inflating: build/bin/c10_generic_math_test 2024-08-07T17:53:00.8335718Z inflating: build/bin/c10_irange_test 2024-08-07T17:53:00.8541886Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-07T17:53:00.8608891Z inflating: build/bin/c10_lazy_test 2024-08-07T17:53:00.8680356Z inflating: build/bin/c10_logging_test 2024-08-07T17:53:00.8773725Z inflating: build/bin/c10_optional_test 2024-08-07T17:53:00.8852120Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-07T17:53:00.8919404Z inflating: build/bin/c10_registry_test 2024-08-07T17:53:00.9108397Z inflating: build/bin/c10_small_vector_test 2024-08-07T17:53:00.9173196Z inflating: build/bin/c10_ssize_test 2024-08-07T17:53:00.9238382Z inflating: build/bin/c10_string_util_test 2024-08-07T17:53:00.9301369Z inflating: build/bin/c10_tempfile_test 2024-08-07T17:53:00.9374842Z inflating: build/bin/c10_string_view_test 2024-08-07T17:53:00.9444889Z inflating: build/bin/c10_typeid_test 2024-08-07T17:53:00.9505466Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-07T17:53:01.0063013Z inflating: build/bin/protoc-3.13.0.0 2024-08-07T17:53:01.0620942Z inflating: build/bin/protoc 2024-08-07T17:53:01.0687266Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-07T17:53:01.0753253Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-07T17:53:01.0819185Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-07T17:53:01.0884340Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-07T17:53:01.0950264Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-07T17:53:01.1011845Z inflating: build/bin/c10_cuda_CUDATest 2024-08-07T17:53:01.1078030Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-07T17:53:01.1144134Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-07T17:53:01.1574188Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-07T17:53:01.1987449Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-07T17:53:01.2437387Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-07T17:53:01.2503369Z inflating: build/bin/FileStoreTest 2024-08-07T17:53:01.2570526Z inflating: build/bin/HashStoreTest 2024-08-07T17:53:01.2636414Z inflating: build/bin/BackoffTest 2024-08-07T17:53:01.2706077Z inflating: build/bin/TCPStoreTest 2024-08-07T17:53:01.2723002Z inflating: build/bin/ProcessGroupMPITest 2024-08-07T17:53:01.2728252Z inflating: build/bin/torch_shm_manager 2024-08-07T17:53:01.2795813Z inflating: build/bin/test_edge_op_registration 2024-08-07T17:53:01.2799520Z inflating: build/bin/example_allreduce 2024-08-07T17:53:01.2864338Z inflating: build/bin/Dimname_test 2024-08-07T17:53:01.2956691Z inflating: build/bin/Dict_test 2024-08-07T17:53:01.3037337Z inflating: build/bin/MaybeOwned_test 2024-08-07T17:53:01.3111126Z inflating: build/bin/apply_utils_test 2024-08-07T17:53:01.3182811Z inflating: build/bin/NamedTensor_test 2024-08-07T17:53:01.3256971Z inflating: build/bin/atest 2024-08-07T17:53:01.3335648Z inflating: build/bin/basic 2024-08-07T17:53:01.3404487Z inflating: build/bin/broadcast_test 2024-08-07T17:53:01.3467776Z inflating: build/bin/cpu_allocator_test 2024-08-07T17:53:01.3541004Z inflating: build/bin/cpu_generator_test 2024-08-07T17:53:01.3607745Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-07T17:53:01.3724353Z inflating: build/bin/cpu_rng_test 2024-08-07T17:53:01.3786909Z inflating: build/bin/dispatch_key_set_test 2024-08-07T17:53:01.3849727Z inflating: build/bin/dlconvertor_test 2024-08-07T17:53:01.3922638Z inflating: build/bin/extension_backend_test 2024-08-07T17:53:01.3991868Z inflating: build/bin/half_test 2024-08-07T17:53:01.4111619Z inflating: build/bin/ivalue_test 2024-08-07T17:53:01.4173967Z inflating: build/bin/lazy_tensor_test 2024-08-07T17:53:01.4242374Z inflating: build/bin/math_kernel_test 2024-08-07T17:53:01.4309739Z inflating: build/bin/memory_format_test 2024-08-07T17:53:01.4375981Z inflating: build/bin/memory_overlapping_test 2024-08-07T17:53:01.4442500Z inflating: build/bin/mobile_memory_cleanup 2024-08-07T17:53:01.4505695Z inflating: build/bin/operator_name_test 2024-08-07T17:53:01.4575717Z inflating: build/bin/native_test 2024-08-07T17:53:01.4640211Z inflating: build/bin/operators_test 2024-08-07T17:53:01.4705036Z inflating: build/bin/packedtensoraccessor_test 2024-08-07T17:53:01.4789284Z inflating: build/bin/pow_test 2024-08-07T17:53:01.4860677Z inflating: build/bin/quantized_test 2024-08-07T17:53:01.4923504Z inflating: build/bin/reduce_ops_test 2024-08-07T17:53:01.4987342Z inflating: build/bin/reportMemoryUsage_test 2024-08-07T17:53:01.5058063Z inflating: build/bin/scalar_tensor_test 2024-08-07T17:53:01.5131935Z inflating: build/bin/scalar_test 2024-08-07T17:53:01.5197260Z inflating: build/bin/StorageUtils_test 2024-08-07T17:53:01.5262901Z inflating: build/bin/stride_properties_test 2024-08-07T17:53:01.5361659Z inflating: build/bin/tensor_iterator_test 2024-08-07T17:53:01.5365271Z inflating: build/bin/thread_init_test 2024-08-07T17:53:01.5435089Z inflating: build/bin/type_ptr_test 2024-08-07T17:53:01.5502823Z inflating: build/bin/test_parallel 2024-08-07T17:53:01.5578270Z inflating: build/bin/type_test 2024-08-07T17:53:01.5643613Z inflating: build/bin/undefined_tensor_test 2024-08-07T17:53:01.5645531Z inflating: build/bin/verify_api_visibility 2024-08-07T17:53:01.5732114Z inflating: build/bin/legacy_vmap_test 2024-08-07T17:53:01.5796737Z inflating: build/bin/wrapdim_test 2024-08-07T17:53:01.5862107Z inflating: build/bin/weakref_test 2024-08-07T17:53:01.5926739Z inflating: build/bin/xla_tensor_test 2024-08-07T17:53:01.6001820Z inflating: build/bin/IListRef_test 2024-08-07T17:53:01.6134783Z inflating: build/bin/List_test 2024-08-07T17:53:01.6287633Z inflating: build/bin/kernel_function_legacy_test 2024-08-07T17:53:01.6408406Z inflating: build/bin/kernel_function_test 2024-08-07T17:53:01.6491876Z inflating: build/bin/KernelFunction_test 2024-08-07T17:53:01.6652332Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-07T17:53:01.6781920Z inflating: build/bin/kernel_lambda_test 2024-08-07T17:53:01.6858602Z inflating: build/bin/kernel_stackbased_test 2024-08-07T17:53:01.6979192Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-07T17:53:01.7043039Z inflating: build/bin/CppSignature_test 2024-08-07T17:53:01.7104077Z inflating: build/bin/op_allowlist_test 2024-08-07T17:53:01.7174191Z inflating: build/bin/backend_fallback_test 2024-08-07T17:53:01.7551910Z inflating: build/bin/op_registration_test 2024-08-07T17:53:01.7631510Z inflating: build/bin/inline_container_test 2024-08-07T17:53:01.7697754Z inflating: build/bin/cuda_apply_test 2024-08-07T17:53:01.7762935Z inflating: build/bin/cuda_allocator_test 2024-08-07T17:53:01.7831191Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-07T17:53:01.7905098Z inflating: build/bin/cuda_atomic_ops_test 2024-08-07T17:53:01.7993382Z inflating: build/bin/cuda_complex_math_test 2024-08-07T17:53:01.8067186Z inflating: build/bin/cuda_complex_test 2024-08-07T17:53:01.8129439Z inflating: build/bin/cuda_device_test 2024-08-07T17:53:01.8201963Z inflating: build/bin/cuda_cub_test 2024-08-07T17:53:01.8265470Z inflating: build/bin/cuda_dlconvertor_test 2024-08-07T17:53:01.8347491Z inflating: build/bin/cuda_distributions_test 2024-08-07T17:53:01.8411443Z inflating: build/bin/cuda_integer_divider_test 2024-08-07T17:53:01.8473915Z inflating: build/bin/cuda_half_test 2024-08-07T17:53:01.8544978Z inflating: build/bin/cuda_generator_test 2024-08-07T17:53:01.8606826Z inflating: build/bin/cuda_optional_test 2024-08-07T17:53:01.8669023Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-07T17:53:01.8744869Z inflating: build/bin/cuda_stream_test 2024-08-07T17:53:01.8810698Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-07T17:53:01.8875229Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-07T17:53:01.8936844Z inflating: build/bin/cuda_cudnn_test 2024-08-07T17:53:01.9002043Z inflating: build/bin/cuda_vectorized_test 2024-08-07T17:53:01.9084071Z inflating: build/bin/ProcessGroupGlooTest 2024-08-07T17:53:01.9102010Z inflating: build/bin/tutorial_tensorexpr 2024-08-07T17:53:01.9173716Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-07T17:53:01.9253752Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-07T17:53:01.9331437Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-07T17:53:01.9397232Z inflating: build/bin/ProcessGroupUCCTest 2024-08-07T17:53:01.9465683Z inflating: build/bin/test_dist_autograd 2024-08-07T17:53:01.9468881Z inflating: build/bin/parallel_benchmark 2024-08-07T17:53:01.9555180Z inflating: build/bin/test_cpp_rpc 2024-08-07T17:53:01.9638814Z inflating: build/bin/test_mobile_nnc 2024-08-07T17:53:01.9650125Z inflating: build/bin/aot_model_compiler_test 2024-08-07T17:53:02.0083265Z inflating: build/bin/test_lazy 2024-08-07T17:53:02.1115093Z inflating: build/bin/test_tensorexpr 2024-08-07T17:53:02.2589172Z inflating: build/bin/test_api 2024-08-07T17:53:02.3313886Z inflating: build/bin/test_jit 2024-08-07T17:53:02.3314345Z creating: .additional_ci_files/ 2024-08-07T17:53:02.3386261Z inflating: .additional_ci_files/test-times.json 2024-08-07T17:53:02.3671914Z inflating: .additional_ci_files/test-class-times.json 2024-08-07T17:53:02.3713531Z ##[group]Run rm artifacts.zip 2024-08-07T17:53:02.3714013Z rm artifacts.zip 2024-08-07T17:53:02.3722202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.3722701Z env: 2024-08-07T17:53:02.3722999Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.3723423Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.3723894Z ##[endgroup] 2024-08-07T17:53:02.4898646Z ##[group]Run df -H 2024-08-07T17:53:02.4899001Z df -H 2024-08-07T17:53:02.4907057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:02.4907777Z env: 2024-08-07T17:53:02.4908080Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.4908501Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.4908966Z ##[endgroup] 2024-08-07T17:53:02.4942725Z Filesystem Size Used Avail Use% Mounted on 2024-08-07T17:53:02.4943200Z devtmpfs 129G 0 129G 0% /dev 2024-08-07T17:53:02.4943634Z tmpfs 129G 0 129G 0% /dev/shm 2024-08-07T17:53:02.4944560Z tmpfs 129G 594k 129G 1% /run 2024-08-07T17:53:02.4945018Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-08-07T17:53:02.4945458Z /dev/xvda1 162G 37G 125G 23% / 2024-08-07T17:53:02.4945880Z tmpfs 26G 0 26G 0% /run/user/0 2024-08-07T17:53:02.4986257Z Prepare all required actions 2024-08-07T17:53:02.4986799Z Getting action download info 2024-08-07T17:53:02.6454883Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-07T17:53:02.6455352Z with: 2024-08-07T17:53:02.6455633Z env: 2024-08-07T17:53:02.6455927Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.6456371Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.6456842Z ##[endgroup] 2024-08-07T17:53:02.6505106Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-07T17:53:02.6505548Z with: 2024-08-07T17:53:02.6505834Z name: td_results 2024-08-07T17:53:02.6506141Z s3-bucket: gha-artifacts 2024-08-07T17:53:02.6506494Z region: us-east-1 2024-08-07T17:53:02.6506796Z env: 2024-08-07T17:53:02.6507065Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:02.6507501Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:02.6507963Z ##[endgroup] 2024-08-07T17:53:03.2553583Z (node:75921) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-07T17:53:03.2554231Z 2024-08-07T17:53:03.2554489Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-07T17:53:03.2555171Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-07T17:53:03.2555896Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-07T17:53:03.3415458Z Found 1 objects with prefix pytorch/pytorch/10288745067/td_results/ 2024-08-07T17:53:03.3417014Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:03.4270947Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-07T17:53:03.4281956Z Artifact download has finished successfully 2024-08-07T17:53:03.4466830Z ##[group]Run mkdir -p .additional_ci_files 2024-08-07T17:53:03.4467296Z mkdir -p .additional_ci_files 2024-08-07T17:53:03.4467846Z mv td_results.json .additional_ci_files/td_results.json 2024-08-07T17:53:03.4476684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:03.4477155Z env: 2024-08-07T17:53:03.4477448Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:03.4477890Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:03.4478365Z ##[endgroup] 2024-08-07T17:53:03.4576138Z ##[group]Run .github/scripts/parse_ref.py 2024-08-07T17:53:03.4576721Z .github/scripts/parse_ref.py 2024-08-07T17:53:03.4584121Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:03.4584463Z env: 2024-08-07T17:53:03.4584758Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:03.4585197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:03.4585650Z ##[endgroup] 2024-08-07T17:53:03.4887758Z Prepare all required actions 2024-08-07T17:53:03.4937209Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-07T17:53:03.4937644Z with: 2024-08-07T17:53:03.4938175Z github-token: *** 2024-08-07T17:53:03.4938501Z env: 2024-08-07T17:53:03.4938788Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:03.4939207Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:03.4939677Z ##[endgroup] 2024-08-07T17:53:03.4964726Z ##[group]Run set -eux 2024-08-07T17:53:03.4965091Z set -eux 2024-08-07T17:53:03.4965639Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-07T17:53:03.4973355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:03.4973850Z env: 2024-08-07T17:53:03.4974130Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:03.4974569Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:03.4975240Z GITHUB_TOKEN: *** 2024-08-07T17:53:03.4975558Z ##[endgroup] 2024-08-07T17:53:03.4999231Z + python3 .github/scripts/get_workflow_job_id.py 10288745067 i-095fa8d57edae1a62 2024-08-07T17:53:06.9059693Z setting job-id=28476144889 2024-08-07T17:53:06.9060444Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:06.9267567Z Prepare all required actions 2024-08-07T17:53:06.9268096Z Getting action download info 2024-08-07T17:53:07.0338503Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-07T17:53:07.0338941Z with: 2024-08-07T17:53:07.0339458Z github-token: *** 2024-08-07T17:53:07.0340803Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-08-07T17:53:07.0342456Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:07.0343104Z env: 2024-08-07T17:53:07.0343409Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.0343853Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.0344308Z ##[endgroup] 2024-08-07T17:53:07.0395341Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-07T17:53:07.0395891Z with: 2024-08-07T17:53:07.0396255Z shell: bash 2024-08-07T17:53:07.0396548Z timeout_minutes: 10 2024-08-07T17:53:07.0396886Z max_attempts: 5 2024-08-07T17:53:07.0397209Z retry_wait_seconds: 30 2024-08-07T17:53:07.0398279Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:07.0399294Z polling_interval_seconds: 1 2024-08-07T17:53:07.0399671Z warning_on_retry: true 2024-08-07T17:53:07.0400026Z continue_on_error: false 2024-08-07T17:53:07.0400359Z env: 2024-08-07T17:53:07.0400645Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:07.0401083Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:07.0401779Z GITHUB_TOKEN: *** 2024-08-07T17:53:07.0402110Z ##[endgroup] 2024-08-07T17:53:07.1120458Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-07T17:53:07.4123736Z Defaulting to user installation because normal site-packages is not writeable 2024-08-07T17:53:07.4334748Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-08-07T17:53:07.4525463Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-08-07T17:53:07.4537073Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-08-07T17:53:07.4554300Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-08-07T17:53:07.4581042Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-07T17:53:07.4592093Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-08-07T17:53:08.1109873Z Command completed after 1 attempt(s). 2024-08-07T17:53:08.1172614Z ##[group]Run set -x 2024-08-07T17:53:08.1172978Z set -x 2024-08-07T17:53:08.1173296Z  2024-08-07T17:53:08.1173798Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:08.1174597Z # in runner workspace 2024-08-07T17:53:08.1175109Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-07T17:53:08.1183461Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:08.1183940Z env: 2024-08-07T17:53:08.1184244Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:08.1184690Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:08.1185147Z ##[endgroup] 2024-08-07T17:53:08.1211098Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-07T17:53:08.1485935Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:08.1486492Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-07T17:53:08.1486921Z echo "Job name: ${JOB_NAME}" 2024-08-07T17:53:08.1487310Z  2024-08-07T17:53:08.1487802Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-07T17:53:08.1488404Z # in runner workspace 2024-08-07T17:53:08.1488952Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-07T17:53:08.1489558Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-07T17:53:08.1489985Z  --job-name "${JOB_NAME}" \ 2024-08-07T17:53:08.1491410Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-08-07T17:53:08.1492853Z  --selected-test-configs "" \ 2024-08-07T17:53:08.1493301Z  --pr-number "${PR_NUMBER}" \ 2024-08-07T17:53:08.1493703Z  --tag "${TAG}" \ 2024-08-07T17:53:08.1494095Z  --event-name "${EVENT_NAME}" \ 2024-08-07T17:53:08.1494530Z  --schedule "${SCHEDULE}" \ 2024-08-07T17:53:08.1494924Z  --branch "${HEAD_BRANCH}" 2024-08-07T17:53:08.1502160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:08.1502646Z env: 2024-08-07T17:53:08.1502924Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:08.1503362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:08.1504049Z GITHUB_TOKEN: *** 2024-08-07T17:53:08.1504640Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:08.1505283Z PR_NUMBER: 131248 2024-08-07T17:53:08.1505600Z TAG: 2024-08-07T17:53:08.1505893Z EVENT_NAME: pull_request 2024-08-07T17:53:08.1506235Z SCHEDULE: 2024-08-07T17:53:08.1506536Z HEAD_BRANCH: 2024-08-07T17:53:08.1506823Z ##[endgroup] 2024-08-07T17:53:08.1530555Z Workflow: pull 2024-08-07T17:53:08.1531287Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:08.4109184Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-07T17:53:08.5395809Z ##[group]Run echo "Filtered matrix:" 2024-08-07T17:53:08.5396268Z echo "Filtered matrix:" 2024-08-07T17:53:08.5397633Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-08-07T17:53:08.5398996Z  2024-08-07T17:53:08.5399292Z echo 2024-08-07T17:53:08.5399675Z echo "Is the current job unstable? False" 2024-08-07T17:53:08.5400129Z  2024-08-07T17:53:08.5400417Z echo 2024-08-07T17:53:08.5400751Z echo "Is keep-going label set? False" 2024-08-07T17:53:08.5401179Z  2024-08-07T17:53:08.5401464Z echo 2024-08-07T17:53:08.5401772Z echo "Renabled issues? " 2024-08-07T17:53:08.5410011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:08.5410507Z env: 2024-08-07T17:53:08.5410792Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:08.5411234Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:08.5411709Z ##[endgroup] 2024-08-07T17:53:08.5436055Z Filtered matrix: 2024-08-07T17:53:08.5437329Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-08-07T17:53:08.5438552Z 2024-08-07T17:53:08.5438983Z Is the current job unstable? False 2024-08-07T17:53:08.5439283Z 2024-08-07T17:53:08.5439434Z Is keep-going label set? False 2024-08-07T17:53:08.5439698Z 2024-08-07T17:53:08.5439834Z Renabled issues? 2024-08-07T17:53:08.5503269Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:08.5504027Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-07T17:53:08.5511370Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T17:53:08.5511865Z env: 2024-08-07T17:53:08.5512146Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:08.5512589Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:08.5513061Z JOB_TIMEOUT: 360 2024-08-07T17:53:08.5513626Z ##[endgroup] 2024-08-07T17:53:08.5596955Z ##[group]Run set -x 2024-08-07T17:53:08.5597410Z set -x 2024-08-07T17:53:08.5597704Z  2024-08-07T17:53:08.5598047Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-07T17:53:08.5598563Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-07T17:53:08.5599069Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-07T17:53:08.5599544Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-07T17:53:08.5599947Z else 2024-08-07T17:53:08.5600272Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:08.5600709Z fi 2024-08-07T17:53:08.5601002Z  2024-08-07T17:53:08.5601429Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-07T17:53:08.5602102Z # TODO: Stop building test binaries as part of the build phase 2024-08-07T17:53:08.5602706Z # Used for GPU_FLAG since that doesn't play nice 2024-08-07T17:53:08.5603246Z # shellcheck disable=SC2086,SC2090 2024-08-07T17:53:08.5603678Z container_name=$(docker run \ 2024-08-07T17:53:08.5604095Z  ${GPU_FLAG:-} \ 2024-08-07T17:53:08.5604473Z  -e BUILD_ENVIRONMENT \ 2024-08-07T17:53:08.5604850Z  -e PR_NUMBER \ 2024-08-07T17:53:08.5605223Z  -e GITHUB_ACTIONS \ 2024-08-07T17:53:08.5605614Z  -e GITHUB_REPOSITORY \ 2024-08-07T17:53:08.5605997Z  -e GITHUB_WORKFLOW \ 2024-08-07T17:53:08.5606384Z  -e GITHUB_JOB \ 2024-08-07T17:53:08.5606746Z  -e GITHUB_RUN_ID \ 2024-08-07T17:53:08.5607114Z  -e GITHUB_RUN_NUMBER \ 2024-08-07T17:53:08.5607519Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-07T17:53:08.5607904Z  -e JOB_ID \ 2024-08-07T17:53:08.5608230Z  -e JOB_NAME \ 2024-08-07T17:53:08.5608583Z  -e BASE_SHA \ 2024-08-07T17:53:08.5608926Z  -e BRANCH \ 2024-08-07T17:53:08.5609245Z  -e SHA1 \ 2024-08-07T17:53:08.5609594Z  -e AWS_DEFAULT_REGION \ 2024-08-07T17:53:08.5609990Z  -e IN_WHEEL_TEST \ 2024-08-07T17:53:08.5610340Z  -e SHARD_NUMBER \ 2024-08-07T17:53:08.5610708Z  -e TEST_CONFIG \ 2024-08-07T17:53:08.5611075Z  -e NUM_TEST_SHARDS \ 2024-08-07T17:53:08.5611446Z  -e REENABLED_ISSUES \ 2024-08-07T17:53:08.5611844Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-07T17:53:08.5612256Z  -e VERBOSE_TEST_LOGS \ 2024-08-07T17:53:08.5612628Z  -e TEST_SHOWLOCALS \ 2024-08-07T17:53:08.5613011Z  -e NO_TEST_TIMEOUT \ 2024-08-07T17:53:08.5613568Z  -e NO_TD \ 2024-08-07T17:53:08.5613892Z  -e TD_DISTRIBUTED \ 2024-08-07T17:53:08.5614272Z  -e PR_LABELS \ 2024-08-07T17:53:08.5614662Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-07T17:53:08.5615081Z  -e SCCACHE_BUCKET \ 2024-08-07T17:53:08.5615467Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-07T17:53:08.5615867Z  -e XLA_CUDA \ 2024-08-07T17:53:08.5616243Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-07T17:53:08.5616716Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-07T17:53:08.5617198Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-07T17:53:08.5617662Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-07T17:53:08.5618115Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-07T17:53:08.5618527Z  -e DASHBOARD_TAG \ 2024-08-07T17:53:08.5618955Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-07T17:53:08.5619472Z  --security-opt seccomp=unconfined \ 2024-08-07T17:53:08.5619946Z  --cap-add=SYS_PTRACE \ 2024-08-07T17:53:08.5620324Z  --ipc=host \ 2024-08-07T17:53:08.5620686Z  --shm-size="${SHM_SIZE}" \ 2024-08-07T17:53:08.5621079Z  --tty \ 2024-08-07T17:53:08.5621447Z  --detach \ 2024-08-07T17:53:08.5621992Z  --name="${container_name}" \ 2024-08-07T17:53:08.5622622Z  --user jenkins \ 2024-08-07T17:53:08.5623193Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-07T17:53:08.5623897Z  -w /var/lib/jenkins/workspace \ 2024-08-07T17:53:08.5624375Z  "${DOCKER_IMAGE}" 2024-08-07T17:53:08.5624821Z ) 2024-08-07T17:53:08.5625380Z # Propagate download.pytorch.org IP to container 2024-08-07T17:53:08.5626288Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-07T17:53:08.5627265Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-07T17:53:08.5628274Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-07T17:53:08.5636359Z shell: /usr/bin/bash -e {0} 2024-08-07T17:53:08.5636942Z env: 2024-08-07T17:53:08.5637366Z GIT_DEFAULT_BRANCH: main 2024-08-07T17:53:08.5637857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:08.5638570Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:08.5639148Z PR_NUMBER: 131248 2024-08-07T17:53:08.5639636Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-07T17:53:08.5640180Z GITHUB_WORKFLOW: pull 2024-08-07T17:53:08.5640582Z GITHUB_JOB: test 2024-08-07T17:53:08.5641003Z GITHUB_RUN_ID: 10288745067 2024-08-07T17:53:08.5641500Z GITHUB_RUN_NUMBER: 234358 2024-08-07T17:53:08.5641994Z GITHUB_RUN_ATTEMPT: 1 2024-08-07T17:53:08.5642426Z JOB_ID: 28476144889 2024-08-07T17:53:08.5643169Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:08.5643951Z BRANCH: pull/131248 2024-08-07T17:53:08.5644405Z SHA1: 016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:08.5645029Z BASE_SHA: 6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:08.5645562Z TEST_CONFIG: distributed 2024-08-07T17:53:08.5646046Z SHARD_NUMBER: 1 2024-08-07T17:53:08.5646449Z NUM_TEST_SHARDS: 3 2024-08-07T17:53:08.5646826Z REENABLED_ISSUES: 2024-08-07T17:53:08.5647291Z CONTINUE_THROUGH_ERROR: False 2024-08-07T17:53:08.5647752Z VERBOSE_TEST_LOGS: False 2024-08-07T17:53:08.5648147Z TEST_SHOWLOCALS: False 2024-08-07T17:53:08.5648628Z NO_TEST_TIMEOUT: False 2024-08-07T17:53:08.5649047Z NO_TD: False 2024-08-07T17:53:08.5649396Z TD_DISTRIBUTED: False 2024-08-07T17:53:08.5649934Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-07T17:53:08.5650491Z SCCACHE_S3_KEY_PREFIX: pull 2024-08-07T17:53:08.5650919Z SHM_SIZE: 2g 2024-08-07T17:53:08.5651931Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:08.5653151Z XLA_CUDA: 2024-08-07T17:53:08.5653714Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:08.5654323Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-07T17:53:08.5654731Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-07T17:53:08.5655141Z DASHBOARD_TAG: 2024-08-07T17:53:08.5655463Z HUGGING_FACE_HUB_TOKEN: 2024-08-07T17:53:08.5655798Z ##[endgroup] 2024-08-07T17:53:08.5678868Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-07T17:53:08.5679355Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-08-07T17:53:08.5679814Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-07T17:53:08.5689095Z +++ nproc --ignore=2 2024-08-07T17:53:08.5704406Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10288745067 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T17:53:19.8009242Z + container_name=27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T17:53:19.8011696Z + grep download.pytorch.org /etc/hosts 2024-08-07T17:53:19.8012779Z + docker exec -i 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-07T17:53:19.8721851Z + echo DOCKER_CONTAINER_ID=27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T17:53:19.8725594Z ++ echo dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl 2024-08-07T17:53:19.8728369Z + docker exec -t 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac sh -c 'pip install dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-07T17:53:20.4139467Z Processing ./dist/torch-2.5.0a0+git016588f-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git016588f) 2024-08-07T17:53:21.5462853Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.13.1) 2024-08-07T17:53:21.5470547Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (4.12.2) 2024-08-07T17:53:21.5475812Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.8.8) 2024-08-07T17:53:21.5481389Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.1.4) 2024-08-07T17:53:21.5486479Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2024.6.1) 2024-08-07T17:53:21.5500946Z Requirement already satisfied: sympy>=1.13.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.13.1) 2024-08-07T17:53:21.5543679Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (3.3.0) 2024-08-07T17:53:21.5629116Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.21.2) 2024-08-07T17:53:21.5678634Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.0->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (1.3.0) 2024-08-07T17:53:21.7006791Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git016588f->torch==2.5.0a0+git016588f) (2.1.5) 2024-08-07T17:53:22.8902812Z Installing collected packages: torch 2024-08-07T17:53:35.7155629Z Successfully installed torch-2.5.0a0+git016588f 2024-08-07T17:53:35.7905090Z ++ dirname .ci/pytorch/test.sh 2024-08-07T17:53:35.7911944Z + source .ci/pytorch/common.sh 2024-08-07T17:53:35.7915658Z +++ dirname .ci/pytorch/common.sh 2024-08-07T17:53:35.7923248Z ++ source .ci/pytorch/common_utils.sh 2024-08-07T17:53:35.7925507Z +++ declare -f -t trap_add 2024-08-07T17:53:35.7932766Z ++ set -ex 2024-08-07T17:53:35.7935419Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:35.7935942Z ++ BUILD_TEST_LIBTORCH=0 2024-08-07T17:53:35.7936684Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T17:53:35.7937218Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-07T17:53:35.7948502Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-07T17:53:35.7948936Z + trap_add cleanup_workspace EXIT 2024-08-07T17:53:35.7949346Z + trap_add_cmd=cleanup_workspace 2024-08-07T17:53:35.7949703Z + shift 2024-08-07T17:53:35.7950005Z + for trap_add_name in "$@" 2024-08-07T17:53:35.7957885Z +++ trap -p EXIT 2024-08-07T17:53:35.7960711Z ++ eval 'extract_trap_cmd ' 2024-08-07T17:53:35.7961086Z +++ extract_trap_cmd 2024-08-07T17:53:35.7961461Z +++ printf '%s\n' '' 2024-08-07T17:53:35.7961813Z ++ printf '%s\n' cleanup_workspace 2024-08-07T17:53:35.7963886Z + trap -- ' 2024-08-07T17:53:35.7964211Z cleanup_workspace' EXIT 2024-08-07T17:53:35.7964623Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-07T17:53:36.3099728Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-07T17:53:36.3118011Z + echo 'Environment variables:' 2024-08-07T17:53:36.3118423Z Environment variables: 2024-08-07T17:53:36.3118748Z + env 2024-08-07T17:53:36.3127323Z INSTALLED_DB=yes 2024-08-07T17:53:36.3127683Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:36.3128549Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:36.3129150Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:36.3129685Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:36.3130260Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:36.3130709Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:36.3131218Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:36.3131711Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:36.3132188Z HOSTNAME=27d39248f8e7 2024-08-07T17:53:36.3132898Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3133721Z GITHUB_ACTION=__self 2024-08-07T17:53:36.3134083Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:36.3136457Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:36.3138676Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:36.3139143Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:36.3139832Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:36.3140187Z TEST_CONFIG=distributed 2024-08-07T17:53:36.3140530Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:36.3140962Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:36.3141395Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:36.3141789Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:36.3142189Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:36.3142586Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:36.3143015Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:36.3143481Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:36.3143832Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:36.3144179Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:36.3144562Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:36.3144983Z XLA_CUDA= 2024-08-07T17:53:36.3145281Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:36.3145824Z *** 2024-08-07T17:53:36.3146133Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:36.3146533Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:36.3146908Z GITHUB_ACTIONS=true 2024-08-07T17:53:36.3147232Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:36.3147666Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:36.3148145Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:36.3148585Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:36.3149039Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:36.3149675Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:36.3150365Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:36.3150985Z UCC_HOME=/usr 2024-08-07T17:53:36.3151313Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:36.3151702Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:36.3152062Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:36.3152474Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:36.3152923Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:36.3153756Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:36.3154171Z SHARD_NUMBER=1 2024-08-07T17:53:36.3154497Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:36.3154840Z HOME=/var/lib/jenkins 2024-08-07T17:53:36.3155223Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:36.3155669Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:36.3156112Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:36.3156579Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:36.3156928Z CUDA_VERSION=11.8.0 2024-08-07T17:53:36.3157305Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:36.3157744Z NUM_TEST_SHARDS=3 2024-08-07T17:53:36.3158082Z UCX_HOME=/usr 2024-08-07T17:53:36.3158542Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:36.3159491Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3160561Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:36.3161582Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3162587Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:36.3163231Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:36.3163588Z DASHBOARD_TAG= 2024-08-07T17:53:36.3163916Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:36.3164336Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:36.3164806Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:36.3165680Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3166518Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:36.3166854Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:36.3167243Z PR_NUMBER=131248 2024-08-07T17:53:36.3167565Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:36.3167894Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:36.3168534Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:36.3168993Z TERM=xterm 2024-08-07T17:53:36.3169295Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:36.3169701Z INSTALLED_VISION=yes 2024-08-07T17:53:36.3170017Z BRANCH=pull/131248 2024-08-07T17:53:36.3170350Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:36.3170751Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:36.3171160Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:36.3171823Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:36.3172556Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:36.3173011Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:36.3173467Z REENABLED_ISSUES= 2024-08-07T17:53:36.3173783Z SHLVL=1 2024-08-07T17:53:36.3174046Z MAX_JOBS=30 2024-08-07T17:53:36.3174355Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:36.3174716Z NVARCH=x86_64 2024-08-07T17:53:36.3175010Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:36.3175458Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:36.3175961Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:36.3176340Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:36.3176914Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:36.3177472Z GITHUB_JOB=test 2024-08-07T17:53:36.3177806Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:36.3178470Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:36.3179001Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:36.3179326Z TD_DISTRIBUTED=False 2024-08-07T17:53:36.3179689Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:36.3180128Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:36.3180517Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:36.3180888Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:36.3181251Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:36.3181601Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:36.3182555Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:36.3183563Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:36.3183966Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:36.3184344Z CI=true 2024-08-07T17:53:36.3184654Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:36.3185056Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:36.3185438Z JOB_ID=28476144889 2024-08-07T17:53:36.3185767Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:36.3186112Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:36.3186511Z GITHUB_ACTION_REF= 2024-08-07T17:53:36.3186885Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:36.3187355Z TEST_SHOWLOCALS=False 2024-08-07T17:53:36.3187698Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:36.3188069Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:36.3188860Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3189659Z NO_TD=False 2024-08-07T17:53:36.3189958Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:36.3190332Z _=/usr/bin/env 2024-08-07T17:53:36.3190755Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-07T17:53:36.3354373Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-07T17:53:36.3355156Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T17:53:36.3355918Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-07T17:53:36.3357166Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-07T17:53:36.3357741Z + BUILD_DIR=build 2024-08-07T17:53:36.3358124Z + BUILD_RENAMED_DIR=build_renamed 2024-08-07T17:53:36.3358524Z + BUILD_BIN_DIR=build/bin 2024-08-07T17:53:36.3358857Z + SHARD_NUMBER=1 2024-08-07T17:53:36.3359175Z + NUM_TEST_SHARDS=3 2024-08-07T17:53:36.3359501Z + export VALGRIND=ON 2024-08-07T17:53:36.3360028Z + VALGRIND=ON 2024-08-07T17:53:36.3360415Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-08-07T17:53:36.3360874Z + [[ 0 == \1 ]] 2024-08-07T17:53:36.3361294Z + [[ False == \1 ]] 2024-08-07T17:53:36.3361961Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-08-07T17:53:36.3362465Z ++ realpath build/custom_test_artifacts 2024-08-07T17:53:36.3371110Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-07T17:53:36.3371753Z + [[ -n '' ]] 2024-08-07T17:53:36.3372083Z + echo 'Environment variables' 2024-08-07T17:53:36.3372450Z Environment variables 2024-08-07T17:53:36.3372778Z + env 2024-08-07T17:53:36.3379524Z INSTALLED_DB=yes 2024-08-07T17:53:36.3380057Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-08-07T17:53:36.3380825Z NVIDIA_VISIBLE_DEVICES=all 2024-08-07T17:53:36.3381523Z NV_NVML_DEV_VERSION=11.8.86-1 2024-08-07T17:53:36.3382260Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T17:53:36.3382839Z CONTINUE_THROUGH_ERROR=False 2024-08-07T17:53:36.3383291Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-08-07T17:53:36.3384169Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:36.3384668Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-08-07T17:53:36.3385132Z HOSTNAME=27d39248f8e7 2024-08-07T17:53:36.3385842Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3387426Z GITHUB_ACTION=__self 2024-08-07T17:53:36.3388009Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-07T17:53:36.3390942Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-08-07T17:53:36.3392945Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-08-07T17:53:36.3393798Z NV_NVTX_VERSION=11.8.86-1 2024-08-07T17:53:36.3394164Z GITHUB_RUN_NUMBER=234358 2024-08-07T17:53:36.3394509Z TEST_CONFIG=distributed 2024-08-07T17:53:36.3394881Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-07T17:53:36.3395316Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-07T17:53:36.3395727Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-08-07T17:53:36.3396154Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-08-07T17:53:36.3396559Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-08-07T17:53:36.3396929Z GITHUB_TRIGGERING_ACTOR=zdevito 2024-08-07T17:53:36.3397368Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-07T17:53:36.3397809Z GITHUB_REF_TYPE=branch 2024-08-07T17:53:36.3398163Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-07T17:53:36.3398533Z NCCL_VERSION=2.16.2-1 2024-08-07T17:53:36.3398896Z BASE_SHA=6ce09a9bb33e4011761558032e2165ad7b49fb68 2024-08-07T17:53:36.3399332Z XLA_CUDA= 2024-08-07T17:53:36.3399632Z HUGGING_FACE_HUB_TOKEN= 2024-08-07T17:53:36.3400059Z *** 2024-08-07T17:53:36.3400363Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-07T17:53:36.3400763Z GITHUB_REPOSITORY_ID=65600975 2024-08-07T17:53:36.3401113Z GITHUB_ACTIONS=true 2024-08-07T17:53:36.3401461Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T17:53:36.3401898Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-08-07T17:53:36.3402363Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-08-07T17:53:36.3402833Z SHA1=016588f53c6904b840aa56aa86f95460b4d9c996 2024-08-07T17:53:36.3403289Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-07T17:53:36.3403735Z GITHUB_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:36.3404411Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/131248/merge 2024-08-07T17:53:36.3405035Z UCC_HOME=/usr 2024-08-07T17:53:36.3405347Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-08-07T17:53:36.3405753Z VERBOSE_TEST_LOGS=False 2024-08-07T17:53:36.3406111Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-07T17:53:36.3406505Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-08-07T17:53:36.3407201Z GITHUB_REF=refs/pull/131248/merge 2024-08-07T17:53:36.3407603Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-08-07T17:53:36.3407963Z SHARD_NUMBER=1 2024-08-07T17:53:36.3408292Z GITHUB_REF_PROTECTED=false 2024-08-07T17:53:36.3408655Z HOME=/var/lib/jenkins 2024-08-07T17:53:36.3409013Z GITHUB_API_URL=https://api.github.com 2024-08-07T17:53:36.3409452Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-07T17:53:36.3409916Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-07T17:53:36.3410365Z SCCACHE_S3_KEY_PREFIX=pull 2024-08-07T17:53:36.3410726Z CUDA_VERSION=11.8.0 2024-08-07T17:53:36.3411099Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-08-07T17:53:36.3411523Z NUM_TEST_SHARDS=3 2024-08-07T17:53:36.3411842Z UCX_HOME=/usr 2024-08-07T17:53:36.3412331Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-08-07T17:53:36.3413287Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3414343Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) 2024-08-07T17:53:36.3415379Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3416380Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-07T17:53:36.3417009Z GITHUB_EVENT_NAME=pull_request 2024-08-07T17:53:36.3417500Z DASHBOARD_TAG= 2024-08-07T17:53:36.3417842Z GITHUB_RUN_ID=10288745067 2024-08-07T17:53:36.3418236Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-08-07T17:53:36.3418728Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-08-07T17:53:36.3419591Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3420426Z GITHUB_ACTOR=zdevito 2024-08-07T17:53:36.3420777Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-08-07T17:53:36.3421161Z PR_NUMBER=131248 2024-08-07T17:53:36.3421476Z GITHUB_RUN_ATTEMPT=1 2024-08-07T17:53:36.3421804Z VALGRIND=ON 2024-08-07T17:53:36.3422120Z ANACONDA_PYTHON_VERSION=3.10 2024-08-07T17:53:36.3422546Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-07T17:53:36.3423002Z TERM=xterm 2024-08-07T17:53:36.3423322Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-08-07T17:53:36.3423713Z INSTALLED_VISION=yes 2024-08-07T17:53:36.3424046Z BRANCH=pull/131248 2024-08-07T17:53:36.3424374Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-07T17:53:36.3424776Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-07T17:53:36.3425197Z CUDA_PATH=/usr/local/cuda 2024-08-07T17:53:36.3425840Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-07T17:53:36.3426570Z GITHUB_SERVER_URL=https://github.com 2024-08-07T17:53:36.3427049Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-07T17:53:36.3427488Z REENABLED_ISSUES= 2024-08-07T17:53:36.3427802Z SHLVL=1 2024-08-07T17:53:36.3428080Z MAX_JOBS=30 2024-08-07T17:53:36.3428392Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-08-07T17:53:36.3428756Z NVARCH=x86_64 2024-08-07T17:53:36.3429065Z GITHUB_ACTOR_ID=370202 2024-08-07T17:53:36.3429488Z GITHUB_WORKFLOW_SHA=f779f6b7738020e244184bded4026b37de3f9f24 2024-08-07T17:53:36.3429994Z GITHUB_REF_NAME=131248/merge 2024-08-07T17:53:36.3430390Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-08-07T17:53:36.3430954Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-07T17:53:36.3431520Z GITHUB_JOB=test 2024-08-07T17:53:36.3431881Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-08-07T17:53:36.3432397Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-07T17:53:36.3432908Z NO_TEST_TIMEOUT=False 2024-08-07T17:53:36.3433507Z TD_DISTRIBUTED=False 2024-08-07T17:53:36.3433872Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-08-07T17:53:36.3434311Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-07T17:53:36.3434717Z NV_NVPROF_VERSION=11.8.87-1 2024-08-07T17:53:36.3435073Z GITHUB_RETENTION_DAYS=90 2024-08-07T17:53:36.3435610Z OPENSSL_DIR=/opt/openssl 2024-08-07T17:53:36.3435971Z GITHUB_ACTION_REPOSITORY= 2024-08-07T17:53:36.3436904Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:36.3437902Z GITHUB_BASE_REF=gh/zdevito/267/base 2024-08-07T17:53:36.3438319Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-07T17:53:36.3438681Z CI=true 2024-08-07T17:53:36.3438997Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-08-07T17:53:36.3439410Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-07T17:53:36.3439761Z JOB_ID=28476144889 2024-08-07T17:53:36.3440085Z INSTALLED_PROTOBUF=yes 2024-08-07T17:53:36.3440444Z GITHUB_HEAD_REF=gh/zdevito/267/head 2024-08-07T17:53:36.3440816Z GITHUB_ACTION_REF= 2024-08-07T17:53:36.3441208Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-07T17:53:36.3441651Z TEST_SHOWLOCALS=False 2024-08-07T17:53:36.3441994Z GITHUB_WORKFLOW=pull 2024-08-07T17:53:36.3442353Z DEBIAN_FRONTEND=noninteractive 2024-08-07T17:53:36.3443131Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_833dff6c-8028-4ac5-ae9b-8e48aa9c984a 2024-08-07T17:53:36.3443921Z NO_TD=False 2024-08-07T17:53:36.3444241Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-07T17:53:36.3444593Z _=/usr/bin/env 2024-08-07T17:53:36.3444908Z + echo 'Testing pytorch' 2024-08-07T17:53:36.3445259Z Testing pytorch 2024-08-07T17:53:36.3445573Z + export LANG=C.UTF-8 2024-08-07T17:53:36.3446013Z + LANG=C.UTF-8 2024-08-07T17:53:36.3446348Z + PR_NUMBER=131248 2024-08-07T17:53:36.3446668Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-07T17:53:36.3447095Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-07T17:53:36.3447573Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:36.3448016Z + [[ distributed == \s\l\o\w ]] 2024-08-07T17:53:36.3448488Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-08-07T17:53:36.3449045Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T17:53:36.3449535Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:36.3449990Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-07T17:53:36.3450411Z + [[ distributed == *crossref* ]] 2024-08-07T17:53:36.3450833Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-08-07T17:53:36.3451333Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-08-07T17:53:36.3453621Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:36.3454134Z + pip_install --user ninja==1.10.2 2024-08-07T17:53:36.3454613Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-07T17:53:36.8656261Z Collecting ninja==1.10.2 2024-08-07T17:53:36.8865271Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-07T17:53:36.9050215Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-07T17:53:38.1094247Z Installing collected packages: ninja 2024-08-07T17:53:38.1192952Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-07T17:53:38.1194492Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-07T17:53:38.1250724Z Successfully installed ninja-1.10.2 2024-08-07T17:53:38.1981064Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:38.1982962Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:38.1984145Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:38.1984615Z + install_tlparse 2024-08-07T17:53:38.1984951Z + pip_install --user tlparse==0.3.7 2024-08-07T17:53:38.1985434Z + pip install --progress-bar off --user tlparse==0.3.7 2024-08-07T17:53:38.7524726Z Collecting tlparse==0.3.7 2024-08-07T17:53:38.7715948Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-08-07T17:53:38.7818044Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-07T17:53:40.0263501Z Installing collected packages: tlparse 2024-08-07T17:53:40.0760738Z Successfully installed tlparse-0.3.7 2024-08-07T17:53:40.1470108Z ++ python -m site --user-base 2024-08-07T17:53:40.1702579Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-07T17:53:40.1704619Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-08-07T17:53:40.1705147Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-08-07T17:53:40.1705680Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-08-07T17:53:40.1706377Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-08-07T17:53:40.1707635Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-08-07T17:53:40.1708296Z + cd test 2024-08-07T17:53:40.1708729Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-07T17:53:42.2994630Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-07T17:53:42.2995623Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-07T17:53:42.3001027Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-07T17:53:42.3001492Z + [[ distributed == *dynamo_eager* ]] 2024-08-07T17:53:42.3001896Z + [[ distributed == *aot_eager* ]] 2024-08-07T17:53:42.3002301Z + [[ distributed == *aot_inductor* ]] 2024-08-07T17:53:42.3002716Z + [[ distributed == *inductor* ]] 2024-08-07T17:53:42.3003093Z + [[ distributed == *dynamic* ]] 2024-08-07T17:53:42.3003484Z + [[ distributed == *cpu* ]] 2024-08-07T17:53:42.3003889Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-07T17:53:42.3042116Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:42.3042682Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-08-07T17:53:42.3045154Z + cd test 2024-08-07T17:53:42.3045568Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-07T17:53:44.2202786Z PyTorch built with: 2024-08-07T17:53:44.2203196Z - GCC 9.4 2024-08-07T17:53:44.2203488Z - C++ Version: 201703 2024-08-07T17:53:44.2204237Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:44.2205151Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:44.2205701Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:44.2206153Z - LAPACK is enabled (usually provided by MKL) 2024-08-07T17:53:44.2206598Z - NNPACK is enabled 2024-08-07T17:53:44.2206935Z - CPU capability usage: AVX2 2024-08-07T17:53:44.2207315Z - CUDA Runtime 11.8 2024-08-07T17:53:44.2207768Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-08-07T17:53:44.2208270Z - CuDNN 90.1 2024-08-07T17:53:44.2208577Z - Magma 2.6.1 2024-08-07T17:53:44.2214265Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-07T17:53:44.2220570Z 2024-08-07T17:53:44.4756371Z + cd test 2024-08-07T17:53:44.4756869Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-07T17:53:46.3809967Z ATen/Parallel: 2024-08-07T17:53:46.3810392Z at::get_num_threads() : 16 2024-08-07T17:53:46.3810810Z at::get_num_interop_threads() : 16 2024-08-07T17:53:46.3811230Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-07T17:53:46.3811617Z omp_get_max_threads() : 16 2024-08-07T17:53:46.3812337Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-07T17:53:46.3813125Z mkl_get_max_threads() : 16 2024-08-07T17:53:46.3813617Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-07T17:53:46.3814199Z std::thread::hardware_concurrency() : 32 2024-08-07T17:53:46.3814631Z Environment variables: 2024-08-07T17:53:46.3814971Z OMP_NUM_THREADS : [not set] 2024-08-07T17:53:46.3815347Z MKL_NUM_THREADS : [not set] 2024-08-07T17:53:46.3815725Z ATen parallel backend: OpenMP 2024-08-07T17:53:46.3816329Z 2024-08-07T17:53:46.6275232Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-08-07T17:53:46.6275769Z + [[ distributed == *backward* ]] 2024-08-07T17:53:46.6276177Z + [[ distributed == *xla* ]] 2024-08-07T17:53:46.6276567Z + [[ distributed == *executorch* ]] 2024-08-07T17:53:46.6276971Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-07T17:53:46.6277468Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-08-07T17:53:46.6277959Z + [[ distributed == distributed ]] 2024-08-07T17:53:46.6278337Z + test_distributed 2024-08-07T17:53:46.6278738Z + echo 'Testing distributed python tests' 2024-08-07T17:53:46.6279189Z Testing distributed python tests 2024-08-07T17:53:46.6279706Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-08-07T17:53:46.7573203Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T17:53:46.7574253Z import pkg_resources 2024-08-07T17:53:51.2632395Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T17:53:51.3187593Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T17:53:51.3646354Z Ignoring disabled issues: [''] 2024-08-07T17:53:51.3786123Z Found test times from artifacts 2024-08-07T17:53:51.4317517Z Found test times from artifacts 2024-08-07T17:53:51.4334515Z Running 25% of tests based on TD 2024-08-07T17:53:51.4456943Z Running parallel tests on 2 processes 2024-08-07T17:53:51.4457890Z Name: tests to run (est. time: 11.92min) 2024-08-07T17:53:51.4458306Z Serial tests (13): 2024-08-07T17:53:51.4458711Z distributed/_tensor/test_dtensor_compile 1/1 2024-08-07T17:53:51.4459196Z distributed/_tensor/test_dtensor_ops 1/1 2024-08-07T17:53:51.4459632Z distributed/fsdp/test_fsdp_core 2/2 2024-08-07T17:53:51.4460085Z distributed/fsdp/test_fsdp_fx 1/1 2024-08-07T17:53:51.4460580Z distributed/_tensor/test_experimental_ops 1/1 2024-08-07T17:53:51.4461033Z distributed/test_c10d_pypg 1/1 2024-08-07T17:53:51.4461459Z distributed/pipelining/test_stage 1/1 2024-08-07T17:53:51.4461913Z distributed/_tensor/test_xla_integration 1/1 2024-08-07T17:53:51.4462407Z distributed/checkpoint/test_save_load_api 1/1 2024-08-07T17:53:51.4462956Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-08-07T17:53:51.4463497Z distributed/_composable/test_checkpoint 1/1 2024-08-07T17:53:51.4464275Z distributed/tensor/parallel/test_tp_style 1/1 2024-08-07T17:53:51.4464825Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 2024-08-07T17:53:51.4465329Z Parallel tests (0): 2024-08-07T17:53:51.4465695Z Name: excluded (est. time: 72.59min) 2024-08-07T17:53:51.4466090Z Serial tests (55): 2024-08-07T17:53:51.4466460Z distributed/test_distributed_spawn 1/9 2024-08-07T17:53:51.4466906Z distributed/test_distributed_spawn 4/9 2024-08-07T17:53:51.4467360Z distributed/test_distributed_spawn 7/9 2024-08-07T17:53:51.4467796Z distributed/test_c10d_gloo 1/1 2024-08-07T17:53:51.4468229Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-08-07T17:53:51.4468762Z distributed/rpc/cuda/test_tensorpipe_agent 2/2 2024-08-07T17:53:51.4469321Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-08-07T17:53:51.4469842Z distributed/checkpoint/test_state_dict 1/1 2024-08-07T17:53:51.4470367Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-08-07T17:53:51.4470931Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-08-07T17:53:51.4471417Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-08-07T17:53:51.4471872Z distributed/test_pg_wrapper 1/1 2024-08-07T17:53:51.4472296Z distributed/test_functional_api 1/1 2024-08-07T17:53:51.4472726Z distributed/_tensor/test_optimizers 1/1 2024-08-07T17:53:51.4473500Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 2024-08-07T17:53:51.4474148Z distributed/test_store 1/1 2024-08-07T17:53:51.4474579Z distributed/test_c10d_object_collectives 1/1 2024-08-07T17:53:51.4475057Z distributed/_composable/test_compose 1/1 2024-08-07T17:53:51.4475606Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 2024-08-07T17:53:51.4476173Z distributed/elastic/multiprocessing/api_test 1/1 2024-08-07T17:53:51.4476670Z distributed/test_c10d_spawn_gloo 1/1 2024-08-07T17:53:51.4477126Z distributed/checkpoint/test_checkpoint 1/1 2024-08-07T17:53:51.4477665Z distributed/_composable/fully_shard/test_fully_shard_init 1/1 2024-08-07T17:53:51.4478225Z distributed/_tensor/test_view_ops 1/1 2024-08-07T17:53:51.4478733Z distributed/checkpoint/test_file_system_checkpoint 1/1 2024-08-07T17:53:51.4479300Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-08-07T17:53:51.4479864Z distributed/checkpoint/test_state_dict_utils 1/1 2024-08-07T17:53:51.4480646Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-08-07T17:53:51.4481232Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2024-08-07T17:53:51.4481752Z distributed/fsdp/test_fsdp_memory 1/1 2024-08-07T17:53:51.4482302Z distributed/_composable/fully_shard/test_fully_shard_runtime 1/1 2024-08-07T17:53:51.4482869Z distributed/checkpoint/test_format_utils 1/1 2024-08-07T17:53:51.4483402Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-08-07T17:53:51.4483923Z distributed/checkpoint/test_fsspec 1/1 2024-08-07T17:53:51.4484390Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-08-07T17:53:51.4484883Z distributed/fsdp/test_shard_utils 1/1 2024-08-07T17:53:51.4485388Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-08-07T17:53:51.4485953Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-08-07T17:53:51.4486501Z distributed/elastic/utils/distributed_test 1/1 2024-08-07T17:53:51.4487044Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2024-08-07T17:53:51.4487576Z distributed/elastic/timer/local_timer_test 1/1 2024-08-07T17:53:51.4488145Z distributed/_composable/fully_shard/test_fully_shard_util 1/1 2024-08-07T17:53:51.4488701Z distributed/fsdp/test_fsdp_traversal 1/1 2024-08-07T17:53:51.4489163Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-08-07T17:53:51.4489694Z distributed/tensor/parallel/test_tp_random_state 1/1 2024-08-07T17:53:51.4490185Z distributed/test_fake_pg 1/1 2024-08-07T17:53:51.4490559Z distributed/test_nccl 1/1 2024-08-07T17:53:51.4490977Z distributed/_tensor/debug/test_comm_mode 1/1 2024-08-07T17:53:51.4491650Z distributed/elastic/events/lib_test 1/1 2024-08-07T17:53:51.4492090Z distributed/test_control_collectives 1/1 2024-08-07T17:53:51.4492549Z distributed/elastic/metrics/api_test 1/1 2024-08-07T17:53:51.4493106Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-08-07T17:53:51.4493645Z distributed/test_collective_utils 1/1 2024-08-07T17:53:51.4494090Z distributed/rpc/test_faulty_agent 1/1 2024-08-07T17:53:51.4494541Z distributed/elastic/timer/api_test 1/1 2024-08-07T17:53:51.4494947Z Parallel tests (0): 2024-08-07T17:53:51.4495437Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T17:53:51.4530884Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-08-07 17:53:51.452757] 2024-08-07T17:53:51.4535183Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:53:51.453148] 2024-08-07T17:54:24.5447662Z 2024-08-07T17:54:24.5448939Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_e972b11e702c777a_.log 2024-08-07T17:54:24.5463683Z Running 26 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-08-07T17:54:24.5483666Z 2024-08-07T17:54:24.5484029Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-08-07 17:54:24.545293] 2024-08-07T17:54:24.5485531Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:24.545743] 2024-08-07T17:54:29.3772974Z 2024-08-07T17:54:29.3774363Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_84600b34e8be0a46_.log 2024-08-07T17:54:29.3775473Z Running 0 items in this shard: 2024-08-07T17:54:29.3775726Z 2024-08-07T17:54:29.3779146Z Running distributed/fsdp/test_fsdp_core 2/2 ... [2024-08-07 17:54:29.377618] 2024-08-07T17:54:29.3784022Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 17:54:29.378055] 2024-08-07T18:03:30.8530368Z 2024-08-07T18:03:30.8533907Z distributed/fsdp/test_fsdp_core 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_2.2_776d559e857a2cee_.log 2024-08-07T18:03:30.8560604Z Running 38 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_optim_step_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_delayed_reduce_scatter_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_mixture_of_experts_with_delay_before_free_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_always_wrap_model_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_false_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_false_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_no_shard, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDP::test_transformer_offload_true_shard_grad_op, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_after_state_dict, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_pre_backward_hook_registration_cuda_first_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_core.py::TestHooks::test_register_functions_called_cuda_first_True_mixed_precision_True 2024-08-07T18:03:30.8581377Z 2024-08-07T18:03:30.8581686Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-08-07 18:03:30.856332] 2024-08-07T18:03:30.8583138Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:30.857651] 2024-08-07T18:03:34.7299334Z 2024-08-07T18:03:34.7300667Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_25c55fd709ede464_.log 2024-08-07T18:03:34.7302094Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2024-08-07T18:03:34.7302756Z 2024-08-07T18:03:34.7306190Z Running distributed/_tensor/test_experimental_ops 1/1 ... [2024-08-07 18:03:34.730258] 2024-08-07T18:03:34.7310917Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:34.730715] 2024-08-07T18:03:52.0812170Z 2024-08-07T18:03:52.0813789Z distributed/_tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_experimental_ops_1.1_d42eb07d3e9a5ddb_.log 2024-08-07T18:03:52.0816087Z Running 3 items in this shard: test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/_tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2024-08-07T18:03:52.0817556Z 2024-08-07T18:03:52.0817854Z Running distributed/test_c10d_pypg 1/1 ... [2024-08-07 18:03:52.081439] 2024-08-07T18:03:52.0822205Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:03:52.081843] 2024-08-07T18:06:34.4625023Z 2024-08-07T18:06:34.4626126Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_398955911b234d20_.log 2024-08-07T18:06:34.4648378Z Running 42 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input 2024-08-07T18:06:34.4669628Z 2024-08-07T18:06:34.4669961Z Running distributed/pipelining/test_stage 1/1 ... [2024-08-07 18:06:34.462797] 2024-08-07T18:06:34.4671471Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:34.463201] 2024-08-07T18:06:44.1978371Z 2024-08-07T18:06:44.1980101Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_5bb76dcae5892632_.log 2024-08-07T18:06:44.1983449Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-08-07T18:06:44.1988318Z Running 6 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0 2024-08-07T18:06:44.1990773Z 2024-08-07T18:06:44.1991127Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-08-07 18:06:44.198141] 2024-08-07T18:06:44.1992681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:44.198595] 2024-08-07T18:06:48.0708485Z 2024-08-07T18:06:48.0709998Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_89e9116023171a9b_.log 2024-08-07T18:06:48.0712758Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-08-07T18:06:48.0714860Z 2024-08-07T18:06:48.0715254Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2024-08-07 18:06:48.071091] 2024-08-07T18:06:48.0719516Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:48.071551] 2024-08-07T18:06:55.5006618Z 2024-08-07T18:06:55.5008030Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_d8903ddfc4ed3cba_.log 2024-08-07T18:06:55.5009646Z Running 1 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect 2024-08-07T18:06:55.5010316Z 2024-08-07T18:06:55.5012868Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-08-07 18:06:55.500927] 2024-08-07T18:06:55.5017224Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:55.501336] 2024-08-07T18:06:59.4738322Z 2024-08-07T18:06:59.4739711Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_ed12b2489c85b959_.log 2024-08-07T18:06:59.4743836Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-08-07T18:06:59.4746684Z 2024-08-07T18:06:59.4747070Z Running distributed/_composable/test_checkpoint 1/1 ... [2024-08-07 18:06:59.474088] 2024-08-07T18:06:59.4748645Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:06:59.474494] 2024-08-07T18:07:03.6472107Z 2024-08-07T18:07:03.6474013Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_50b60c47f640c01f_.log 2024-08-07T18:07:03.6477651Z Running 6 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_checkpoint_kwargs, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2024-08-07T18:07:03.6480388Z 2024-08-07T18:07:03.6480782Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2024-08-07 18:07:03.647622] 2024-08-07T18:07:03.6484332Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:07:03.648035] 2024-08-07T18:07:48.9556607Z 2024-08-07T18:07:48.9560052Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_34d7503e3952bc28_.log 2024-08-07T18:07:48.9566044Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style 2024-08-07T18:07:48.9571448Z 2024-08-07T18:07:48.9571874Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2024-08-07 18:07:48.955939] 2024-08-07T18:07:48.9573559Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-07 18:07:48.956342] 2024-08-07T18:07:52.5296421Z 2024-08-07T18:07:52.5298611Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_556ae0b1b69d26fb_.log 2024-08-07T18:07:52.5300078Z Running 0 items in this shard: 2024-08-07T18:07:52.5300338Z 2024-08-07T18:07:53.5610376Z 2024-08-07T18:07:53.5611171Z real 14m6.933s 2024-08-07T18:07:53.5611545Z user 19m35.775s 2024-08-07T18:07:53.5611861Z sys 8m42.595s 2024-08-07T18:07:53.5612168Z + assert_git_not_dirty 2024-08-07T18:07:53.5612568Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-08-07T18:07:53.5613090Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-08-07T18:07:53.5616689Z ++ git status --porcelain 2024-08-07T18:07:53.5617460Z ++ grep -v '?? third_party' 2024-08-07T18:07:56.4435955Z ++ true 2024-08-07T18:07:56.4436836Z + git_status= 2024-08-07T18:07:56.4437165Z + [[ -n '' ]] 2024-08-07T18:07:56.4437545Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T18:07:56.4437999Z + [[ 1 == 1 ]] 2024-08-07T18:07:56.4438327Z + echo 'Testing distributed C++ tests' 2024-08-07T18:07:56.4438783Z Testing distributed C++ tests 2024-08-07T18:07:56.4442151Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:07:56.4454648Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:07:56.4464995Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:07:56.4465783Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:07:56.4466487Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-08-07T18:07:56.5777323Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:07:56.5778725Z import pkg_resources 2024-08-07T18:08:01.0067523Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:08:01.0069678Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:08:01.0205423Z Found test times from artifacts 2024-08-07T18:08:01.0730355Z Found test times from artifacts 2024-08-07T18:08:01.0746249Z Running 25% of tests based on TD 2024-08-07T18:08:01.0749394Z Running parallel tests on 2 processes 2024-08-07T18:08:01.0749847Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:08:01.0750231Z Serial tests (1): 2024-08-07T18:08:01.0750568Z cpp/FileStoreTest 1/1 2024-08-07T18:08:01.0750934Z Parallel tests (0): 2024-08-07T18:08:01.0751266Z Name: excluded (est. time: 0.0min) 2024-08-07T18:08:01.0751648Z Serial tests (0): 2024-08-07T18:08:01.0751954Z Parallel tests (0): 2024-08-07T18:08:01.0752460Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:08:01.0851395Z Running cpp/FileStoreTest 1/1 ... [2024-08-07 18:08:01.084814] 2024-08-07T18:08:01.0858751Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-8c53b69cacee2137.xml', '-x', '--reruns=2'] ... [2024-08-07 18:08:01.085441] 2024-08-07T18:08:04.9091395Z 2024-08-07T18:08:04.9092619Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_93f074ed2367ac7c_.log 2024-08-07T18:08:04.9093423Z 2024-08-07T18:08:05.8690734Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-08-07T18:08:06.0014731Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:08:06.0015854Z import pkg_resources 2024-08-07T18:08:10.5406600Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:08:10.5410076Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:08:10.5558297Z Found test times from artifacts 2024-08-07T18:08:10.6092278Z Found test times from artifacts 2024-08-07T18:08:10.6112865Z Running 25% of tests based on TD 2024-08-07T18:08:10.6117256Z Running parallel tests on 2 processes 2024-08-07T18:08:10.6117694Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:08:10.6118136Z Serial tests (1): 2024-08-07T18:08:10.6118460Z cpp/HashStoreTest 1/1 2024-08-07T18:08:10.6118823Z Parallel tests (0): 2024-08-07T18:08:10.6119179Z Name: excluded (est. time: 0.0min) 2024-08-07T18:08:10.6119551Z Serial tests (0): 2024-08-07T18:08:10.6119873Z Parallel tests (0): 2024-08-07T18:08:10.6120369Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:08:10.6231766Z Running cpp/HashStoreTest 1/1 ... [2024-08-07 18:08:10.622813] 2024-08-07T18:08:10.6238482Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-27b449e7aa205365.xml', '-x', '--reruns=2'] ... [2024-08-07 18:08:10.623439] 2024-08-07T18:08:14.0959916Z 2024-08-07T18:08:14.0961093Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_fc5c5ea7539d30b1_.log 2024-08-07T18:08:14.0961872Z 2024-08-07T18:08:15.0619134Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-08-07T18:08:15.1944669Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:08:15.1946147Z import pkg_resources 2024-08-07T18:08:19.6624821Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:08:19.6627901Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:08:19.6762892Z Found test times from artifacts 2024-08-07T18:08:19.7290646Z Found test times from artifacts 2024-08-07T18:08:19.7307004Z Running 25% of tests based on TD 2024-08-07T18:08:19.7310086Z Running parallel tests on 2 processes 2024-08-07T18:08:19.7310517Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:08:19.7310921Z Serial tests (1): 2024-08-07T18:08:19.7311261Z cpp/TCPStoreTest 1/1 2024-08-07T18:08:19.7311604Z Parallel tests (0): 2024-08-07T18:08:19.7311962Z Name: excluded (est. time: 0.0min) 2024-08-07T18:08:19.7312327Z Serial tests (0): 2024-08-07T18:08:19.7312652Z Parallel tests (0): 2024-08-07T18:08:19.7313483Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:08:19.7413705Z Running cpp/TCPStoreTest 1/1 ... [2024-08-07 18:08:19.740984] 2024-08-07T18:08:19.7420537Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-6566576ce797df48.xml', '-x', '--reruns=2'] ... [2024-08-07 18:08:19.741558] 2024-08-07T18:08:35.7899664Z 2024-08-07T18:08:35.7900836Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_4c77fc0945f2a509_.log 2024-08-07T18:08:35.7901603Z 2024-08-07T18:08:36.7914358Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-08-07T18:08:36.7917384Z ++ command -v mpiexec 2024-08-07T18:08:36.7921169Z + MPIEXEC=/usr/bin/mpiexec 2024-08-07T18:08:36.7921552Z + [[ -n /usr/bin/mpiexec ]] 2024-08-07T18:08:36.7922300Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-08-07T18:08:36.7923434Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-08-07T18:08:36.7924493Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-08-07T18:08:39.9385964Z Test successful 2024-08-07T18:08:39.9434623Z Test successful 2024-08-07T18:08:40.0980138Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-08-07T18:08:40.2273746Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:08:40.2274816Z import pkg_resources 2024-08-07T18:08:44.7407909Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:08:44.7409734Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:08:44.7545763Z Found test times from artifacts 2024-08-07T18:08:44.8073542Z Found test times from artifacts 2024-08-07T18:08:44.8089773Z Running 25% of tests based on TD 2024-08-07T18:08:44.8092731Z Running parallel tests on 2 processes 2024-08-07T18:08:44.8093169Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:08:44.8093596Z Serial tests (1): 2024-08-07T18:08:44.8093961Z cpp/ProcessGroupGlooTest 1/1 2024-08-07T18:08:44.8094358Z Parallel tests (0): 2024-08-07T18:08:44.8094691Z Name: excluded (est. time: 0.0min) 2024-08-07T18:08:44.8095075Z Serial tests (0): 2024-08-07T18:08:44.8095400Z Parallel tests (0): 2024-08-07T18:08:44.8095867Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:08:44.8191692Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-08-07 18:08:44.818821] 2024-08-07T18:08:44.8198155Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b41769771e8f0e9f.xml', '-x', '--reruns=2'] ... [2024-08-07 18:08:44.819423] 2024-08-07T18:08:54.0054564Z 2024-08-07T18:08:54.0055972Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_a40ff419bac27115_.log 2024-08-07T18:08:54.0056881Z 2024-08-07T18:08:55.0220834Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-08-07T18:08:55.1546482Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:08:55.1547555Z import pkg_resources 2024-08-07T18:08:59.6332878Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:08:59.6336309Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:08:59.6471559Z Found test times from artifacts 2024-08-07T18:08:59.7002679Z Found test times from artifacts 2024-08-07T18:08:59.7019055Z Running 25% of tests based on TD 2024-08-07T18:08:59.7022001Z Running parallel tests on 2 processes 2024-08-07T18:08:59.7022752Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:08:59.7023216Z Serial tests (1): 2024-08-07T18:08:59.7023576Z cpp/ProcessGroupNCCLTest 1/1 2024-08-07T18:08:59.7023962Z Parallel tests (0): 2024-08-07T18:08:59.7024323Z Name: excluded (est. time: 0.0min) 2024-08-07T18:08:59.7024714Z Serial tests (0): 2024-08-07T18:08:59.7025027Z Parallel tests (0): 2024-08-07T18:08:59.7025514Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:08:59.7121243Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-08-07 18:08:59.711759] 2024-08-07T18:08:59.7127557Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-eb7317281ff6f326.xml', '-x', '--reruns=2'] ... [2024-08-07 18:08:59.712332] 2024-08-07T18:09:26.2282496Z 2024-08-07T18:09:26.2283744Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_c1b066b14b96c72b_.log 2024-08-07T18:09:26.2284829Z 2024-08-07T18:09:27.1954422Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-08-07T18:09:27.3273497Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:09:27.3274577Z import pkg_resources 2024-08-07T18:09:31.8249447Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:09:31.8252606Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:09:31.8388660Z Found test times from artifacts 2024-08-07T18:09:31.8918938Z Found test times from artifacts 2024-08-07T18:09:31.8935911Z Running 25% of tests based on TD 2024-08-07T18:09:31.8939676Z Running parallel tests on 2 processes 2024-08-07T18:09:31.8940109Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:09:31.8940518Z Serial tests (1): 2024-08-07T18:09:31.8940891Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-08-07T18:09:31.8941300Z Parallel tests (0): 2024-08-07T18:09:31.8941652Z Name: excluded (est. time: 0.0min) 2024-08-07T18:09:31.8942034Z Serial tests (0): 2024-08-07T18:09:31.8942344Z Parallel tests (0): 2024-08-07T18:09:31.8942832Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:09:31.9041233Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-08-07 18:09:31.903804] 2024-08-07T18:09:31.9047950Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-f6e6ba7ce346dd12.xml', '-x', '--reruns=2'] ... [2024-08-07 18:09:31.904403] 2024-08-07T18:09:44.1964845Z 2024-08-07T18:09:44.1966554Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_9b369eca72f27c88_.log 2024-08-07T18:09:44.1967545Z 2024-08-07T18:09:45.1721466Z + [[ 1 == 1 ]] 2024-08-07T18:09:45.1721866Z + test_rpc 2024-08-07T18:09:45.1722169Z + echo 'Testing RPC C++ tests' 2024-08-07T18:09:45.1722564Z Testing RPC C++ tests 2024-08-07T18:09:45.1726262Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:09:45.1739290Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:09:45.1750687Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-07T18:09:45.1751394Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-08-07T18:09:45.3044505Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-07T18:09:45.3045548Z import pkg_resources 2024-08-07T18:09:49.7343144Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-08-07T18:09:49.7345717Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-07T18:09:49.7482320Z Found test times from artifacts 2024-08-07T18:09:49.8008816Z Found test times from artifacts 2024-08-07T18:09:49.8026354Z Running 25% of tests based on TD 2024-08-07T18:09:49.8029612Z Running parallel tests on 2 processes 2024-08-07T18:09:49.8030398Z Name: tests to run (est. time: 0.0min) 2024-08-07T18:09:49.8031134Z Serial tests (1): 2024-08-07T18:09:49.8031789Z cpp/test_cpp_rpc 1/1 2024-08-07T18:09:49.8032430Z Parallel tests (0): 2024-08-07T18:09:49.8033116Z Name: excluded (est. time: 0.0min) 2024-08-07T18:09:49.8034214Z Serial tests (0): 2024-08-07T18:09:49.8034761Z Parallel tests (0): 2024-08-07T18:09:49.8035647Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-07T18:09:49.8131334Z Running cpp/test_cpp_rpc 1/1 ... [2024-08-07 18:09:49.812732] 2024-08-07T18:09:49.8139335Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-74d72ab201c085c5.xml', '-x', '--reruns=2'] ... [2024-08-07 18:09:49.813409] 2024-08-07T18:09:55.8416677Z 2024-08-07T18:09:55.8417991Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_cdeb357e55ec592b_.log 2024-08-07T18:09:55.8420948Z 2024-08-07T18:09:56.8209344Z + cleanup_workspace 2024-08-07T18:09:56.8210050Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-07T18:09:56.8211444Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-07T18:09:56.8212237Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-07T18:09:56.8212829Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-07T18:09:56.8213512Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-07T18:09:56.8214263Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-07T18:09:56.8214856Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-07T18:09:57.3469069Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-07T18:09:57.3469606Z cat test/**/*_toprint.log || true 2024-08-07T18:09:57.3478238Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.3478738Z env: 2024-08-07T18:09:57.3479019Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.3479484Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.3480173Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.3480775Z ##[endgroup] 2024-08-07T18:09:57.3551872Z cat: test/**/*_toprint.log: No such file or directory 2024-08-07T18:09:57.3590839Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:09:57.3591312Z kill "$MONITOR_SCRIPT_PID" 2024-08-07T18:09:57.3599780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.3600271Z env: 2024-08-07T18:09:57.3600550Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.3600990Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.3601673Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.3602288Z MONITOR_SCRIPT_PID: 75881 2024-08-07T18:09:57.3602649Z ##[endgroup] 2024-08-07T18:09:57.3774722Z Prepare all required actions 2024-08-07T18:09:57.3775284Z Getting action download info 2024-08-07T18:09:57.5492262Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-07T18:09:57.7109850Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-07T18:09:57.7110277Z with: 2024-08-07T18:09:57.7110718Z file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889 2024-08-07T18:09:57.7111274Z s3-bucket: gha-artifacts 2024-08-07T18:09:57.7111589Z env: 2024-08-07T18:09:57.7111847Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.7112254Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.7112910Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.7113940Z ##[endgroup] 2024-08-07T18:09:57.7149904Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-07T18:09:57.7150493Z # Remove any previous test jsons if they exist 2024-08-07T18:09:57.7150950Z rm -f test-jsons-*.zip 2024-08-07T18:09:57.7151437Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-07T18:09:57.7159888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.7160345Z env: 2024-08-07T18:09:57.7160605Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.7161005Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.7161659Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.7162419Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889 2024-08-07T18:09:57.7162946Z ##[endgroup] 2024-08-07T18:09:57.7409406Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-07T18:09:57.7445645Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-07T18:09:57.7446333Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-07T18:09:57.7453774Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-07T18:09:57.7456044Z adding: test/test-reports/td_exclusions-19d3c0884e02dfb9118c.json (deflated 86%) 2024-08-07T18:09:57.7457573Z adding: test/test-reports/td_exclusions-535a47e1970d74b1d372.json (deflated 15%) 2024-08-07T18:09:57.7458516Z adding: test/test-reports/td_exclusions-d1bec898f33826a62ec1.json (deflated 14%) 2024-08-07T18:09:57.7459314Z adding: test/test-reports/td_exclusions-ebf439315b548825ca01.json (deflated 14%) 2024-08-07T18:09:57.7460116Z adding: test/test-reports/td_exclusions-5579a7a66cce93639e32.json (deflated 12%) 2024-08-07T18:09:57.7460919Z adding: test/test-reports/td_exclusions-4ccb9e8ac9ba0d1c8afe.json (deflated 12%) 2024-08-07T18:09:57.7461705Z adding: test/test-reports/td_exclusions-4ec5cda571188d841510.json (deflated 11%) 2024-08-07T18:09:57.7462497Z adding: test/test-reports/td_exclusions-e1b40fc4c4d45e7137c6.json (deflated 18%) 2024-08-07T18:09:57.7465299Z adding: test/.pytorch-slow-tests.json (deflated 81%) 2024-08-07T18:09:57.7481559Z adding: test/.pytorch-disabled-tests.json (deflated 89%) 2024-08-07T18:09:57.7519708Z ##[group]Run # Remove any previous test reports if they exist 2024-08-07T18:09:57.7520323Z # Remove any previous test reports if they exist 2024-08-07T18:09:57.7520805Z rm -f test-reports-*.zip 2024-08-07T18:09:57.7521360Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-07T18:09:57.7529132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.7529627Z env: 2024-08-07T18:09:57.7529925Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.7530348Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.7531042Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.7531842Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889 2024-08-07T18:09:57.7532394Z ##[endgroup] 2024-08-07T18:09:57.7756006Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-3cd38fbd15af33a1.xml (deflated 84%) 2024-08-07T18:09:57.7757636Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-82521779192d57ee.xml (deflated 28%) 2024-08-07T18:09:57.7759085Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_core/distributed.fsdp.test_fsdp_core-26699f7c0df91757.xml (deflated 88%) 2024-08-07T18:09:57.7760494Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-43eaaf7c102560d4.xml (deflated 37%) 2024-08-07T18:09:57.7762007Z adding: test/test-reports/python-pytest/distributed._tensor.test_experimental_ops/distributed._tensor.test_experimental_ops-e8fbce659286d4fa.xml (deflated 56%) 2024-08-07T18:09:57.7763479Z adding: test/test-reports/python-pytest/distributed.test_c10d_pypg/distributed.test_c10d_pypg-357aa9dda99d2f42.xml (deflated 90%) 2024-08-07T18:09:57.7764900Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-3c651f87ee57ab65.xml (deflated 60%) 2024-08-07T18:09:57.7766428Z adding: test/test-reports/python-pytest/distributed.pipelining.test_stage/distributed.pipelining.test_stage-02b3e947c582f87a.xml (deflated 69%) 2024-08-07T18:09:57.7767980Z adding: test/test-reports/python-pytest/distributed._tensor.test_xla_integration/distributed._tensor.test_xla_integration-eb2416b31ead3137.xml (deflated 74%) 2024-08-07T18:09:57.7769585Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_save_load_api/distributed.checkpoint.test_save_load_api-728c9014521ebf9c.xml (deflated 49%) 2024-08-07T18:09:57.7771302Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-69bb9ec8f8fafc70.xml (deflated 73%) 2024-08-07T18:09:57.7773326Z adding: test/test-reports/python-pytest/distributed._composable.test_checkpoint/distributed._composable.test_checkpoint-5672b6a068dc8645.xml (deflated 72%) 2024-08-07T18:09:57.7775230Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_tp_style/distributed.tensor.parallel.test_tp_style-2be86b37cccf51ac.xml (deflated 77%) 2024-08-07T18:09:57.7776983Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_logging/distributed._composable.fsdp.test_fully_shard_logging-30a575aa0cabe236.xml (deflated 28%) 2024-08-07T18:09:57.7778435Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-8c53b69cacee2137.xml (deflated 60%) 2024-08-07T18:09:57.7779552Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-27b449e7aa205365.xml (deflated 60%) 2024-08-07T18:09:57.7780579Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-6566576ce797df48.xml (deflated 69%) 2024-08-07T18:09:57.7781603Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-b41769771e8f0e9f.xml (deflated 78%) 2024-08-07T18:09:57.7782631Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-eb7317281ff6f326.xml (deflated 74%) 2024-08-07T18:09:57.7783661Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-f6e6ba7ce346dd12.xml (deflated 75%) 2024-08-07T18:09:57.7784676Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-74d72ab201c085c5.xml (deflated 64%) 2024-08-07T18:09:57.7818606Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-07T18:09:57.7819204Z # Remove any previous usage logs if they exist 2024-08-07T18:09:57.7819684Z rm -f logs-*.zip 2024-08-07T18:09:57.7820261Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-07T18:09:57.7820948Z # so check to see if the file exists first 2024-08-07T18:09:57.7821417Z if [ -f 'usage_log.txt' ]; then 2024-08-07T18:09:57.7821881Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-07T18:09:57.7822508Z fi 2024-08-07T18:09:57.7822902Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-07T18:09:57.7823440Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-07T18:09:57.7823906Z fi 2024-08-07T18:09:57.7831094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.7831591Z env: 2024-08-07T18:09:57.7831889Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.7832315Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.7833003Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.7834211Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889 2024-08-07T18:09:57.7834773Z ##[endgroup] 2024-08-07T18:09:57.7884433Z adding: usage_log.txt (deflated 92%) 2024-08-07T18:09:57.8115510Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_e972b11e702c777a_.log (deflated 85%) 2024-08-07T18:09:57.8116597Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_84600b34e8be0a46_.log (deflated 52%) 2024-08-07T18:09:57.8191167Z adding: test/test-reports/distributed.fsdp.test_fsdp_core_2.2_776d559e857a2cee_.log (deflated 96%) 2024-08-07T18:09:57.8192132Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_25c55fd709ede464_.log (deflated 53%) 2024-08-07T18:09:57.8193465Z adding: test/test-reports/distributed._tensor.test_experimental_ops_1.1_d42eb07d3e9a5ddb_.log (deflated 62%) 2024-08-07T18:09:57.8195183Z adding: test/test-reports/distributed.test_c10d_pypg_1.1_398955911b234d20_.log (deflated 90%) 2024-08-07T18:09:57.8196333Z adding: test/test-reports/distributed.pipelining.test_stage_1.1_5bb76dcae5892632_.log (deflated 82%) 2024-08-07T18:09:57.8197728Z adding: test/test-reports/distributed._tensor.test_xla_integration_1.1_89e9116023171a9b_.log (deflated 66%) 2024-08-07T18:09:57.8198849Z adding: test/test-reports/distributed.checkpoint.test_save_load_api_1.1_d8903ddfc4ed3cba_.log (deflated 51%) 2024-08-07T18:09:57.8199991Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_ed12b2489c85b959_.log (deflated 71%) 2024-08-07T18:09:57.8201395Z adding: test/test-reports/distributed._composable.test_checkpoint_1.1_50b60c47f640c01f_.log (deflated 70%) 2024-08-07T18:09:57.8202504Z adding: test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_34d7503e3952bc28_.log (deflated 76%) 2024-08-07T18:09:57.8203640Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_556ae0b1b69d26fb_.log (deflated 53%) 2024-08-07T18:09:57.8204698Z adding: test/test-reports/cpp.FileStoreTest_1.1_93f074ed2367ac7c_.log (deflated 71%) 2024-08-07T18:09:57.8205540Z adding: test/test-reports/cpp.HashStoreTest_1.1_fc5c5ea7539d30b1_.log (deflated 71%) 2024-08-07T18:09:57.8206379Z adding: test/test-reports/cpp.TCPStoreTest_1.1_4c77fc0945f2a509_.log (deflated 79%) 2024-08-07T18:09:57.8207238Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_a40ff419bac27115_.log (deflated 85%) 2024-08-07T18:09:57.8208167Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_c1b066b14b96c72b_.log (deflated 82%) 2024-08-07T18:09:57.8209186Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_9b369eca72f27c88_.log (deflated 78%) 2024-08-07T18:09:57.8210344Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_cdeb357e55ec592b_.log (deflated 79%) 2024-08-07T18:09:57.8246917Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-07T18:09:57.8247595Z # Remove any previous debugging artifacts if they exist 2024-08-07T18:09:57.8248097Z rm -f debug-*.zip 2024-08-07T18:09:57.8248481Z if [ -d 'test/debug' ]; then 2024-08-07T18:09:57.8248955Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-07T18:09:57.8249387Z fi 2024-08-07T18:09:57.8256627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:09:57.8257123Z env: 2024-08-07T18:09:57.8257596Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.8258042Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.8258744Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.8259542Z FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889 2024-08-07T18:09:57.8260089Z ##[endgroup] 2024-08-07T18:09:57.8352040Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:09:57.8352494Z with: 2024-08-07T18:09:57.8352785Z s3-bucket: gha-artifacts 2024-08-07T18:09:57.8353686Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:57.8354261Z retention-days: 14 2024-08-07T18:09:57.8354668Z if-no-files-found: warn 2024-08-07T18:09:57.8355044Z path: test-jsons-*.zip 2024-08-07T18:09:57.8355397Z name: artifact 2024-08-07T18:09:57.8355699Z region: us-east-1 2024-08-07T18:09:57.8356016Z env: 2024-08-07T18:09:57.8356311Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:57.8356751Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:57.8357479Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:57.8358150Z ##[endgroup] 2024-08-07T18:09:58.2839333Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:09:58.2840295Z With the provided path, there will be 1 file uploaded 2024-08-07T18:09:58.2840885Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:58.2888713Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:58.5215763Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:58.5377956Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:09:58.5378472Z with: 2024-08-07T18:09:58.5378762Z s3-bucket: gha-artifacts 2024-08-07T18:09:58.5379187Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:58.5379649Z retention-days: 14 2024-08-07T18:09:58.5379981Z if-no-files-found: error 2024-08-07T18:09:58.5380378Z path: test-reports-*.zip 2024-08-07T18:09:58.5380741Z name: artifact 2024-08-07T18:09:58.5381259Z region: us-east-1 2024-08-07T18:09:58.5381575Z env: 2024-08-07T18:09:58.5381866Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:58.5382287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:58.5382974Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:58.5383589Z ##[endgroup] 2024-08-07T18:09:58.9890554Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:09:58.9891694Z With the provided path, there will be 1 file uploaded 2024-08-07T18:09:58.9892282Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:58.9942528Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:59.2525145Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:59.2712034Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:09:59.2712509Z with: 2024-08-07T18:09:59.2712838Z s3-bucket: gha-artifacts 2024-08-07T18:09:59.2713811Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:59.2714330Z retention-days: 14 2024-08-07T18:09:59.2714670Z if-no-files-found: ignore 2024-08-07T18:09:59.2715055Z path: logs-*.zip 2024-08-07T18:09:59.2715397Z name: artifact 2024-08-07T18:09:59.2715752Z region: us-east-1 2024-08-07T18:09:59.2716085Z env: 2024-08-07T18:09:59.2716397Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:59.2716839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:59.2717550Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:59.2718189Z ##[endgroup] 2024-08-07T18:09:59.7166634Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-07T18:09:59.7167501Z With the provided path, there will be 1 file uploaded 2024-08-07T18:09:59.7168466Z Uploading to s3 prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:59.7215202Z Starting upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:59.8400054Z Finished upload of logs-test-distributed-1-3-linux.8xlarge.nvidia.gpu_28476144889.zip 2024-08-07T18:09:59.8565662Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-07T18:09:59.8566105Z with: 2024-08-07T18:09:59.8566396Z s3-bucket: gha-artifacts 2024-08-07T18:09:59.8566824Z s3-prefix: pytorch/pytorch/10288745067/1/artifact 2024-08-07T18:09:59.8567286Z retention-days: 14 2024-08-07T18:09:59.8567616Z if-no-files-found: ignore 2024-08-07T18:09:59.8567985Z path: debug-*.zip 2024-08-07T18:09:59.8568305Z name: artifact 2024-08-07T18:09:59.8568602Z region: us-east-1 2024-08-07T18:09:59.8568915Z env: 2024-08-07T18:09:59.8569206Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:09:59.8569623Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:09:59.8570325Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:09:59.8570940Z ##[endgroup] 2024-08-07T18:10:00.2940416Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-07T18:10:00.3147719Z ##[group]Run # shellcheck disable=SC2156 2024-08-07T18:10:00.3148317Z # shellcheck disable=SC2156 2024-08-07T18:10:00.3149052Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-07T18:10:00.3157788Z shell: /usr/bin/bash -e {0} 2024-08-07T18:10:00.3158143Z env: 2024-08-07T18:10:00.3158437Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:10:00.3158857Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:10:00.3159561Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:10:00.3160182Z ##[endgroup] 2024-08-07T18:10:00.6164552Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-07T18:10:00.6165138Z with: 2024-08-07T18:10:00.6165406Z env: 2024-08-07T18:10:00.6165703Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:10:00.6166383Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:10:00.6167067Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:10:00.6167689Z ##[endgroup] 2024-08-07T18:10:00.6195588Z ##[group]Run set -eou pipefail 2024-08-07T18:10:00.6196016Z set -eou pipefail 2024-08-07T18:10:00.6196357Z  2024-08-07T18:10:00.6196839Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-07T18:10:00.6197514Z for _ in $(seq 1440); do 2024-08-07T18:10:00.6197936Z  # Break if no ssh session exists anymore 2024-08-07T18:10:00.6198394Z  if [ "$(who)" = "" ]; then 2024-08-07T18:10:00.6198785Z  break 2024-08-07T18:10:00.6199160Z  fi 2024-08-07T18:10:00.6199461Z  echo "." 2024-08-07T18:10:00.6199787Z  sleep 5 2024-08-07T18:10:00.6200112Z done 2024-08-07T18:10:00.6207701Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:10:00.6208212Z env: 2024-08-07T18:10:00.6208515Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:10:00.6208944Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:10:00.6209636Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:10:00.6210251Z ##[endgroup] 2024-08-07T18:10:00.6234295Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-07T18:10:00.6292187Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:10:00.6292874Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-07T18:10:00.6293416Z # shellcheck disable=SC2046 2024-08-07T18:10:00.6293875Z docker stop $(docker ps -q) || true 2024-08-07T18:10:00.6294319Z # Prune all of the docker images 2024-08-07T18:10:00.6294743Z docker system prune -af 2024-08-07T18:10:00.6302054Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:10:00.6302559Z env: 2024-08-07T18:10:00.6302852Z GIT_DEFAULT_BRANCH: main 2024-08-07T18:10:00.6303270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-07T18:10:00.6303952Z DOCKER_CONTAINER_ID: 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:10:00.6304567Z ##[endgroup] 2024-08-07T18:10:00.9957881Z 27d39248f8e7 2024-08-07T18:10:01.5541128Z Deleted Containers: 2024-08-07T18:10:01.5541657Z 27d39248f8e76219a2b9836ed1018236e16ed28a8436a0b58d0e3a059de106ac 2024-08-07T18:10:01.5542079Z 2024-08-07T18:10:08.1155991Z Deleted Images: 2024-08-07T18:10:08.1157078Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-07T18:10:08.1158755Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:66181a134b3ebc35388dabcc46dc11e767152fd72514dad76711590f50b650f7 2024-08-07T18:10:08.1160086Z deleted: sha256:8e4e19ae0377c273bc603b24bc49d406d6f755072d4534d040c86afbe761a129 2024-08-07T18:10:08.1160934Z deleted: sha256:bee5df81b6fe1ee0aae7e536a9bda16351d0515be5b789ad50bf0ac60fd41e33 2024-08-07T18:10:08.1161747Z deleted: sha256:8ee1ee099b65b2d73d66dc3330f82f8be2dd6cbdccf3d718b448a5e3419c214e 2024-08-07T18:10:08.1162552Z deleted: sha256:aed7424cdc4428e32e398a33e98e6e252075e6280e7068e20df7626451fa312a 2024-08-07T18:10:08.1163344Z deleted: sha256:61b7d2994ed16481c3de07839b5a3bb779dc2f3f716ad49acb91537042c3b7d1 2024-08-07T18:10:08.1164149Z deleted: sha256:79999dfec9b5df258486b014aa99c30979900c4131effd2a7bc18251cd5fc372 2024-08-07T18:10:08.1164932Z deleted: sha256:9020e3bab07b431af26c788ac2d12cbca9a76ffb1c923ea45a38e872942e1f34 2024-08-07T18:10:08.1165997Z deleted: sha256:387e2f2f359df5f69c500ae9589c224d8283ae5418520509b3c9ef0a45d116fa 2024-08-07T18:10:08.1166830Z deleted: sha256:5ddfc73c14e2e0a762c51ddecfb2710258366e1d048a8bf56d70268486907b99 2024-08-07T18:10:08.1167614Z deleted: sha256:4dffb251634d1c8105e68310ef1b2ffb923dac4c4139f494be0938ad032a1bca 2024-08-07T18:10:08.1168608Z deleted: sha256:e0df7617281f0e4396253d2f0c40d1418ad604dc0912cae13f153fe22e316fa0 2024-08-07T18:10:08.1169413Z deleted: sha256:d239b3ccf05c2850364c3f73acbc6e0017a25f8faa4c3abaa8524e63dc9f9c29 2024-08-07T18:10:08.1170205Z deleted: sha256:918d0912faf4251f3be58f9a5aaa80f59a180ace4771b9ce903cedf94a57723a 2024-08-07T18:10:08.1170996Z deleted: sha256:09d5696392736fa88c52a0656d1d831d2bcb4015630810412565a5a086eb98ff 2024-08-07T18:10:08.1171776Z deleted: sha256:67542af70388f69b345fc7f0965e1fce3c552ee987689f804c8406536f16534f 2024-08-07T18:10:08.1172553Z deleted: sha256:350337bafbf2976e8d4e0b7ce3cc66b2cb43ebb8622fa6e538b02f561f929393 2024-08-07T18:10:08.1173388Z deleted: sha256:00fe9fe12fefe10d5dbfb2f855aa385e840f360deeccbc9ee92cae51c6eea4ba 2024-08-07T18:10:08.1174217Z deleted: sha256:9895af0586ff7d94936462a92d7cec0d8ec3b01df27b664f7e4ac277ce7bbe9e 2024-08-07T18:10:08.1175020Z deleted: sha256:7dd7dbc3ec792cf9d043ff867afe169d15fe414138097ad9247a5f388d3eeccf 2024-08-07T18:10:08.1175813Z deleted: sha256:e0a3ace1b87c67d71f5db76d211740431660a7fbcd80415333c9e7b7b9a2e89b 2024-08-07T18:10:08.1177202Z deleted: sha256:b9c56b5040d5d5998961b788563b85e5be12f678bc6b3976d089a81f1c93b7e8 2024-08-07T18:10:08.1178006Z deleted: sha256:9232fe8591973938b7bdb0a7a31dcdbc79f9667d6e011a2facf4e17fb77d4922 2024-08-07T18:10:08.1178779Z deleted: sha256:52f5fbe778d735123327825c6a16dcec9bc84c2623087bb4c7c094ac99764ae7 2024-08-07T18:10:08.1179581Z deleted: sha256:4b7c4121b76b04d5c7e4a0b77add435c0caa99cccff506ea704eba27fd2b2166 2024-08-07T18:10:08.1180377Z deleted: sha256:795dd765ace5b617808fc40cd576b449ae01b8671290e6df6b49d86ea79f4145 2024-08-07T18:10:08.1181164Z deleted: sha256:51cc651f1ffbeef49ad010ab2d4cf2d5ce5f8f5767c2133346a959c773d837bb 2024-08-07T18:10:08.1181978Z deleted: sha256:4c3019549b40f7a268d344eae162d7a7cc78855880db865944da136e01c190e9 2024-08-07T18:10:08.1182783Z deleted: sha256:855b789ec056edaab8cc8dad50a82e9dbfb9b20bcf39c70d9c3bd40080b2b65f 2024-08-07T18:10:08.1183586Z deleted: sha256:cef853cb7e0f5d6351692a4b9a0413a2496ea3c7bfe5d8063528ac2d6c17696f 2024-08-07T18:10:08.1184377Z deleted: sha256:4f7b163ef5ff693ce9f7380bdf7b526131f2567489342f0630949c9341e1c63f 2024-08-07T18:10:08.1185167Z deleted: sha256:09f12e5e102232b4ec9fd5168f4aa4b20ee0dfc755b053ac57690657e72e3fe3 2024-08-07T18:10:08.1185942Z deleted: sha256:c9962be393e03353af0d6a26f5478a94cacb5e528ee3f97609fac594c9a5d387 2024-08-07T18:10:08.1186744Z deleted: sha256:ac7aa4ec25b0d9a035c438effaa309071729de3f3a0f96e8e481b9839e7460b8 2024-08-07T18:10:08.1187538Z deleted: sha256:63ff2106018988888d022ee38d6ce1805aa29d8368a18dec1ab8eb587799ab44 2024-08-07T18:10:08.1188919Z deleted: sha256:3298c1acdeb701f11351366cfe680a84b8290666fdcd76af5ac79f226422fc0d 2024-08-07T18:10:08.1190620Z deleted: sha256:f076cc29eff5dae969bc7c05003d680b6e683eaef971c0a3f67a8a4d12693ee4 2024-08-07T18:10:08.1191441Z deleted: sha256:2604655e98c557e6de2e650d21cd0306c01f12204f3538386a9f374d003a4d70 2024-08-07T18:10:08.1192230Z deleted: sha256:fb942194d6d4fc6f9c8a4c8cba17330563877c92369892edc9b005b8b491f204 2024-08-07T18:10:08.1193016Z deleted: sha256:d349cc387708a540eee839ccd78494458dc5f1ccf989db608d25c5f798ec8c76 2024-08-07T18:10:08.1194308Z deleted: sha256:269864865f9b12ee399ca9ee3e15a16cedb86fd4889e01a83567afb357a774e6 2024-08-07T18:10:08.1195109Z deleted: sha256:236a91ce743cd500427338a78915ff834ae80331fcc06afccfca82a35bffb63c 2024-08-07T18:10:08.1195900Z deleted: sha256:de968bbc1d8bc2ad420d6a1c78a9a6cf8a763bb855ca3aefe7de6c5c52201f7a 2024-08-07T18:10:08.1196723Z deleted: sha256:0f15ecc513ffdac89965bd76cd9f8c7f4e37bbe7be0bfa086353133e2dd4f086 2024-08-07T18:10:08.1197556Z deleted: sha256:754bb9b5a5af40b6ad88f0ab14be37b9e512e25edccbfc9fd5e0c5a88ca09c0f 2024-08-07T18:10:08.1198530Z deleted: sha256:427104437dfa1883feec2c441eb4e4f825481883c6d3d31759361cbc924a27fe 2024-08-07T18:10:08.1199357Z deleted: sha256:0b404434cf2c3087ef28ea3cdc5813cc555f6e261c2514a7ab5bf5d484b5b575 2024-08-07T18:10:08.1200170Z deleted: sha256:bfb91e062bab9dafb63d0d4ab366026692fbd63e642da9ab28f04c8bba9b56a3 2024-08-07T18:10:08.1201125Z deleted: sha256:54617b36a396e9e82f2092257d2f621c90de460e0e54b1cf8df065573fbaa0ba 2024-08-07T18:10:08.1201904Z deleted: sha256:d8bdc337db1750bb3ebc7c8e4e1bd48e722218f261cb35ec2056ceaa6d3c4e9c 2024-08-07T18:10:08.1202733Z deleted: sha256:25cbacf4be7b2bbcbeaf7375d0eff0eaa33e94c5bef20ff0993bf074fdaa54eb 2024-08-07T18:10:08.1203555Z deleted: sha256:43744f05dcf6dc44c2f5e06533b13e085319a78b6b5e17eba5e7bd4419f5c905 2024-08-07T18:10:08.1204333Z deleted: sha256:2ea7d74f63d16a8d6e14a87e0e916f67f99536b21ac53e80b27786dcd1ce36fd 2024-08-07T18:10:08.1205135Z deleted: sha256:dbfb6031f18e50c89ff1a268c1e80e2fa88e727c48656718b326b60e8ef0d383 2024-08-07T18:10:08.1205958Z deleted: sha256:3eb3bb1755fcc3d4aba9abe2d2270ed2a55ec6254b3e87ca7688370e67e707a3 2024-08-07T18:10:08.1206746Z deleted: sha256:04605aba2d51924a85499584edda24c28390eb777c1c88eff2f4191b48e3da8c 2024-08-07T18:10:08.1207546Z deleted: sha256:117ff9e5eddba4582a39d416380eadf1c245237612c5ae0d6781d36c36bcb87f 2024-08-07T18:10:08.1208361Z deleted: sha256:ebf86c84eafa12a81b8eb66541a56c8056f4f9bc30e4be761771e53c6e1342aa 2024-08-07T18:10:08.1209155Z deleted: sha256:23d7f7e0c9d7e60acb59e60da9b4ee4e1840c1c406bf2082822c49430975529d 2024-08-07T18:10:08.1209965Z deleted: sha256:3e7e5583645195af5eeb60fe49024a4aa2f0eb3be28ff95c9859bb6233c29c4b 2024-08-07T18:10:08.1210759Z deleted: sha256:45b8f0dd78e72f0156b1fbc4dcb14200c61954f37a75129c753275e5a45ccd9b 2024-08-07T18:10:08.1211540Z deleted: sha256:f6b7a042384e2f2bb3401da2b9f4e9b9f942adb70be7e7f89c46fa81a42774d8 2024-08-07T18:10:08.1212355Z deleted: sha256:eb547a7b8fe732bffe8cc9a66aa19f2b41f6e0616a8a27040307c022d00fbc0d 2024-08-07T18:10:08.1213162Z deleted: sha256:20f729283bb4f2063f3a8baa0b2afc17911da345bf367e1918d8cc09ddd6d964 2024-08-07T18:10:08.1213968Z deleted: sha256:8ce8da9ab0319625a66c35d24ddadf8d2688d16c5024ba2345099431e0aa7ff8 2024-08-07T18:10:08.1214750Z deleted: sha256:e7a91181c74f879360530f0bbe957b8ff96a1ec19adde8a41d9e3fa2038d0d16 2024-08-07T18:10:08.1215543Z deleted: sha256:212b7a0fad7208624a2bc56768d8c489d2690142d0db0fa20f69eb648234e8d1 2024-08-07T18:10:08.1216343Z deleted: sha256:72315e37f51ba8bcc88fc5b56f92e5bb6264456d9a755afa01fd19f9128c1df7 2024-08-07T18:10:08.1217178Z deleted: sha256:bfc5ab07cfab1698147152988eabdc7ebbdeb35b6d9aefd9360dd64104cd3888 2024-08-07T18:10:08.1217982Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-08-07T18:10:08.1218787Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-08-07T18:10:08.1219573Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-08-07T18:10:08.1220359Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-08-07T18:10:08.1221165Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-08-07T18:10:08.1221961Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-08-07T18:10:08.1222761Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-08-07T18:10:08.1223557Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-08-07T18:10:08.1224364Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-08-07T18:10:08.1225143Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-08-07T18:10:08.1225927Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-07T18:10:08.1226397Z 2024-08-07T18:10:08.1226586Z Total reclaimed space: 26.03GB 2024-08-07T18:10:08.1341617Z Post job cleanup. 2024-08-07T18:10:08.1401829Z Post job cleanup. 2024-08-07T18:10:08.2574263Z [command]/usr/bin/git version 2024-08-07T18:10:08.2628159Z git version 2.40.1 2024-08-07T18:10:08.2691913Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/eefaa625-861f-4055-827e-16cd4b7d9f0d' before making global git config changes 2024-08-07T18:10:08.2693138Z Adding repository directory to the temporary git global config as a safe directory 2024-08-07T18:10:08.2699898Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-07T18:10:08.2742632Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-07T18:10:08.2779023Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-07T18:10:08.3093412Z Entering 'android/libs/fbjni' 2024-08-07T18:10:08.3144958Z Entering 'third_party/FP16' 2024-08-07T18:10:08.3197431Z Entering 'third_party/FXdiv' 2024-08-07T18:10:08.3250475Z Entering 'third_party/NNPACK' 2024-08-07T18:10:08.3301899Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:10:08.3353102Z Entering 'third_party/XNNPACK' 2024-08-07T18:10:08.3424028Z Entering 'third_party/benchmark' 2024-08-07T18:10:08.3476768Z Entering 'third_party/cpp-httplib' 2024-08-07T18:10:08.3527180Z Entering 'third_party/cpuinfo' 2024-08-07T18:10:08.3579115Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:10:08.3629792Z Entering 'third_party/cutlass' 2024-08-07T18:10:08.3688732Z Entering 'third_party/eigen' 2024-08-07T18:10:08.3742861Z Entering 'third_party/fbgemm' 2024-08-07T18:10:08.3793777Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:10:08.3845490Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:10:08.3896943Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:10:08.3955503Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:10:08.4005446Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:10:08.4057703Z Entering 'third_party/flatbuffers' 2024-08-07T18:10:08.4112999Z Entering 'third_party/fmt' 2024-08-07T18:10:08.4164143Z Entering 'third_party/foxi' 2024-08-07T18:10:08.4215232Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:10:08.4267145Z Entering 'third_party/gloo' 2024-08-07T18:10:08.4319124Z Entering 'third_party/googletest' 2024-08-07T18:10:08.4370370Z Entering 'third_party/ideep' 2024-08-07T18:10:08.4420987Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:10:08.4480875Z Entering 'third_party/ittapi' 2024-08-07T18:10:08.4532617Z Entering 'third_party/kineto' 2024-08-07T18:10:08.4584642Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:10:08.4634790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:10:08.4688395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:10:08.4739945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:10:08.4791565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:10:08.4842717Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:10:08.4894751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:10:08.4946100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:10:08.4997577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:10:08.5049274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:10:08.5101156Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:10:08.5151381Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:10:08.5203423Z Entering 'third_party/mimalloc' 2024-08-07T18:10:08.5256199Z Entering 'third_party/nccl/nccl' 2024-08-07T18:10:08.5308625Z Entering 'third_party/nlohmann' 2024-08-07T18:10:08.5361489Z Entering 'third_party/onnx' 2024-08-07T18:10:08.5431571Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:10:08.5482811Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:10:08.5536374Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:10:08.5588784Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:10:08.5640554Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:10:08.5690808Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:10:08.5740427Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:10:08.5791579Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:10:08.5843971Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:10:08.5893817Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:10:08.5943801Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:10:08.5995942Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:10:08.6047979Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:10:08.6121439Z Entering 'third_party/pocketfft' 2024-08-07T18:10:08.6172278Z Entering 'third_party/protobuf' 2024-08-07T18:10:08.6226971Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:10:08.6278268Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:10:08.6331296Z Entering 'third_party/psimd' 2024-08-07T18:10:08.6383407Z Entering 'third_party/pthreadpool' 2024-08-07T18:10:08.6435599Z Entering 'third_party/pybind11' 2024-08-07T18:10:08.6486669Z Entering 'third_party/python-peachpy' 2024-08-07T18:10:08.6538168Z Entering 'third_party/sleef' 2024-08-07T18:10:08.6589199Z Entering 'third_party/tensorpipe' 2024-08-07T18:10:08.6642424Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:10:08.6691907Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:10:08.6741886Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:10:08.6792794Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:10:08.6841652Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:10:08.6912552Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-07T18:10:08.6938590Z http.https://github.com/.extraheader 2024-08-07T18:10:08.6950231Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-07T18:10:08.6987524Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-07T18:10:08.7295506Z Entering 'android/libs/fbjni' 2024-08-07T18:10:08.7329076Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7363786Z Entering 'third_party/FP16' 2024-08-07T18:10:08.7396997Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7430173Z Entering 'third_party/FXdiv' 2024-08-07T18:10:08.7463963Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7498159Z Entering 'third_party/NNPACK' 2024-08-07T18:10:08.7531812Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7566349Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-07T18:10:08.7600159Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7634817Z Entering 'third_party/XNNPACK' 2024-08-07T18:10:08.7666813Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7719947Z Entering 'third_party/benchmark' 2024-08-07T18:10:08.7752378Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7786197Z Entering 'third_party/cpp-httplib' 2024-08-07T18:10:08.7819536Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7853439Z Entering 'third_party/cpuinfo' 2024-08-07T18:10:08.7886645Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7921060Z Entering 'third_party/cudnn_frontend' 2024-08-07T18:10:08.7952901Z http.https://github.com/.extraheader 2024-08-07T18:10:08.7987218Z Entering 'third_party/cutlass' 2024-08-07T18:10:08.8020945Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8063820Z Entering 'third_party/eigen' 2024-08-07T18:10:08.8096964Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8133902Z Entering 'third_party/fbgemm' 2024-08-07T18:10:08.8166883Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8202049Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-07T18:10:08.8234885Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8268543Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-07T18:10:08.8301040Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8335111Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-07T18:10:08.8367185Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8408185Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-07T18:10:08.8441168Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8474037Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-07T18:10:08.8506195Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8540668Z Entering 'third_party/flatbuffers' 2024-08-07T18:10:08.8575134Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8612055Z Entering 'third_party/fmt' 2024-08-07T18:10:08.8645296Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8678876Z Entering 'third_party/foxi' 2024-08-07T18:10:08.8710978Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8744616Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-07T18:10:08.8777931Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8811470Z Entering 'third_party/gloo' 2024-08-07T18:10:08.8844083Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8878499Z Entering 'third_party/googletest' 2024-08-07T18:10:08.8910330Z http.https://github.com/.extraheader 2024-08-07T18:10:08.8944561Z Entering 'third_party/ideep' 2024-08-07T18:10:08.8977474Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9009586Z Entering 'third_party/ideep/mkl-dnn' 2024-08-07T18:10:08.9041784Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9083524Z Entering 'third_party/ittapi' 2024-08-07T18:10:08.9116216Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9149522Z Entering 'third_party/kineto' 2024-08-07T18:10:08.9182425Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9216588Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-07T18:10:08.9249005Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9282786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-07T18:10:08.9316259Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9351955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-07T18:10:08.9384153Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9418613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-07T18:10:08.9451805Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9486226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-07T18:10:08.9519548Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9553181Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-07T18:10:08.9586162Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9622320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-07T18:10:08.9654751Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9690017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-07T18:10:08.9723042Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9757231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-07T18:10:08.9790023Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9826323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-07T18:10:08.9858402Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9893703Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-07T18:10:08.9925867Z http.https://github.com/.extraheader 2024-08-07T18:10:08.9959192Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-07T18:10:08.9990698Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0026357Z Entering 'third_party/mimalloc' 2024-08-07T18:10:09.0059349Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0093178Z Entering 'third_party/nccl/nccl' 2024-08-07T18:10:09.0125792Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0159721Z Entering 'third_party/nlohmann' 2024-08-07T18:10:09.0191864Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0227261Z Entering 'third_party/onnx' 2024-08-07T18:10:09.0260499Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0313640Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-07T18:10:09.0346297Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0379985Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-07T18:10:09.0413306Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0449851Z Entering 'third_party/opentelemetry-cpp' 2024-08-07T18:10:09.0483093Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0519454Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-07T18:10:09.0550674Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0583831Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-07T18:10:09.0615752Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0649575Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-07T18:10:09.0681647Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0714841Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-07T18:10:09.0747492Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0782811Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-07T18:10:09.0814346Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0847530Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-07T18:10:09.0880526Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0913520Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-07T18:10:09.0945716Z http.https://github.com/.extraheader 2024-08-07T18:10:09.0978653Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-07T18:10:09.1011033Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1048052Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-07T18:10:09.1079567Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1114361Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-07T18:10:09.1148413Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1206114Z Entering 'third_party/pocketfft' 2024-08-07T18:10:09.1239184Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1272873Z Entering 'third_party/protobuf' 2024-08-07T18:10:09.1305878Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1343005Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-07T18:10:09.1375253Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1408402Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-07T18:10:09.1440711Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1475895Z Entering 'third_party/psimd' 2024-08-07T18:10:09.1508481Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1542780Z Entering 'third_party/pthreadpool' 2024-08-07T18:10:09.1575971Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1609413Z Entering 'third_party/pybind11' 2024-08-07T18:10:09.1642951Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1677178Z Entering 'third_party/python-peachpy' 2024-08-07T18:10:09.1710300Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1744813Z Entering 'third_party/sleef' 2024-08-07T18:10:09.1777528Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1811408Z Entering 'third_party/tensorpipe' 2024-08-07T18:10:09.1844419Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1878957Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-07T18:10:09.1910173Z http.https://github.com/.extraheader 2024-08-07T18:10:09.1943723Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-07T18:10:09.1977675Z http.https://github.com/.extraheader 2024-08-07T18:10:09.2011189Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-07T18:10:09.2044128Z http.https://github.com/.extraheader 2024-08-07T18:10:09.2078251Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-07T18:10:09.2110850Z http.https://github.com/.extraheader 2024-08-07T18:10:09.2143691Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-07T18:10:09.2175985Z http.https://github.com/.extraheader 2024-08-07T18:10:09.2310639Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-07T18:10:09.2340684Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-07T18:10:09.2347666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-07T18:10:09.2348169Z ##[endgroup] 2024-08-07T18:10:15.0710935Z Cleaning up orphan processes