2024-11-01T16:42:17.2554209Z Current runner version: '2.320.0' 2024-11-01T16:42:17.2561760Z Runner name: 'i-0a5b4d9b9fef8d246' 2024-11-01T16:42:17.2562786Z Runner group name: 'Default' 2024-11-01T16:42:17.2563653Z Machine name: 'ip-10-0-58-136' 2024-11-01T16:42:17.2580524Z Testing runner upgrade compatibility 2024-11-01T16:42:17.3710432Z ##[group]GITHUB_TOKEN Permissions 2024-11-01T16:42:17.3712548Z Actions: read 2024-11-01T16:42:17.3713108Z Attestations: read 2024-11-01T16:42:17.3713627Z Checks: read 2024-11-01T16:42:17.3714091Z Contents: read 2024-11-01T16:42:17.3714614Z Deployments: read 2024-11-01T16:42:17.3715113Z Discussions: read 2024-11-01T16:42:17.3715651Z Issues: read 2024-11-01T16:42:17.3716369Z Metadata: read 2024-11-01T16:42:17.3716888Z Packages: read 2024-11-01T16:42:17.3717443Z Pages: read 2024-11-01T16:42:17.3717959Z PullRequests: read 2024-11-01T16:42:17.3718494Z RepositoryProjects: read 2024-11-01T16:42:17.3719147Z SecurityEvents: read 2024-11-01T16:42:17.3719719Z Statuses: read 2024-11-01T16:42:17.3720405Z ##[endgroup] 2024-11-01T16:42:17.3724401Z Secret source: Actions 2024-11-01T16:42:17.3725281Z Prepare workflow directory 2024-11-01T16:42:17.7423621Z Prepare all required actions 2024-11-01T16:42:17.7608878Z Getting action download info 2024-11-01T16:42:17.9626989Z Download action repository 'pytorch/test-infra@main' (SHA:49fb39b5efb49007791d74b09885044cd64544bf) 2024-11-01T16:42:18.3146263Z Download action repository 'pytorch/pytorch@main' (SHA:d79c5143d8e625947b93f8821bbe181419f47dee) 2024-11-01T16:42:21.4733782Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-11-01T16:42:21.5966024Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-11-01T16:42:21.8463620Z Getting action download info 2024-11-01T16:42:21.9457443Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-11-01T16:42:22.1121587Z Getting action download info 2024-11-01T16:42:22.2691527Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-11-01T16:42:22.4132784Z Getting action download info 2024-11-01T16:42:22.5146566Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-11-01T16:42:22.6541220Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/138766/merge (368da7510f5d755034681777a4e5ae6d33c07b38) 2024-11-01T16:42:22.6543988Z ##[group] Inputs 2024-11-01T16:42:22.6544482Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:42:22.6546589Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:42:22.6549097Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:42:22.6550240Z sync-tag: 2024-11-01T16:42:22.6550944Z timeout-minutes: 360 2024-11-01T16:42:22.6551281Z use-gha: 2024-11-01T16:42:22.6551566Z dashboard-tag: 2024-11-01T16:42:22.6551886Z s3-bucket: gha-artifacts 2024-11-01T16:42:22.6552244Z aws-role-to-assume: 2024-11-01T16:42:22.6552562Z ##[endgroup] 2024-11-01T16:42:22.6553493Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:42:22.7444915Z A job started hook has been configured by the self-hosted runner administrator 2024-11-01T16:42:22.7598649Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-11-01T16:42:22.7609452Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:22.7610001Z ##[endgroup] 2024-11-01T16:42:24.2917041Z Runner Type: linux.g4dn.12xlarge.nvidia.gpu 2024-11-01T16:42:24.2918058Z Instance Type: g4dn.12xlarge 2024-11-01T16:42:24.2918497Z AMI Name: unknown 2024-11-01T16:42:24.2919107Z AMI ID: ami-0fff1b9a61dec8a5f 2024-11-01T16:42:30.2143064Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-11-01T16:42:30.2143684Z with: 2024-11-01T16:42:30.2144387Z github-secret: *** 2024-11-01T16:42:30.2145390Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:30.2146480Z activate-with-label: false 2024-11-01T16:42:30.2146845Z label: with-ssh 2024-11-01T16:42:30.2147166Z remove-existing-keys: true 2024-11-01T16:42:30.2147517Z fail-silently: true 2024-11-01T16:42:30.2147825Z env: 2024-11-01T16:42:30.2148089Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.2148428Z ##[endgroup] 2024-11-01T16:42:30.3056088Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-11-01T16:42:30.6147453Z Grabbing public ssh keys from https://github.com/c00w.keys 2024-11-01T16:42:30.6712308Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-11-01T16:42:30.6727523Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-11-01T16:42:30.6754267Z Login using: ssh ec2-user@ec2-50-17-86-4.compute-1.amazonaws.com 2024-11-01T16:42:30.6755198Z All testing is done inside the container, to start an interactive session run: 2024-11-01T16:42:30.6756434Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-11-01T16:42:30.6888126Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-11-01T16:42:30.6888743Z with: 2024-11-01T16:42:30.6889020Z submodules: recursive 2024-11-01T16:42:30.6889384Z fetch-depth: 0 2024-11-01T16:42:30.6889686Z env: 2024-11-01T16:42:30.6889967Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.6890333Z ##[endgroup] 2024-11-01T16:42:30.7091836Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:30.7093365Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:42:30.7100961Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:30.7101481Z env: 2024-11-01T16:42:30.7101745Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.7102086Z ##[endgroup] 2024-11-01T16:42:30.7192471Z ##[group]Run retry () { 2024-11-01T16:42:30.7192918Z retry () { 2024-11-01T16:42:30.7193472Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-11-01T16:42:30.7194085Z } 2024-11-01T16:42:30.7194415Z echo "${GITHUB_WORKSPACE}" 2024-11-01T16:42:30.7194876Z if [ -z "${NO_SUDO}" ]; then 2024-11-01T16:42:30.7195404Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:30.7196180Z else 2024-11-01T16:42:30.7196577Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:42:30.7197046Z fi 2024-11-01T16:42:30.7197381Z mkdir "${GITHUB_WORKSPACE}" 2024-11-01T16:42:30.7203793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:42:30.7204364Z env: 2024-11-01T16:42:30.7204653Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:30.7205018Z NO_SUDO: 2024-11-01T16:42:30.7205311Z ##[endgroup] 2024-11-01T16:42:30.7230537Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.2394050Z ##[group]Run malfet/checkout@silent-checkout 2024-11-01T16:42:33.2394538Z with: 2024-11-01T16:42:33.2394894Z ref: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:42:33.2395379Z fetch-depth: 0 2024-11-01T16:42:33.2395814Z submodules: recursive 2024-11-01T16:42:33.2396359Z quiet-checkout: true 2024-11-01T16:42:33.2396743Z repository: pytorch/pytorch 2024-11-01T16:42:33.2397297Z token: *** 2024-11-01T16:42:33.2397592Z ssh-strict: true 2024-11-01T16:42:33.2398171Z persist-credentials: true 2024-11-01T16:42:33.2398568Z clean: true 2024-11-01T16:42:33.2398915Z sparse-checkout-cone-mode: true 2024-11-01T16:42:33.2399348Z lfs: false 2024-11-01T16:42:33.2399676Z set-safe-directory: true 2024-11-01T16:42:33.2400037Z env: 2024-11-01T16:42:33.2400329Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:42:33.2400707Z ##[endgroup] 2024-11-01T16:42:33.3406441Z Syncing repository: pytorch/pytorch 2024-11-01T16:42:33.3408331Z ##[group]Getting Git version info 2024-11-01T16:42:33.3409215Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:33.3410200Z [command]/usr/bin/git version 2024-11-01T16:42:33.3410588Z git version 2.40.1 2024-11-01T16:42:33.3412067Z ##[endgroup] 2024-11-01T16:42:33.3424451Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c51f6d47-9671-4c70-a0e3-a25069d21388' before making global git config changes 2024-11-01T16:42:33.3425917Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T16:42:33.3427247Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.3462108Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:42:33.3466477Z ##[group]Initializing the repository 2024-11-01T16:42:33.3469303Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:42:33.3502573Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-11-01T16:42:33.3504040Z hint: is subject to change. To configure the initial branch name to use in all 2024-11-01T16:42:33.3504984Z hint: of your new repositories, which will suppress this warning, call: 2024-11-01T16:42:33.3505622Z hint: 2024-11-01T16:42:33.3506279Z hint: git config --global init.defaultBranch 2024-11-01T16:42:33.3506803Z hint: 2024-11-01T16:42:33.3507377Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-11-01T16:42:33.3508367Z hint: 'development'. The just-created branch can be renamed via this command: 2024-11-01T16:42:33.3509027Z hint: 2024-11-01T16:42:33.3509364Z hint: git branch -m 2024-11-01T16:42:33.3510237Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-11-01T16:42:33.3511901Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-11-01T16:42:33.3545573Z ##[endgroup] 2024-11-01T16:42:33.3546204Z ##[group]Disabling automatic garbage collection 2024-11-01T16:42:33.3547069Z [command]/usr/bin/git config --local gc.auto 0 2024-11-01T16:42:33.3579315Z ##[endgroup] 2024-11-01T16:42:33.3579876Z ##[group]Setting up auth 2024-11-01T16:42:33.3618767Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T16:42:33.3621998Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T16:42:33.3910025Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T16:42:33.3941118Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T16:42:33.4236109Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:42:33.4282272Z ##[endgroup] 2024-11-01T16:42:33.4282908Z ##[group]Fetching the repository 2024-11-01T16:42:33.4289277Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-11-01T16:42:36.3118179Z remote: Enumerating objects: 1053148 2024-11-01T16:42:36.3119250Z remote: Enumerating objects: 1053710, done. 2024-11-01T16:42:36.3120425Z remote: Counting objects: 0% (1/562) 2024-11-01T16:42:36.3121885Z remote: Counting objects: 1% (6/562) 2024-11-01T16:42:36.3122618Z remote: Counting objects: 2% (12/562) 2024-11-01T16:42:36.3123172Z remote: Counting objects: 3% (17/562) 2024-11-01T16:42:36.3123720Z remote: Counting objects: 4% (23/562) 2024-11-01T16:42:36.3124278Z remote: Counting objects: 5% (29/562) 2024-11-01T16:42:36.3124829Z remote: Counting objects: 6% (34/562) 2024-11-01T16:42:36.3125376Z remote: Counting objects: 7% (40/562) 2024-11-01T16:42:36.3125915Z remote: Counting objects: 8% (45/562) 2024-11-01T16:42:36.3126465Z remote: Counting objects: 9% (51/562) 2024-11-01T16:42:36.3127006Z remote: Counting objects: 10% (57/562) 2024-11-01T16:42:36.3127550Z remote: Counting objects: 11% (62/562) 2024-11-01T16:42:36.3128094Z remote: Counting objects: 12% (68/562) 2024-11-01T16:42:36.3128632Z remote: Counting objects: 13% (74/562) 2024-11-01T16:42:36.3129160Z remote: Counting objects: 14% (79/562) 2024-11-01T16:42:36.3129711Z remote: Counting objects: 15% (85/562) 2024-11-01T16:42:36.3130257Z remote: Counting objects: 16% (90/562) 2024-11-01T16:42:36.3130799Z remote: Counting objects: 17% (96/562) 2024-11-01T16:42:36.3131349Z remote: Counting objects: 18% (102/562) 2024-11-01T16:42:36.3131897Z remote: Counting objects: 19% (107/562) 2024-11-01T16:42:36.3132458Z remote: Counting objects: 20% (113/562) 2024-11-01T16:42:36.3133164Z remote: Counting objects: 21% (119/562) 2024-11-01T16:42:36.3133729Z remote: Counting objects: 22% (124/562) 2024-11-01T16:42:36.3134289Z remote: Counting objects: 23% (130/562) 2024-11-01T16:42:36.3134842Z remote: Counting objects: 24% (135/562) 2024-11-01T16:42:36.3135377Z remote: Counting objects: 25% (141/562) 2024-11-01T16:42:36.3135932Z remote: Counting objects: 26% (147/562) 2024-11-01T16:42:36.3136491Z remote: Counting objects: 27% (152/562) 2024-11-01T16:42:36.3137047Z remote: Counting objects: 28% (158/562) 2024-11-01T16:42:36.3137599Z remote: Counting objects: 29% (163/562) 2024-11-01T16:42:36.3138149Z remote: Counting objects: 30% (169/562) 2024-11-01T16:42:36.3138683Z remote: Counting objects: 31% (175/562) 2024-11-01T16:42:36.3139236Z remote: Counting objects: 32% (180/562) 2024-11-01T16:42:36.3139785Z remote: Counting objects: 33% (186/562) 2024-11-01T16:42:36.3140338Z remote: Counting objects: 34% (192/562) 2024-11-01T16:42:36.3140888Z remote: Counting objects: 35% (197/562) 2024-11-01T16:42:36.3141419Z remote: Counting objects: 36% (203/562) 2024-11-01T16:42:36.3141970Z remote: Counting objects: 37% (208/562) 2024-11-01T16:42:36.3142521Z remote: Counting objects: 38% (214/562) 2024-11-01T16:42:36.3143070Z remote: Counting objects: 39% (220/562) 2024-11-01T16:42:36.3143626Z remote: Counting objects: 40% (225/562) 2024-11-01T16:42:36.3144174Z remote: Counting objects: 41% (231/562) 2024-11-01T16:42:36.3144707Z remote: Counting objects: 42% (237/562) 2024-11-01T16:42:36.3145253Z remote: Counting objects: 43% (242/562) 2024-11-01T16:42:36.3145802Z remote: Counting objects: 44% (248/562) 2024-11-01T16:42:36.3146349Z remote: Counting objects: 45% (253/562) 2024-11-01T16:42:36.3146896Z remote: Counting objects: 46% (259/562) 2024-11-01T16:42:36.3147435Z remote: Counting objects: 47% (265/562) 2024-11-01T16:42:36.3147979Z remote: Counting objects: 48% (270/562) 2024-11-01T16:42:36.3148530Z remote: Counting objects: 49% (276/562) 2024-11-01T16:42:36.3149076Z remote: Counting objects: 50% (281/562) 2024-11-01T16:42:36.3149622Z remote: Counting objects: 51% (287/562) 2024-11-01T16:42:36.3150171Z remote: Counting objects: 52% (293/562) 2024-11-01T16:42:36.3150714Z remote: Counting objects: 53% (298/562) 2024-11-01T16:42:36.3151344Z remote: Counting objects: 54% (304/562) 2024-11-01T16:42:36.3151893Z remote: Counting objects: 55% (310/562) 2024-11-01T16:42:36.3152445Z remote: Counting objects: 56% (315/562) 2024-11-01T16:42:36.3153058Z remote: Counting objects: 57% (321/562) 2024-11-01T16:42:36.3153612Z remote: Counting objects: 58% (326/562) 2024-11-01T16:42:36.3154144Z remote: Counting objects: 59% (332/562) 2024-11-01T16:42:36.3154700Z remote: Counting objects: 60% (338/562) 2024-11-01T16:42:36.3155253Z remote: Counting objects: 61% (343/562) 2024-11-01T16:42:36.3156072Z remote: Counting objects: 62% (349/562) 2024-11-01T16:42:36.3156694Z remote: Counting objects: 63% (355/562) 2024-11-01T16:42:36.3157265Z remote: Counting objects: 64% (360/562) 2024-11-01T16:42:36.3157829Z remote: Counting objects: 65% (366/562) 2024-11-01T16:42:36.3158381Z remote: Counting objects: 66% (371/562) 2024-11-01T16:42:36.3158959Z remote: Counting objects: 67% (377/562) 2024-11-01T16:42:36.3159527Z remote: Counting objects: 68% (383/562) 2024-11-01T16:42:36.3160095Z remote: Counting objects: 69% (388/562) 2024-11-01T16:42:36.3160654Z remote: Counting objects: 70% (394/562) 2024-11-01T16:42:36.3161204Z remote: Counting objects: 71% (400/562) 2024-11-01T16:42:36.3161869Z remote: Counting objects: 72% (405/562) 2024-11-01T16:42:36.3162495Z remote: Counting objects: 73% (411/562) 2024-11-01T16:42:36.3163046Z remote: Counting objects: 74% (416/562) 2024-11-01T16:42:36.3163597Z remote: Counting objects: 75% (422/562) 2024-11-01T16:42:36.3164153Z remote: Counting objects: 76% (428/562) 2024-11-01T16:42:36.3164687Z remote: Counting objects: 77% (433/562) 2024-11-01T16:42:36.3165236Z remote: Counting objects: 78% (439/562) 2024-11-01T16:42:36.3165782Z remote: Counting objects: 79% (444/562) 2024-11-01T16:42:36.3166339Z remote: Counting objects: 80% (450/562) 2024-11-01T16:42:36.3166888Z remote: Counting objects: 81% (456/562) 2024-11-01T16:42:36.3167436Z remote: Counting objects: 82% (461/562) 2024-11-01T16:42:36.3167969Z remote: Counting objects: 83% (467/562) 2024-11-01T16:42:36.3168520Z remote: Counting objects: 84% (473/562) 2024-11-01T16:42:36.3169067Z remote: Counting objects: 85% (478/562) 2024-11-01T16:42:36.3169613Z remote: Counting objects: 86% (484/562) 2024-11-01T16:42:36.3170164Z remote: Counting objects: 87% (489/562) 2024-11-01T16:42:36.3170698Z remote: Counting objects: 88% (495/562) 2024-11-01T16:42:36.3171248Z remote: Counting objects: 89% (501/562) 2024-11-01T16:42:36.3171795Z remote: Counting objects: 90% (506/562) 2024-11-01T16:42:36.3172348Z remote: Counting objects: 91% (512/562) 2024-11-01T16:42:36.3172892Z remote: Counting objects: 92% (518/562) 2024-11-01T16:42:36.3173444Z remote: Counting objects: 93% (523/562) 2024-11-01T16:42:36.3173981Z remote: Counting objects: 94% (529/562) 2024-11-01T16:42:36.3174531Z remote: Counting objects: 95% (534/562) 2024-11-01T16:42:36.3175079Z remote: Counting objects: 96% (540/562) 2024-11-01T16:42:36.3175629Z remote: Counting objects: 97% (546/562) 2024-11-01T16:42:36.3176175Z remote: Counting objects: 98% (551/562) 2024-11-01T16:42:36.3176711Z remote: Counting objects: 99% (557/562) 2024-11-01T16:42:36.3177264Z remote: Counting objects: 100% (562/562) 2024-11-01T16:42:36.3177855Z remote: Counting objects: 100% (562/562), done. 2024-11-01T16:42:36.3178469Z remote: Compressing objects: 0% (1/298) 2024-11-01T16:42:36.3179058Z remote: Compressing objects: 1% (3/298) 2024-11-01T16:42:36.3242426Z remote: Compressing objects: 2% (6/298) 2024-11-01T16:42:36.3375765Z remote: Compressing objects: 3% (9/298) 2024-11-01T16:42:36.3575369Z remote: Compressing objects: 4% (12/298) 2024-11-01T16:42:36.3657162Z remote: Compressing objects: 5% (15/298) 2024-11-01T16:42:36.3844625Z remote: Compressing objects: 6% (18/298) 2024-11-01T16:42:36.4032918Z remote: Compressing objects: 7% (21/298) 2024-11-01T16:42:36.4168822Z remote: Compressing objects: 8% (24/298) 2024-11-01T16:42:36.4234762Z remote: Compressing objects: 9% (27/298) 2024-11-01T16:42:36.4261501Z remote: Compressing objects: 10% (30/298) 2024-11-01T16:42:36.4262133Z remote: Compressing objects: 11% (33/298) 2024-11-01T16:42:36.4262853Z remote: Compressing objects: 12% (36/298) 2024-11-01T16:42:36.4263452Z remote: Compressing objects: 13% (39/298) 2024-11-01T16:42:36.4264049Z remote: Compressing objects: 14% (42/298) 2024-11-01T16:42:36.4264648Z remote: Compressing objects: 15% (45/298) 2024-11-01T16:42:36.4265231Z remote: Compressing objects: 16% (48/298) 2024-11-01T16:42:36.4265814Z remote: Compressing objects: 17% (51/298) 2024-11-01T16:42:36.4266403Z remote: Compressing objects: 18% (54/298) 2024-11-01T16:42:36.4266981Z remote: Compressing objects: 19% (57/298) 2024-11-01T16:42:36.4267574Z remote: Compressing objects: 20% (60/298) 2024-11-01T16:42:36.4268795Z remote: Compressing objects: 21% (63/298) 2024-11-01T16:42:36.4271459Z remote: Compressing objects: 22% (66/298) 2024-11-01T16:42:36.4274068Z remote: Compressing objects: 23% (69/298) 2024-11-01T16:42:36.4276786Z remote: Compressing objects: 24% (72/298) 2024-11-01T16:42:36.4277395Z remote: Compressing objects: 25% (75/298) 2024-11-01T16:42:36.4279056Z remote: Compressing objects: 26% (78/298) 2024-11-01T16:42:36.4279664Z remote: Compressing objects: 27% (81/298) 2024-11-01T16:42:36.4281619Z remote: Compressing objects: 28% (84/298) 2024-11-01T16:42:36.4282319Z remote: Compressing objects: 29% (87/298) 2024-11-01T16:42:36.4284118Z remote: Compressing objects: 30% (90/298) 2024-11-01T16:42:36.4284703Z remote: Compressing objects: 31% (93/298) 2024-11-01T16:42:36.4286882Z remote: Compressing objects: 32% (96/298) 2024-11-01T16:42:36.4289434Z remote: Compressing objects: 33% (99/298) 2024-11-01T16:42:36.4292153Z remote: Compressing objects: 34% (102/298) 2024-11-01T16:42:36.4296206Z remote: Compressing objects: 35% (105/298) 2024-11-01T16:42:36.4299032Z remote: Compressing objects: 36% (108/298) 2024-11-01T16:42:36.4301673Z remote: Compressing objects: 37% (111/298) 2024-11-01T16:42:36.4313591Z remote: Compressing objects: 38% (114/298) 2024-11-01T16:42:36.4315225Z remote: Compressing objects: 39% (117/298) 2024-11-01T16:42:36.4318201Z remote: Compressing objects: 40% (120/298) 2024-11-01T16:42:36.4320112Z remote: Compressing objects: 41% (123/298) 2024-11-01T16:42:36.4323128Z remote: Compressing objects: 42% (126/298) 2024-11-01T16:42:36.4325509Z remote: Compressing objects: 43% (129/298) 2024-11-01T16:42:36.4328132Z remote: Compressing objects: 44% (132/298) 2024-11-01T16:42:36.4331783Z remote: Compressing objects: 45% (135/298) 2024-11-01T16:42:36.4334576Z remote: Compressing objects: 46% (138/298) 2024-11-01T16:42:36.4337095Z remote: Compressing objects: 47% (141/298) 2024-11-01T16:42:36.4340100Z remote: Compressing objects: 48% (144/298) 2024-11-01T16:42:36.4344414Z remote: Compressing objects: 49% (147/298) 2024-11-01T16:42:36.4347357Z remote: Compressing objects: 50% (149/298) 2024-11-01T16:42:36.4350049Z remote: Compressing objects: 51% (152/298) 2024-11-01T16:42:36.4352786Z remote: Compressing objects: 52% (155/298) 2024-11-01T16:42:36.4355458Z remote: Compressing objects: 53% (158/298) 2024-11-01T16:42:36.4357346Z remote: Compressing objects: 54% (161/298) 2024-11-01T16:42:36.4358860Z remote: Compressing objects: 55% (164/298) 2024-11-01T16:42:36.4361426Z remote: Compressing objects: 56% (167/298) 2024-11-01T16:42:36.4365392Z remote: Compressing objects: 57% (170/298) 2024-11-01T16:42:36.4367992Z remote: Compressing objects: 58% (173/298) 2024-11-01T16:42:36.4369923Z remote: Compressing objects: 59% (176/298) 2024-11-01T16:42:36.4370957Z remote: Compressing objects: 60% (179/298) 2024-11-01T16:42:36.4371554Z remote: Compressing objects: 61% (182/298) 2024-11-01T16:42:36.4372139Z remote: Compressing objects: 62% (185/298) 2024-11-01T16:42:36.4372726Z remote: Compressing objects: 63% (188/298) 2024-11-01T16:42:36.4373299Z remote: Compressing objects: 64% (191/298) 2024-11-01T16:42:36.4373883Z remote: Compressing objects: 65% (194/298) 2024-11-01T16:42:36.4374470Z remote: Compressing objects: 66% (197/298) 2024-11-01T16:42:36.4375056Z remote: Compressing objects: 67% (200/298) 2024-11-01T16:42:36.4375644Z remote: Compressing objects: 68% (203/298) 2024-11-01T16:42:36.4376226Z remote: Compressing objects: 69% (206/298) 2024-11-01T16:42:36.4376795Z remote: Compressing objects: 70% (209/298) 2024-11-01T16:42:36.4377378Z remote: Compressing objects: 71% (212/298) 2024-11-01T16:42:36.4377962Z remote: Compressing objects: 72% (215/298) 2024-11-01T16:42:36.4378547Z remote: Compressing objects: 73% (218/298) 2024-11-01T16:42:36.4379246Z remote: Compressing objects: 74% (221/298) 2024-11-01T16:42:36.4379837Z remote: Compressing objects: 75% (224/298) 2024-11-01T16:42:36.4380410Z remote: Compressing objects: 76% (227/298) 2024-11-01T16:42:36.4380994Z remote: Compressing objects: 77% (230/298) 2024-11-01T16:42:36.4381583Z remote: Compressing objects: 78% (233/298) 2024-11-01T16:42:36.4382170Z remote: Compressing objects: 79% (236/298) 2024-11-01T16:42:36.4382767Z remote: Compressing objects: 80% (239/298) 2024-11-01T16:42:36.4383338Z remote: Compressing objects: 81% (242/298) 2024-11-01T16:42:36.4383926Z remote: Compressing objects: 82% (245/298) 2024-11-01T16:42:36.4384513Z remote: Compressing objects: 83% (248/298) 2024-11-01T16:42:36.4385285Z remote: Compressing objects: 84% (251/298) 2024-11-01T16:42:36.4385886Z remote: Compressing objects: 85% (254/298) 2024-11-01T16:42:36.4387122Z remote: Compressing objects: 86% (257/298) 2024-11-01T16:42:36.4387717Z remote: Compressing objects: 87% (260/298) 2024-11-01T16:42:36.4388298Z remote: Compressing objects: 88% (263/298) 2024-11-01T16:42:36.4389716Z remote: Compressing objects: 89% (266/298) 2024-11-01T16:42:36.4390307Z remote: Compressing objects: 90% (269/298) 2024-11-01T16:42:36.4390887Z remote: Compressing objects: 91% (272/298) 2024-11-01T16:42:36.4392204Z remote: Compressing objects: 92% (275/298) 2024-11-01T16:42:36.4392802Z remote: Compressing objects: 93% (278/298) 2024-11-01T16:42:36.4393393Z remote: Compressing objects: 94% (281/298) 2024-11-01T16:42:36.4394810Z remote: Compressing objects: 95% (284/298) 2024-11-01T16:42:36.4395397Z remote: Compressing objects: 96% (287/298) 2024-11-01T16:42:36.4396247Z remote: Compressing objects: 97% (290/298) 2024-11-01T16:42:36.4396846Z remote: Compressing objects: 98% (293/298) 2024-11-01T16:42:36.4397458Z remote: Compressing objects: 99% (296/298) 2024-11-01T16:42:36.4398063Z remote: Compressing objects: 100% (298/298) 2024-11-01T16:42:36.4398721Z remote: Compressing objects: 100% (298/298), done. 2024-11-01T16:42:58.6410177Z remote: Total 1053710 (delta 346), reused 431 (delta 263), pack-reused 1053148 (from 1) 2024-11-01T16:43:21.1155021Z [command]/usr/bin/git rev-parse --verify --quiet d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea^{object} 2024-11-01T16:43:21.1185441Z d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:21.1190509Z ##[endgroup] 2024-11-01T16:43:21.1191117Z ##[group]Determining the checkout info 2024-11-01T16:43:21.1191819Z ##[endgroup] 2024-11-01T16:43:21.1192472Z ##[group]Checking out the ref 2024-11-01T16:43:21.1193568Z [command]/usr/bin/git checkout --quiet --force d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:43:22.6158127Z ##[endgroup] 2024-11-01T16:43:22.6158816Z ##[group]Setting up auth for fetching submodules 2024-11-01T16:43:22.6160695Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:43:22.6217952Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-11-01T16:43:22.6249641Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-11-01T16:43:22.6281285Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-11-01T16:43:22.6313196Z ##[endgroup] 2024-11-01T16:43:22.6313790Z ##[group]Fetching submodules 2024-11-01T16:43:22.6315176Z [command]/usr/bin/git submodule sync --recursive 2024-11-01T16:43:22.6633453Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-11-01T16:43:22.6949037Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-11-01T16:43:22.6950987Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-11-01T16:43:22.6952686Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-11-01T16:43:22.6954313Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-11-01T16:43:22.6956517Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-11-01T16:43:22.6960043Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-11-01T16:43:22.6962397Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-11-01T16:43:22.6965548Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-11-01T16:43:22.6969397Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-11-01T16:43:22.6972532Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-11-01T16:43:22.6975711Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-11-01T16:43:22.6979276Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-11-01T16:43:22.6982587Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-11-01T16:43:22.6986552Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-11-01T16:43:22.6990411Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-11-01T16:43:22.6996112Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-11-01T16:43:22.7002135Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-11-01T16:43:22.7006578Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:43:22.7010704Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-11-01T16:43:22.7015182Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-11-01T16:43:22.7019401Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-11-01T16:43:22.7023844Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-11-01T16:43:22.7028350Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-11-01T16:43:22.7033045Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-11-01T16:43:22.7038060Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-11-01T16:43:22.7043064Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-11-01T16:43:22.7048233Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-11-01T16:43:22.7053736Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-11-01T16:43:22.7058691Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-11-01T16:43:22.7064212Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-11-01T16:43:22.7069526Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-11-01T16:43:22.7075129Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-11-01T16:43:22.7081105Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-11-01T16:43:22.7088871Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-11-01T16:43:22.7094833Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-11-01T16:43:22.7101311Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-11-01T16:43:22.7133209Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-11-01T16:43:23.0154998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-11-01T16:43:23.2171886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-11-01T16:43:23.4463448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-11-01T16:43:23.6725145Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-11-01T16:43:23.9820914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-11-01T16:43:25.7743277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-11-01T16:43:34.8158270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-11-01T16:43:35.2178405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-11-01T16:43:36.8230879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-11-01T16:43:37.3062610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-11-01T16:43:37.9039338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-11-01T16:43:39.1151550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-11-01T16:43:41.2181246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-11-01T16:43:46.5411524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-11-01T16:43:47.9425914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-11-01T16:43:49.5254997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-11-01T16:43:50.5434077Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-11-01T16:43:51.0288613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-11-01T16:43:51.4289749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-11-01T16:43:52.3705078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-11-01T16:43:52.7553923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-11-01T16:43:53.0928110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-11-01T16:43:55.2034346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-11-01T16:43:55.8989470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-11-01T16:43:56.2696984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-11-01T16:44:01.8512704Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-11-01T16:44:03.5928643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-11-01T16:44:10.1773635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-11-01T16:44:10.4266390Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-11-01T16:44:19.0866382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-11-01T16:44:19.3171140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-11-01T16:44:19.5558752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-11-01T16:44:20.5891074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-11-01T16:44:20.8788629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-11-01T16:44:21.5521302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-11-01T16:44:21.9534523Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-11-01T16:44:21.9660538Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-11-01T16:44:21.9755750Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-11-01T16:44:22.0013440Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-11-01T16:44:22.0326328Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-11-01T16:44:22.0736443Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-11-01T16:44:22.8964507Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-11-01T16:44:22.9213753Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-11-01T16:44:23.1359902Z Submodule path 'third_party/composable_kernel': checked out 'cedccd59c94cb0c74e7ec0d0f6c791aed081febc' 2024-11-01T16:44:23.1852220Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-11-01T16:44:23.2856393Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-11-01T16:44:23.3200811Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-11-01T16:44:23.8439019Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-11-01T16:44:24.1078317Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-11-01T16:44:24.2003212Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-11-01T16:44:24.2022685Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:44:24.2024461Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:44:24.2026224Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:44:24.2028243Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:44:24.2031231Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:44:24.2062284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-11-01T16:44:25.1220456Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-11-01T16:44:25.7247182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-11-01T16:44:27.7595911Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-11-01T16:44:28.6888415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-11-01T16:44:29.0647835Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-11-01T16:44:29.1640706Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-11-01T16:44:29.5788847Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-11-01T16:44:29.6434420Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-11-01T16:44:29.6561795Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-11-01T16:44:29.7777300Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-11-01T16:44:29.8178020Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-11-01T16:44:29.8584714Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-11-01T16:44:29.8858584Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-11-01T16:44:29.9322071Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:44:29.9458960Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-11-01T16:44:29.9477054Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-11-01T16:44:29.9505360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-11-01T16:44:43.4793341Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-11-01T16:44:43.4983676Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-11-01T16:44:43.5904190Z Submodule path 'third_party/kineto': checked out 'ed052ea024b9468908d558b15cd3f7584fb0f492' 2024-11-01T16:44:43.5920636Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:44:43.5923233Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:44:43.5925799Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:44:43.5954382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-11-01T16:44:44.1188957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-11-01T16:44:45.1602093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-11-01T16:44:46.1442501Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-11-01T16:44:46.1462185Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:44:46.1465092Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:44:46.1467588Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:44:46.1469793Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:44:46.1472171Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:44:46.1474823Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:44:46.1477805Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:44:46.1480753Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:44:46.1512041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-11-01T16:44:46.9750479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-11-01T16:44:47.3639024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-11-01T16:44:48.4018673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-11-01T16:44:48.7259406Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-11-01T16:44:49.3240783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-11-01T16:44:50.2197243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-11-01T16:44:55.7161242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-11-01T16:44:56.2601010Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-11-01T16:44:56.2797097Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-11-01T16:44:56.3184838Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-11-01T16:44:56.3325499Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-11-01T16:44:56.3343213Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:44:56.3372654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-11-01T16:44:57.0202030Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-11-01T16:44:57.0401846Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-11-01T16:44:57.0831885Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-11-01T16:44:57.1940369Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-11-01T16:44:57.2115375Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-11-01T16:44:57.2517730Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-11-01T16:44:57.3124493Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-11-01T16:44:57.3519085Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-11-01T16:44:57.3824591Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-11-01T16:44:57.4908836Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-11-01T16:44:57.8700427Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-11-01T16:44:57.8739019Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-11-01T16:44:57.8767794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-11-01T16:44:59.4679850Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-11-01T16:44:59.5398387Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-11-01T16:44:59.5418416Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:44:59.5420441Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:44:59.5422385Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:44:59.5424317Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:44:59.5426826Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:44:59.5429734Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:44:59.5432611Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:44:59.5435824Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:44:59.5465623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-11-01T16:45:00.1732831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-11-01T16:45:01.1029601Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-11-01T16:45:01.4795111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-11-01T16:45:06.9879066Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-11-01T16:45:07.2719470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-11-01T16:45:07.5216092Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-11-01T16:45:07.8482671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-11-01T16:45:13.8758952Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-11-01T16:45:13.9181640Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-11-01T16:45:13.9345313Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-11-01T16:45:14.0362822Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-11-01T16:45:14.0501310Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-11-01T16:45:14.0654105Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-11-01T16:45:14.0818952Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-11-01T16:45:14.0835092Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:14.0837699Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:14.0864774Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-11-01T16:45:15.6956625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-11-01T16:45:16.9406694Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-11-01T16:45:16.9903018Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:45:17.4568451Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-11-01T16:45:17.4692862Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-11-01T16:45:17.7643071Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-11-01T16:45:17.7667559Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:17.7669545Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:17.7696270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-11-01T16:45:18.2134871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-11-01T16:45:19.1180584Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-11-01T16:45:19.1916931Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-11-01T16:45:19.2017264Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-11-01T16:45:19.2147592Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-11-01T16:45:19.2508292Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-11-01T16:45:19.2818584Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-11-01T16:45:19.3256853Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-11-01T16:45:19.3527479Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-11-01T16:45:19.3546646Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:19.3548577Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:19.3550327Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:19.3552145Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:19.3581653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-11-01T16:45:20.2439185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-11-01T16:45:20.4854993Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-11-01T16:45:21.6827445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-11-01T16:45:22.6753124Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-11-01T16:45:22.6914126Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-11-01T16:45:22.7637465Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-11-01T16:45:22.7945477Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-11-01T16:45:22.7960609Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:22.7987895Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-11-01T16:45:23.0418926Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-11-01T16:45:23.0456896Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-11-01T16:45:23.0774008Z Entering 'android/libs/fbjni' 2024-11-01T16:45:23.0817437Z Entering 'third_party/FP16' 2024-11-01T16:45:23.0860235Z Entering 'third_party/FXdiv' 2024-11-01T16:45:23.0905842Z Entering 'third_party/NNPACK' 2024-11-01T16:45:23.0948668Z Entering 'third_party/NVTX' 2024-11-01T16:45:23.0993302Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:23.1042959Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:23.1107863Z Entering 'third_party/benchmark' 2024-11-01T16:45:23.1154244Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:23.1202501Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:23.1245082Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:23.1295736Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:23.1339385Z Entering 'third_party/cutlass' 2024-11-01T16:45:23.1390161Z Entering 'third_party/eigen' 2024-11-01T16:45:23.1437668Z Entering 'third_party/fbgemm' 2024-11-01T16:45:23.1479241Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:23.1525382Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:23.1575423Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:23.1625645Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:23.1668739Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:23.1712209Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:23.1758182Z Entering 'third_party/fmt' 2024-11-01T16:45:23.1801917Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:23.1844425Z Entering 'third_party/gloo' 2024-11-01T16:45:23.1888704Z Entering 'third_party/googletest' 2024-11-01T16:45:23.1937338Z Entering 'third_party/ideep' 2024-11-01T16:45:23.1978534Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:23.2031677Z Entering 'third_party/ittapi' 2024-11-01T16:45:23.2076040Z Entering 'third_party/kineto' 2024-11-01T16:45:23.2119530Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:23.2162124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:23.2205905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:23.2248242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:23.2290530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:23.2333554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:23.2378012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:23.2421801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:23.2476813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:23.2524788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:23.2578124Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:23.2620297Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:23.2668074Z Entering 'third_party/mimalloc' 2024-11-01T16:45:23.2712213Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:23.2756908Z Entering 'third_party/nlohmann' 2024-11-01T16:45:23.2800783Z Entering 'third_party/onnx' 2024-11-01T16:45:23.2860861Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:23.2908357Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:23.2953921Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:23.2997940Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:23.3039460Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:23.3080406Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:23.3124841Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:23.3166019Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:23.3207642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:23.3253428Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:23.3302319Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:23.3347945Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:23.3417596Z Entering 'third_party/pocketfft' 2024-11-01T16:45:23.3463552Z Entering 'third_party/protobuf' 2024-11-01T16:45:23.3511144Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:23.3553564Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:23.3600727Z Entering 'third_party/psimd' 2024-11-01T16:45:23.3643162Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:23.3685990Z Entering 'third_party/pybind11' 2024-11-01T16:45:23.3731960Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:23.3775083Z Entering 'third_party/sleef' 2024-11-01T16:45:23.3819411Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:23.3863071Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:23.3910232Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:23.3953094Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:23.3995129Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:23.4036869Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:23.4099396Z ##[endgroup] 2024-11-01T16:45:23.4100149Z ##[group]Persisting credentials for submodules 2024-11-01T16:45:23.4104717Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-11-01T16:45:23.4420287Z Entering 'android/libs/fbjni' 2024-11-01T16:45:23.4476881Z Entering 'third_party/FP16' 2024-11-01T16:45:23.4534470Z Entering 'third_party/FXdiv' 2024-11-01T16:45:23.4587909Z Entering 'third_party/NNPACK' 2024-11-01T16:45:23.4644930Z Entering 'third_party/NVTX' 2024-11-01T16:45:23.4706889Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:23.4763608Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:23.4837605Z Entering 'third_party/benchmark' 2024-11-01T16:45:23.4893207Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:23.4956634Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:23.5011875Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:23.5068127Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:23.5124064Z Entering 'third_party/cutlass' 2024-11-01T16:45:23.5187581Z Entering 'third_party/eigen' 2024-11-01T16:45:23.5248009Z Entering 'third_party/fbgemm' 2024-11-01T16:45:23.5306293Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:23.5359282Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:23.5416592Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:23.5480669Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:23.5536543Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:23.5591128Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:23.5652981Z Entering 'third_party/fmt' 2024-11-01T16:45:23.5708609Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:23.5765071Z Entering 'third_party/gloo' 2024-11-01T16:45:23.5821539Z Entering 'third_party/googletest' 2024-11-01T16:45:23.5877791Z Entering 'third_party/ideep' 2024-11-01T16:45:23.5934552Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:23.5997769Z Entering 'third_party/ittapi' 2024-11-01T16:45:23.6053905Z Entering 'third_party/kineto' 2024-11-01T16:45:23.6109633Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:23.6164497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:23.6222408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:23.6281039Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:23.6337665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:23.6395180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:23.6454953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:23.6510799Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:23.6565245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:23.6623311Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:23.6682867Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:23.6738977Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:23.6796312Z Entering 'third_party/mimalloc' 2024-11-01T16:45:23.6854304Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:23.6909961Z Entering 'third_party/nlohmann' 2024-11-01T16:45:23.6966889Z Entering 'third_party/onnx' 2024-11-01T16:45:23.7038020Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:23.7100916Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:23.7158001Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:23.7215567Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:23.7277128Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:23.7333957Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:23.7389911Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:23.7444525Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:23.7500926Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:23.7561847Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:23.7620859Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:23.7677691Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:23.7756633Z Entering 'third_party/pocketfft' 2024-11-01T16:45:23.7814039Z Entering 'third_party/protobuf' 2024-11-01T16:45:23.7877397Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:23.7935154Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:23.7994673Z Entering 'third_party/psimd' 2024-11-01T16:45:23.8052369Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:23.8107884Z Entering 'third_party/pybind11' 2024-11-01T16:45:23.8164347Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:23.8219690Z Entering 'third_party/sleef' 2024-11-01T16:45:23.8276677Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:23.8339596Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:23.8393102Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:23.8450133Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:23.8507030Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:23.8564358Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:23.8639175Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-11-01T16:45:23.8957690Z Entering 'android/libs/fbjni' 2024-11-01T16:45:23.9012089Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-11-01T16:45:23.9029133Z Entering 'third_party/FP16' 2024-11-01T16:45:23.9080719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-11-01T16:45:23.9096851Z Entering 'third_party/FXdiv' 2024-11-01T16:45:23.9151196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-11-01T16:45:23.9166390Z Entering 'third_party/NNPACK' 2024-11-01T16:45:23.9220707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-11-01T16:45:23.9237663Z Entering 'third_party/NVTX' 2024-11-01T16:45:23.9288279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-11-01T16:45:23.9307499Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:23.9358673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-11-01T16:45:23.9377806Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:23.9431176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-11-01T16:45:23.9467334Z Entering 'third_party/benchmark' 2024-11-01T16:45:23.9519853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:23.9538540Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:23.9591987Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-11-01T16:45:23.9617163Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:23.9673628Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-11-01T16:45:23.9690573Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:23.9746082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:23.9761665Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:23.9816930Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-11-01T16:45:23.9834722Z Entering 'third_party/cutlass' 2024-11-01T16:45:23.9886015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:23.9913558Z Entering 'third_party/eigen' 2024-11-01T16:45:23.9966426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-11-01T16:45:23.9987024Z Entering 'third_party/fbgemm' 2024-11-01T16:45:24.0038732Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-11-01T16:45:24.0056130Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:24.0107054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-11-01T16:45:24.0121504Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:24.0174533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:45:24.0190279Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:24.0241848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:45:24.0267036Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:24.0318803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.0370077Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:24.0388060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-11-01T16:45:24.0404593Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:24.0459166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-11-01T16:45:24.0478542Z Entering 'third_party/fmt' 2024-11-01T16:45:24.0532552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:24.0550318Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:24.0601142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-11-01T16:45:24.0619617Z Entering 'third_party/gloo' 2024-11-01T16:45:24.0671694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-11-01T16:45:24.0688863Z Entering 'third_party/googletest' 2024-11-01T16:45:24.0745559Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.0760956Z Entering 'third_party/ideep' 2024-11-01T16:45:24.0815204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-11-01T16:45:24.0831416Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:24.0880896Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-11-01T16:45:24.0909157Z Entering 'third_party/ittapi' 2024-11-01T16:45:24.0961073Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-11-01T16:45:24.0979373Z Entering 'third_party/kineto' 2024-11-01T16:45:24.1032290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-11-01T16:45:24.1047353Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:24.1103000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-11-01T16:45:24.1117927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:24.1170797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-11-01T16:45:24.1188183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:24.1239353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-11-01T16:45:24.1257287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:24.1310321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:45:24.1325386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:24.1384101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-11-01T16:45:24.1398457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:24.1453877Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-11-01T16:45:24.1473201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:24.1525298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-11-01T16:45:24.1543053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:24.1594575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.1614058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:24.1666491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-11-01T16:45:24.1683313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:24.1737462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-11-01T16:45:24.1756519Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:24.1808701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-11-01T16:45:24.1826791Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:24.1878670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.1895093Z Entering 'third_party/mimalloc' 2024-11-01T16:45:24.1949257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-11-01T16:45:24.1965130Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:24.2019409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-11-01T16:45:24.2037409Z Entering 'third_party/nlohmann' 2024-11-01T16:45:24.2088354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-11-01T16:45:24.2106644Z Entering 'third_party/onnx' 2024-11-01T16:45:24.2158485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-11-01T16:45:24.2191911Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:24.2246812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:24.2266834Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:24.2319147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-11-01T16:45:24.2338740Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:24.2388160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:24.2403428Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:24.2457462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.2473999Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:24.2534582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-11-01T16:45:24.2551192Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:24.2600940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-11-01T16:45:24.2620668Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:24.2670581Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-11-01T16:45:24.2685708Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:24.2739256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-11-01T16:45:24.2756169Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:24.2808119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-11-01T16:45:24.2825156Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:24.2875803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-11-01T16:45:24.2892542Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:24.2946604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-11-01T16:45:24.2963406Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:24.3016231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-11-01T16:45:24.3055960Z Entering 'third_party/pocketfft' 2024-11-01T16:45:24.3108261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-11-01T16:45:24.3123152Z Entering 'third_party/protobuf' 2024-11-01T16:45:24.3176490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-11-01T16:45:24.3196015Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:24.3247045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:45:24.3264557Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:24.3317770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.3336013Z Entering 'third_party/psimd' 2024-11-01T16:45:24.3388194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-11-01T16:45:24.3403484Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:24.3457045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-11-01T16:45:24.3474502Z Entering 'third_party/pybind11' 2024-11-01T16:45:24.3533690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:24.3552394Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:24.3602712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-11-01T16:45:24.3621242Z Entering 'third_party/sleef' 2024-11-01T16:45:24.3673471Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-11-01T16:45:24.3689672Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:24.3745207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-11-01T16:45:24.3760284Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:24.3815235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:45:24.3832294Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:24.3881694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-11-01T16:45:24.3900087Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:24.3951235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-11-01T16:45:24.3966588Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:24.4021187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:45:24.4035896Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:24.4088328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-11-01T16:45:24.4670902Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-11-01T16:45:24.4989017Z Entering 'android/libs/fbjni' 2024-11-01T16:45:24.5032775Z Entering 'third_party/FP16' 2024-11-01T16:45:24.5076627Z Entering 'third_party/FXdiv' 2024-11-01T16:45:24.5120013Z Entering 'third_party/NNPACK' 2024-11-01T16:45:24.5167853Z Entering 'third_party/NVTX' 2024-11-01T16:45:24.5212825Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:24.5255665Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:24.5318040Z Entering 'third_party/benchmark' 2024-11-01T16:45:24.5358486Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:24.5408980Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:24.5453572Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:24.5495747Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:24.5539793Z Entering 'third_party/cutlass' 2024-11-01T16:45:24.5589491Z Entering 'third_party/eigen' 2024-11-01T16:45:24.5636620Z Entering 'third_party/fbgemm' 2024-11-01T16:45:24.5678802Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:24.5723012Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:24.5764704Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:24.5817681Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:24.5859006Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:24.5903046Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:24.5951373Z Entering 'third_party/fmt' 2024-11-01T16:45:24.5994303Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:24.6037780Z Entering 'third_party/gloo' 2024-11-01T16:45:24.6084571Z Entering 'third_party/googletest' 2024-11-01T16:45:24.6136603Z Entering 'third_party/ideep' 2024-11-01T16:45:24.6178583Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:24.6230238Z Entering 'third_party/ittapi' 2024-11-01T16:45:24.6273797Z Entering 'third_party/kineto' 2024-11-01T16:45:24.6322802Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:24.6365101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:24.6411381Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:24.6456360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:24.6497685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:24.6541762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:24.6586160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:24.6630561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:24.6677789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:24.6721359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:24.6766286Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:24.6810015Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:24.6856656Z Entering 'third_party/mimalloc' 2024-11-01T16:45:24.6897846Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:24.6942739Z Entering 'third_party/nlohmann' 2024-11-01T16:45:24.6985744Z Entering 'third_party/onnx' 2024-11-01T16:45:24.7054051Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:24.7096958Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:24.7143438Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:24.7184873Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:24.7228174Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:24.7276680Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:24.7325014Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:24.7366595Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:24.7407942Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:24.7450791Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:24.7500466Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:24.7544291Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:24.7610693Z Entering 'third_party/pocketfft' 2024-11-01T16:45:24.7654273Z Entering 'third_party/protobuf' 2024-11-01T16:45:24.7700560Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:24.7742606Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:24.7787183Z Entering 'third_party/psimd' 2024-11-01T16:45:24.7831146Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:24.7877235Z Entering 'third_party/pybind11' 2024-11-01T16:45:24.7924652Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:24.7966622Z Entering 'third_party/sleef' 2024-11-01T16:45:24.8011375Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:24.8054180Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:24.8094389Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:24.8137112Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:24.8178383Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:24.8220708Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:24.8279801Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-11-01T16:45:24.8593189Z Entering 'android/libs/fbjni' 2024-11-01T16:45:24.8643526Z Entering 'third_party/FP16' 2024-11-01T16:45:24.8685506Z Entering 'third_party/FXdiv' 2024-11-01T16:45:24.8729713Z Entering 'third_party/NNPACK' 2024-11-01T16:45:24.8777419Z Entering 'third_party/NVTX' 2024-11-01T16:45:24.8820512Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:24.8863279Z Entering 'third_party/XNNPACK' 2024-11-01T16:45:24.8921800Z Entering 'third_party/benchmark' 2024-11-01T16:45:24.8964817Z Entering 'third_party/composable_kernel' 2024-11-01T16:45:24.9016763Z Entering 'third_party/cpp-httplib' 2024-11-01T16:45:24.9059186Z Entering 'third_party/cpuinfo' 2024-11-01T16:45:24.9104996Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:45:24.9148841Z Entering 'third_party/cutlass' 2024-11-01T16:45:24.9200241Z Entering 'third_party/eigen' 2024-11-01T16:45:24.9246231Z Entering 'third_party/fbgemm' 2024-11-01T16:45:24.9289524Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:45:24.9335145Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:45:24.9377163Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:45:24.9427272Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:45:24.9470492Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:45:24.9515673Z Entering 'third_party/flatbuffers' 2024-11-01T16:45:24.9559631Z Entering 'third_party/fmt' 2024-11-01T16:45:24.9603128Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:24.9646914Z Entering 'third_party/gloo' 2024-11-01T16:45:24.9690149Z Entering 'third_party/googletest' 2024-11-01T16:45:24.9734709Z Entering 'third_party/ideep' 2024-11-01T16:45:24.9777823Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:45:24.9830035Z Entering 'third_party/ittapi' 2024-11-01T16:45:24.9876943Z Entering 'third_party/kineto' 2024-11-01T16:45:24.9919499Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:45:24.9967187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:45:25.0014659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:45:25.0057992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:45:25.0102585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:45:25.0146503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:45:25.0190585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:45:25.0234588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:45:25.0277474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:45:25.0321285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:45:25.0364365Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:45:25.0406537Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:45:25.0457724Z Entering 'third_party/mimalloc' 2024-11-01T16:45:25.0502419Z Entering 'third_party/nccl/nccl' 2024-11-01T16:45:25.0544995Z Entering 'third_party/nlohmann' 2024-11-01T16:45:25.0596010Z Entering 'third_party/onnx' 2024-11-01T16:45:25.0657308Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:45:25.0705323Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:45:25.0750647Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:45:25.0795737Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:45:25.0837834Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:45:25.0881362Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:45:25.0928577Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:45:25.0973333Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:45:25.1015680Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:45:25.1057172Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:45:25.1107091Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:45:25.1157474Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:45:25.1226479Z Entering 'third_party/pocketfft' 2024-11-01T16:45:25.1270641Z Entering 'third_party/protobuf' 2024-11-01T16:45:25.1317499Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:45:25.1358211Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:45:25.1402468Z Entering 'third_party/psimd' 2024-11-01T16:45:25.1445142Z Entering 'third_party/pthreadpool' 2024-11-01T16:45:25.1488237Z Entering 'third_party/pybind11' 2024-11-01T16:45:25.1537151Z Entering 'third_party/python-peachpy' 2024-11-01T16:45:25.1579908Z Entering 'third_party/sleef' 2024-11-01T16:45:25.1622756Z Entering 'third_party/tensorpipe' 2024-11-01T16:45:25.1666944Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:45:25.1709149Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:45:25.1752429Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:45:25.1793562Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:45:25.1836295Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:45:25.1899041Z ##[endgroup] 2024-11-01T16:45:25.1939851Z [command]/usr/bin/git log -1 --format='%H' 2024-11-01T16:45:25.1972211Z 'd1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea' 2024-11-01T16:45:25.2177180Z Prepare all required actions 2024-11-01T16:45:25.2177858Z Getting action download info 2024-11-01T16:45:25.3790971Z ##[group]Run ./.github/actions/setup-linux 2024-11-01T16:45:25.3791450Z env: 2024-11-01T16:45:25.3791733Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.3792115Z ##[endgroup] 2024-11-01T16:45:25.3859487Z ##[group]Run set -euo pipefail 2024-11-01T16:45:25.3859922Z set -euo pipefail 2024-11-01T16:45:25.3860315Z function get_ec2_metadata() { 2024-11-01T16:45:25.3860852Z  # Pulled from instance metadata endpoint for EC2 2024-11-01T16:45:25.3861806Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-11-01T16:45:25.3862639Z  category=$1 2024-11-01T16:45:25.3863180Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-11-01T16:45:25.3863848Z  runner_name_str=i-0a5b4d9b9fef8d246 2024-11-01T16:45:25.3864355Z  if [[ -f /.inarc ]]; then 2024-11-01T16:45:25.3864854Z  echo "ARC Runner, no info on ec2 metadata" 2024-11-01T16:45:25.3865439Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-11-01T16:45:25.3866156Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-11-01T16:45:25.3866792Z  else 2024-11-01T16:45:25.3868110Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-11-01T16:45:25.3869506Z  fi 2024-11-01T16:45:25.3869772Z } 2024-11-01T16:45:25.3870132Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-11-01T16:45:25.3870733Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-11-01T16:45:25.3871413Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-11-01T16:45:25.3872000Z echo "system info $(uname -a)" 2024-11-01T16:45:25.3879301Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.3879859Z env: 2024-11-01T16:45:25.3880157Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.3880543Z ##[endgroup] 2024-11-01T16:45:25.4028689Z ami-id: ami-0fff1b9a61dec8a5f 2024-11-01T16:45:25.4130177Z instance-id: i-0a5b4d9b9fef8d246 2024-11-01T16:45:25.4232559Z instance-type: g4dn.12xlarge 2024-11-01T16:45:25.4244975Z system info Linux ip-10-0-58-136.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-11-01T16:45:25.4269533Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:25.4270919Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:25.4277651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.4278217Z env: 2024-11-01T16:45:25.4278518Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.4278907Z ##[endgroup] 2024-11-01T16:45:25.4337659Z ##[group]Run if systemctl is-active --quiet docker; then 2024-11-01T16:45:25.4338276Z if systemctl is-active --quiet docker; then 2024-11-01T16:45:25.4338830Z  echo "Docker daemon is running..."; 2024-11-01T16:45:25.4339281Z else 2024-11-01T16:45:25.4339778Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-11-01T16:45:25.4340381Z fi 2024-11-01T16:45:25.4345870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:25.4346493Z env: 2024-11-01T16:45:25.4346764Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.4347104Z ##[endgroup] 2024-11-01T16:45:25.4422126Z Docker daemon is running... 2024-11-01T16:45:25.4490162Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:45:25.4490599Z with: 2024-11-01T16:45:25.4490879Z shell: bash 2024-11-01T16:45:25.4491341Z timeout_minutes: 5 2024-11-01T16:45:25.4491697Z max_attempts: 3 2024-11-01T16:45:25.4492033Z retry_wait_seconds: 30 2024-11-01T16:45:25.4495627Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-11-01T16:45:25.4499279Z polling_interval_seconds: 1 2024-11-01T16:45:25.4499654Z warning_on_retry: true 2024-11-01T16:45:25.4500001Z continue_on_error: false 2024-11-01T16:45:25.4500337Z env: 2024-11-01T16:45:25.4500595Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:25.4500942Z AWS_RETRY_MODE: standard 2024-11-01T16:45:25.4501287Z AWS_MAX_ATTEMPTS: 5 2024-11-01T16:45:25.4501624Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:45:25.4501991Z ##[endgroup] 2024-11-01T16:45:26.7525091Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:26.7526078Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:26.7527082Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:26.7527724Z 2024-11-01T16:45:26.7527842Z Login Succeeded 2024-11-01T16:45:27.6277172Z Command completed after 1 attempt(s). 2024-11-01T16:45:27.6349662Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:27.6350440Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:27.6351165Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:45:27.6358353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.6358932Z env: 2024-11-01T16:45:27.6359232Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.6359617Z ##[endgroup] 2024-11-01T16:45:27.6457914Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:27.6458718Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:45:27.6459324Z # shellcheck disable=SC2046 2024-11-01T16:45:27.6459786Z docker stop $(docker ps -q) || true 2024-11-01T16:45:27.6460268Z # Prune all of the docker images 2024-11-01T16:45:27.6460719Z docker system prune -af 2024-11-01T16:45:27.6466230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.6466735Z env: 2024-11-01T16:45:27.6467003Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.6467346Z ##[endgroup] 2024-11-01T16:45:27.6742156Z "docker stop" requires at least 1 argument. 2024-11-01T16:45:27.6742979Z See 'docker stop --help'. 2024-11-01T16:45:27.6743252Z 2024-11-01T16:45:27.6743508Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-11-01T16:45:27.6743933Z 2024-11-01T16:45:27.6744100Z Stop one or more running containers 2024-11-01T16:45:27.6891833Z Total reclaimed space: 0B 2024-11-01T16:45:27.6929324Z ##[group]Run set +e 2024-11-01T16:45:27.6929705Z set +e 2024-11-01T16:45:27.6930118Z set -x 2024-11-01T16:45:27.6930414Z  2024-11-01T16:45:27.6930747Z PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:27.6931614Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-11-01T16:45:27.6932838Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-11-01T16:45:27.6933792Z # one is returned at random 2024-11-01T16:45:27.6934374Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-11-01T16:45:27.6934915Z  2024-11-01T16:45:27.6935360Z if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:27.6936038Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-11-01T16:45:27.6936997Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-11-01T16:45:27.6937567Z  2024-11-01T16:45:27.6937868Z  if [ -z "${RESOLVED_IP}" ]; then 2024-11-01T16:45:27.6938431Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-11-01T16:45:27.6938948Z  exit 1 2024-11-01T16:45:27.6939251Z  fi 2024-11-01T16:45:27.6939525Z fi 2024-11-01T16:45:27.6939785Z  2024-11-01T16:45:27.6940119Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-11-01T16:45:27.6940651Z  # Clean up any old records first 2024-11-01T16:45:27.6941166Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-11-01T16:45:27.6941623Z fi 2024-11-01T16:45:27.6941885Z  2024-11-01T16:45:27.6942304Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-11-01T16:45:27.6942867Z cat /etc/hosts 2024-11-01T16:45:27.6948620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.6949136Z env: 2024-11-01T16:45:27.6949403Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.6949745Z ##[endgroup] 2024-11-01T16:45:27.6973398Z + PT_DOMAIN=download.pytorch.org 2024-11-01T16:45:27.6980009Z ++ dig -4 +short download.pytorch.org 2024-11-01T16:45:27.6980492Z ++ tail -n1 2024-11-01T16:45:27.7547070Z + RESOLVED_IP=18.160.10.36 2024-11-01T16:45:27.7548149Z + '[' -z 18.160.10.36 ']' 2024-11-01T16:45:27.7548792Z + grep -r download.pytorch.org /etc/hosts 2024-11-01T16:45:27.7559828Z 18.160.10.22 download.pytorch.org 2024-11-01T16:45:27.7561310Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-11-01T16:45:27.9263632Z + echo '18.160.10.36 download.pytorch.org' 2024-11-01T16:45:27.9264227Z + sudo tee -a /etc/hosts 2024-11-01T16:45:27.9676745Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:27.9690104Z + cat /etc/hosts 2024-11-01T16:45:27.9699254Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-11-01T16:45:27.9711761Z ::1 localhost6 localhost6.localdomain6 2024-11-01T16:45:27.9712390Z 18.160.10.36 download.pytorch.org 2024-11-01T16:45:27.9929788Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-11-01T16:45:27.9930413Z with: 2024-11-01T16:45:27.9931680Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:27.9932930Z docker-build-dir: .ci/docker 2024-11-01T16:45:27.9933330Z working-directory: . 2024-11-01T16:45:27.9933833Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:27.9934413Z force-push: false 2024-11-01T16:45:27.9934723Z env: 2024-11-01T16:45:27.9934999Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.9935354Z ##[endgroup] 2024-11-01T16:45:27.9959458Z ##[group]Run set -ex 2024-11-01T16:45:27.9959856Z set -ex 2024-11-01T16:45:27.9960175Z  2024-11-01T16:45:27.9960824Z # If the docker build directory or the build script doesn't exist, the action will 2024-11-01T16:45:27.9962034Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-11-01T16:45:27.9962989Z # job could then download the pre-built image as usual 2024-11-01T16:45:27.9963846Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-11-01T16:45:27.9964638Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9965374Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9966814Z  2024-11-01T16:45:27.9967394Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-11-01T16:45:27.9968121Z  exit 0 2024-11-01T16:45:27.9968535Z else 2024-11-01T16:45:27.9968891Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9969345Z fi 2024-11-01T16:45:27.9969605Z  2024-11-01T16:45:27.9970071Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-11-01T16:45:27.9970975Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-11-01T16:45:27.9971776Z  # use it as it is, but first let's extract the tag 2024-11-01T16:45:27.9972507Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-11-01T16:45:27.9973265Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9973987Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9974560Z else 2024-11-01T16:45:27.9974996Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-11-01T16:45:27.9975677Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9976607Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:27.9977401Z fi 2024-11-01T16:45:27.9983251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:27.9983759Z env: 2024-11-01T16:45:27.9984021Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:27.9984375Z REPO_NAME: pytorch 2024-11-01T16:45:27.9985806Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:27.9987077Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:27.9987634Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:27.9988217Z ##[endgroup] 2024-11-01T16:45:28.0013692Z + [[ ! -d .ci/docker ]] 2024-11-01T16:45:28.0014145Z + [[ ! -f .ci/docker/build.sh ]] 2024-11-01T16:45:28.0014555Z + echo skip=false 2024-11-01T16:45:28.0016524Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-11-01T16:45:28.0021641Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0022897Z ++ awk -F '[:,]' '{print $2}' 2024-11-01T16:45:28.0042204Z + DOCKER_TAG=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0042987Z + echo docker-tag=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0044727Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0074706Z ##[group]Run set +e 2024-11-01T16:45:28.0075097Z set +e 2024-11-01T16:45:28.0075407Z set -x 2024-11-01T16:45:28.0075785Z  2024-11-01T16:45:28.0076240Z login() { 2024-11-01T16:45:28.0076996Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:28.0077846Z } 2024-11-01T16:45:28.0078142Z  2024-11-01T16:45:28.0078447Z retry () { 2024-11-01T16:45:28.0078880Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:28.0079386Z } 2024-11-01T16:45:28.0079668Z  2024-11-01T16:45:28.0080015Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:28.0080484Z  2024-11-01T16:45:28.0081019Z # Check if image already exists, if it does then skip building it 2024-11-01T16:45:28.0081967Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-11-01T16:45:28.0082531Z  exit 0 2024-11-01T16:45:28.0082857Z fi 2024-11-01T16:45:28.0083153Z  2024-11-01T16:45:28.0083728Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-11-01T16:45:28.0084710Z # be empty. The default action would be to continue rebuild the image 2024-11-01T16:45:28.0085783Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-11-01T16:45:28.0086560Z  # if we're on the base branch then use the parent commit 2024-11-01T16:45:28.0087248Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-11-01T16:45:28.0087737Z else 2024-11-01T16:45:28.0088381Z  # otherwise we're on a PR, so use the most recent base commit 2024-11-01T16:45:28.0089156Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-11-01T16:45:28.0089715Z fi 2024-11-01T16:45:28.0089985Z  2024-11-01T16:45:28.0090322Z if [[ -z "${MERGE_BASE}" ]]; then 2024-11-01T16:45:28.0090874Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:28.0091370Z  2024-11-01T16:45:28.0092104Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-11-01T16:45:28.0092978Z  exit 0 2024-11-01T16:45:28.0093278Z fi 2024-11-01T16:45:28.0093561Z  2024-11-01T16:45:28.0094029Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-11-01T16:45:28.0095155Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-11-01T16:45:28.0096086Z  exit 1 2024-11-01T16:45:28.0096398Z fi 2024-11-01T16:45:28.0096665Z  2024-11-01T16:45:28.0097409Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-11-01T16:45:28.0098418Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-11-01T16:45:28.0099340Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-11-01T16:45:28.0100368Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-11-01T16:45:28.0101543Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-11-01T16:45:28.0102205Z fi 2024-11-01T16:45:28.0102469Z  2024-11-01T16:45:28.0102946Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:45:28.0108601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:28.0109114Z env: 2024-11-01T16:45:28.0109378Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:28.0109723Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:45:28.0110194Z BASE_REVISION: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:45:28.0111420Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0112641Z DOCKER_TAG: bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.0113257Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.0113787Z ##[endgroup] 2024-11-01T16:45:28.0138532Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.0139294Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.0141343Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:28.0142537Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.5644202Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:28.5645425Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:28.5646429Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:28.5647342Z 2024-11-01T16:45:28.5647465Z Login Succeeded 2024-11-01T16:45:28.5661026Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.7686862Z { 2024-11-01T16:45:28.7687195Z "schemaVersion": 2, 2024-11-01T16:45:28.7687807Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-11-01T16:45:28.7688644Z "config": { 2024-11-01T16:45:28.7689193Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-11-01T16:45:28.7707338Z "size": 49259, 2024-11-01T16:45:28.7707990Z "digest": "sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b" 2024-11-01T16:45:28.7708733Z }, 2024-11-01T16:45:28.7709038Z "layers": [ 2024-11-01T16:45:28.7709329Z { 2024-11-01T16:45:28.7709818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7710443Z "size": 28580681, 2024-11-01T16:45:28.7711088Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-11-01T16:45:28.7711797Z }, 2024-11-01T16:45:28.7712225Z { 2024-11-01T16:45:28.7712718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7713358Z "size": 7943353, 2024-11-01T16:45:28.7714007Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-11-01T16:45:28.7714746Z }, 2024-11-01T16:45:28.7715006Z { 2024-11-01T16:45:28.7715511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7716282Z "size": 56226389, 2024-11-01T16:45:28.7716927Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-11-01T16:45:28.7717654Z }, 2024-11-01T16:45:28.7717919Z { 2024-11-01T16:45:28.7718409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7719267Z + exit 0 2024-11-01T16:45:28.7719544Z "size": 187, 2024-11-01T16:45:28.7720177Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-11-01T16:45:28.7720926Z }, 2024-11-01T16:45:28.7721185Z { 2024-11-01T16:45:28.7721674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7722301Z "size": 6885, 2024-11-01T16:45:28.7722938Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-11-01T16:45:28.7723669Z }, 2024-11-01T16:45:28.7723914Z { 2024-11-01T16:45:28.7724624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7725268Z "size": 1383662475, 2024-11-01T16:45:28.7725876Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-11-01T16:45:28.7726784Z }, 2024-11-01T16:45:28.7727032Z { 2024-11-01T16:45:28.7727498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7728110Z "size": 62282, 2024-11-01T16:45:28.7728687Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-11-01T16:45:28.7729361Z }, 2024-11-01T16:45:28.7729606Z { 2024-11-01T16:45:28.7730069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7730669Z "size": 1684, 2024-11-01T16:45:28.7731259Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-11-01T16:45:28.7732123Z }, 2024-11-01T16:45:28.7732374Z { 2024-11-01T16:45:28.7732845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7733470Z "size": 1522, 2024-11-01T16:45:28.7734060Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-11-01T16:45:28.7734750Z }, 2024-11-01T16:45:28.7734995Z { 2024-11-01T16:45:28.7735468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7736197Z "size": 2445884909, 2024-11-01T16:45:28.7736799Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-11-01T16:45:28.7737571Z }, 2024-11-01T16:45:28.7737929Z { 2024-11-01T16:45:28.7738356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7738932Z "size": 85882, 2024-11-01T16:45:28.7739497Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-11-01T16:45:28.7740134Z }, 2024-11-01T16:45:28.7740368Z { 2024-11-01T16:45:28.7740804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7741369Z "size": 1822, 2024-11-01T16:45:28.7741940Z "digest": "sha256:353d86cedcdc414207029061854fbc806c438ad1f5c00ac8aebf6daa9c2e9e9f" 2024-11-01T16:45:28.7742578Z }, 2024-11-01T16:45:28.7742816Z { 2024-11-01T16:45:28.7743436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7744049Z "size": 655141106, 2024-11-01T16:45:28.7744841Z "digest": "sha256:0299f4e4d5284c3c947696ef5f6ca6f7b5fab7881f4c4a6096399ca3e64f2c44" 2024-11-01T16:45:28.7745538Z }, 2024-11-01T16:45:28.7745792Z { 2024-11-01T16:45:28.7746270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7746895Z "size": 703, 2024-11-01T16:45:28.7747506Z "digest": "sha256:86c353c3bc581a31d9abdc8486224b29531a0b199d09e0ea8c5dadce351325b9" 2024-11-01T16:45:28.7748221Z }, 2024-11-01T16:45:28.7748460Z { 2024-11-01T16:45:28.7749041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7749645Z "size": 1259, 2024-11-01T16:45:28.7750361Z "digest": "sha256:4f10af237965d5a943f19bb5409eef94fae79f531a7e7137ea8a8815aff476b1" 2024-11-01T16:45:28.7751005Z }, 2024-11-01T16:45:28.7751226Z { 2024-11-01T16:45:28.7751663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7752240Z "size": 484, 2024-11-01T16:45:28.7752809Z "digest": "sha256:b7b00fcb07ca8c1ce9232c36e9c4bcd528d3167479c20a9e0b05a4c318417a21" 2024-11-01T16:45:28.7753469Z }, 2024-11-01T16:45:28.7753688Z { 2024-11-01T16:45:28.7754136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7754714Z "size": 91712745, 2024-11-01T16:45:28.7755317Z "digest": "sha256:db97392a2fce6d95ddfdcafbdee9fc4102fa976535ce9687d8f32dde3e188f61" 2024-11-01T16:45:28.7756272Z }, 2024-11-01T16:45:28.7756522Z { 2024-11-01T16:45:28.7757019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7757667Z "size": 3678, 2024-11-01T16:45:28.7758438Z "digest": "sha256:27e2d54f402a8b72082e53cf941bbc159517167ef364e2f9bee99de378effd71" 2024-11-01T16:45:28.7759168Z }, 2024-11-01T16:45:28.7759431Z { 2024-11-01T16:45:28.7759907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7760556Z "size": 1801, 2024-11-01T16:45:28.7761178Z "digest": "sha256:8269c56845c337b349db7a6d46a81a462af0d1183f2761ab3489909f0663638c" 2024-11-01T16:45:28.7761888Z }, 2024-11-01T16:45:28.7762267Z { 2024-11-01T16:45:28.7762691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7763277Z "size": 702, 2024-11-01T16:45:28.7763833Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:28.7764472Z }, 2024-11-01T16:45:28.7764708Z { 2024-11-01T16:45:28.7765132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7765713Z "size": 2921113290, 2024-11-01T16:45:28.7766290Z "digest": "sha256:82c15385d36d076d32b291ce1d513b5a7a94f02b56c06254edfca098f5e7ecba" 2024-11-01T16:45:28.7766931Z }, 2024-11-01T16:45:28.7767167Z { 2024-11-01T16:45:28.7767590Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7768167Z "size": 380, 2024-11-01T16:45:28.7768719Z "digest": "sha256:5ffae31f204af230644f54b5d1a9178b0313f3591541a866397342ba4177e782" 2024-11-01T16:45:28.7769354Z }, 2024-11-01T16:45:28.7769589Z { 2024-11-01T16:45:28.7770013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7770654Z "size": 12890, 2024-11-01T16:45:28.7771204Z "digest": "sha256:879b9cee6016a801826c1899830336c41370e72ae7b77305190ac6d2016bc8df" 2024-11-01T16:45:28.7771833Z }, 2024-11-01T16:45:28.7772070Z { 2024-11-01T16:45:28.7772514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7773072Z "size": 862, 2024-11-01T16:45:28.7773624Z "digest": "sha256:ead67a6283927375ef4a93495b43516a6a077bce28c304063c69a1c09c40cf72" 2024-11-01T16:45:28.7774260Z }, 2024-11-01T16:45:28.7774495Z { 2024-11-01T16:45:28.7774936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7775495Z "size": 106, 2024-11-01T16:45:28.7776064Z "digest": "sha256:2f25291ccc900ebaceb5c80edef126cf7641178b30cbc9ae518fad289b0beef0" 2024-11-01T16:45:28.7776729Z }, 2024-11-01T16:45:28.7776964Z { 2024-11-01T16:45:28.7777402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7777968Z "size": 505, 2024-11-01T16:45:28.7778542Z "digest": "sha256:5cdf685c64c3bd498aa6eb1dfce3a7e059b6f631b0c53f0bed7a78dc821391ef" 2024-11-01T16:45:28.7779208Z }, 2024-11-01T16:45:28.7779442Z { 2024-11-01T16:45:28.7779880Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7780457Z "size": 121477626, 2024-11-01T16:45:28.7781031Z "digest": "sha256:4eab1e4b631667fa7e17e7f2631d88a23b6869b53b25f0bce68b294ff2a25753" 2024-11-01T16:45:28.7781682Z }, 2024-11-01T16:45:28.7781917Z { 2024-11-01T16:45:28.7782359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7782936Z "size": 109, 2024-11-01T16:45:28.7783476Z "digest": "sha256:7caa1481847d89677e0ca3d3e4cfb637968c2e5fa17d24a78d061617629477a0" 2024-11-01T16:45:28.7784117Z }, 2024-11-01T16:45:28.7784351Z { 2024-11-01T16:45:28.7784787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7785750Z "size": 489, 2024-11-01T16:45:28.7786402Z "digest": "sha256:dafa8b6784dd9f01c5b1aefb90f8c8e21681ad0e63c7f6a4bb43db1e7e1eb0e2" 2024-11-01T16:45:28.7787158Z }, 2024-11-01T16:45:28.7787419Z { 2024-11-01T16:45:28.7787907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7788553Z "size": 296, 2024-11-01T16:45:28.7789195Z "digest": "sha256:dfccec371c6fccbb4240dd4ebb403f011e78b159e5dce84145a1ea58da02937e" 2024-11-01T16:45:28.7789951Z }, 2024-11-01T16:45:28.7790214Z { 2024-11-01T16:45:28.7790838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7791488Z "size": 103, 2024-11-01T16:45:28.7792106Z "digest": "sha256:e41e511e406bc1694c2e5da2a5ee58d9450141ae66b41c3e9ca24989dacc957c" 2024-11-01T16:45:28.7792838Z }, 2024-11-01T16:45:28.7793098Z { 2024-11-01T16:45:28.7793590Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7794234Z "size": 1473, 2024-11-01T16:45:28.7794842Z "digest": "sha256:7cd18613846984c3f25803460d6ec45d1d41db66bd4f9527b365889495421604" 2024-11-01T16:45:28.7795619Z }, 2024-11-01T16:45:28.7795892Z { 2024-11-01T16:45:28.7796381Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7797030Z "size": 426740495, 2024-11-01T16:45:28.7797694Z "digest": "sha256:38aadcbec80791616741dde49d32a2b56f6d3079d6cf8e823b4220a02eabcf1c" 2024-11-01T16:45:28.7798417Z }, 2024-11-01T16:45:28.7798686Z { 2024-11-01T16:45:28.7799177Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7799827Z "size": 164, 2024-11-01T16:45:28.7800459Z "digest": "sha256:f0b80c9d9a164550e271aa228fb110aefe145520e814c2fa44aa345ecb40f815" 2024-11-01T16:45:28.7801171Z }, 2024-11-01T16:45:28.7801433Z { 2024-11-01T16:45:28.7801924Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7802566Z "size": 565, 2024-11-01T16:45:28.7803195Z "digest": "sha256:bad6bd964ba483454e0e3f1a2b99160c38b3900658032c6597e8afb9a4c9c9c8" 2024-11-01T16:45:28.7804016Z }, 2024-11-01T16:45:28.7804262Z { 2024-11-01T16:45:28.7804750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7805396Z "size": 35806344, 2024-11-01T16:45:28.7806037Z "digest": "sha256:a348147f958bd269f1391a5802dc1c9d7c6414005de995ca5e2d1d8b4867d087" 2024-11-01T16:45:28.7806751Z }, 2024-11-01T16:45:28.7806995Z { 2024-11-01T16:45:28.7807606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7808179Z "size": 104, 2024-11-01T16:45:28.7808754Z "digest": "sha256:a3dfd1341f5cb909b01d7c3816d56e35af65a2728bffc76a810ca40cfc3a6dc3" 2024-11-01T16:45:28.7809414Z }, 2024-11-01T16:45:28.7809634Z { 2024-11-01T16:45:28.7810074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7810651Z "size": 425, 2024-11-01T16:45:28.7811217Z "digest": "sha256:c6d6b49273d1097ce850bdcaa1a0f5a19ad7a7897388642b190dcbdfb906f7f1" 2024-11-01T16:45:28.7811874Z }, 2024-11-01T16:45:28.7812093Z { 2024-11-01T16:45:28.7812532Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7813109Z "size": 20262172, 2024-11-01T16:45:28.7813679Z "digest": "sha256:9dec96f6a1c93d47374302f17840755ee42274e1a7c0aa05659042c2e3b3c5fb" 2024-11-01T16:45:28.7814321Z }, 2024-11-01T16:45:28.7814543Z { 2024-11-01T16:45:28.7814981Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7815556Z "size": 440, 2024-11-01T16:45:28.7816115Z "digest": "sha256:98a2657c4e6010d698d905ba80b545f25d8103e49e3da9e17c15359af100b9ca" 2024-11-01T16:45:28.7816758Z }, 2024-11-01T16:45:28.7816989Z { 2024-11-01T16:45:28.7817411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7817984Z "size": 702, 2024-11-01T16:45:28.7818539Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:28.7819179Z }, 2024-11-01T16:45:28.7819413Z { 2024-11-01T16:45:28.7819845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7820420Z "size": 141, 2024-11-01T16:45:28.7820979Z "digest": "sha256:955cea5b7f364a6e63ba32e894866f7a53b915e04275cc94a39cddd565d9649f" 2024-11-01T16:45:28.7821625Z }, 2024-11-01T16:45:28.7821863Z { 2024-11-01T16:45:28.7822289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7822861Z "size": 135, 2024-11-01T16:45:28.7825015Z "digest": "sha256:f38b3a72f6838f3a8895fa755edba3b26894109b9b4cf953a046145ffead84c7" 2024-11-01T16:45:28.7825686Z }, 2024-11-01T16:45:28.7825922Z { 2024-11-01T16:45:28.7826350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7826932Z "size": 32, 2024-11-01T16:45:28.7827496Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7828148Z }, 2024-11-01T16:45:28.7828379Z { 2024-11-01T16:45:28.7828818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7829390Z "size": 195, 2024-11-01T16:45:28.7829977Z "digest": "sha256:c5cb1a39bf9af5ffdd89e40e62f8019aae4b2dccfd34bc5972e4ab5ae8300688" 2024-11-01T16:45:28.7830649Z }, 2024-11-01T16:45:28.7830882Z { 2024-11-01T16:45:28.7831318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7831879Z "size": 565, 2024-11-01T16:45:28.7832451Z "digest": "sha256:6765d3ee658a82d6af47206ebbed1dd650339da0dacbd454d6c3dbb8f635a9ab" 2024-11-01T16:45:28.7833113Z }, 2024-11-01T16:45:28.7833351Z { 2024-11-01T16:45:28.7833787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7834353Z "size": 43163402, 2024-11-01T16:45:28.7834923Z "digest": "sha256:cf81cf49414b46152b641c670668b110a04ac48412f9c4dc6613311cf3a0942d" 2024-11-01T16:45:28.7835644Z }, 2024-11-01T16:45:28.7835887Z { 2024-11-01T16:45:28.7836613Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7837331Z "size": 106, 2024-11-01T16:45:28.7837954Z "digest": "sha256:303676327e274ec659dcce26b1e2613e6952e538ab3568f619a5e1a8b603ec89" 2024-11-01T16:45:28.7838670Z }, 2024-11-01T16:45:28.7838933Z { 2024-11-01T16:45:28.7839425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7840103Z "size": 1401, 2024-11-01T16:45:28.7840733Z "digest": "sha256:09aac48ef4835fec34e2ae2ba94b6081c53d8a2d4fd257e6a0b73349800b9864" 2024-11-01T16:45:28.7841477Z }, 2024-11-01T16:45:28.7841738Z { 2024-11-01T16:45:28.7842236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7842888Z "size": 702, 2024-11-01T16:45:28.7843500Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:28.7844219Z }, 2024-11-01T16:45:28.7844481Z { 2024-11-01T16:45:28.7844972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7845615Z "size": 136, 2024-11-01T16:45:28.7846260Z "digest": "sha256:ce24c94fd6add217e412ea0dd91c6e0d8cee5c7f27d8700aa6fe56f32a46ef85" 2024-11-01T16:45:28.7847009Z }, 2024-11-01T16:45:28.7847275Z { 2024-11-01T16:45:28.7847766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7848519Z "size": 120, 2024-11-01T16:45:28.7849066Z "digest": "sha256:8fd476fea7ee8bf52b382a9037418721f15c17b35692cc1f728e52380dfa773c" 2024-11-01T16:45:28.7849716Z }, 2024-11-01T16:45:28.7849951Z { 2024-11-01T16:45:28.7850397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7850975Z "size": 4310238020, 2024-11-01T16:45:28.7851567Z "digest": "sha256:4cf3da11584940e52b04ab8799eb5c048bd93de4b8ae9a7a8cde0f70aad2326e" 2024-11-01T16:45:28.7852210Z }, 2024-11-01T16:45:28.7852446Z { 2024-11-01T16:45:28.7852886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7853461Z "size": 176, 2024-11-01T16:45:28.7854035Z "digest": "sha256:1de6b248716cf53af496379d6fd310a4a060fd7d801c9398c3cb180e9b1358a6" 2024-11-01T16:45:28.7854672Z }, 2024-11-01T16:45:28.7854909Z { 2024-11-01T16:45:28.7855348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7855926Z "size": 907, 2024-11-01T16:45:28.7856503Z "digest": "sha256:f4eec39b74d5ab5024e1ee6c0a12e5c875dff461e7f65e95cab5adf5b7b0e558" 2024-11-01T16:45:28.7857155Z }, 2024-11-01T16:45:28.7857390Z { 2024-11-01T16:45:28.7857827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7858479Z "size": 702, 2024-11-01T16:45:28.7859038Z "digest": "sha256:6c7e3cfd6320c1a2e3323c5b6ed269549b5102449ba19199811783455948b5cd" 2024-11-01T16:45:28.7859666Z }, 2024-11-01T16:45:28.7859905Z { 2024-11-01T16:45:28.7860346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7860925Z "size": 135, 2024-11-01T16:45:28.7861494Z "digest": "sha256:bc86c61c49d69deaa90f9b4c601c930d1ae073fcac2b1e505aa15210056bfc49" 2024-11-01T16:45:28.7862161Z }, 2024-11-01T16:45:28.7862386Z { 2024-11-01T16:45:28.7862827Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7863407Z "size": 32, 2024-11-01T16:45:28.7863976Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7864636Z }, 2024-11-01T16:45:28.7864860Z { 2024-11-01T16:45:28.7865301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7865881Z "size": 158, 2024-11-01T16:45:28.7866449Z "digest": "sha256:4790041b71a9d94be048d9d17de7eaf5ccc4fc6824ba1988a6c0e8d5d8bb39d8" 2024-11-01T16:45:28.7867100Z }, 2024-11-01T16:45:28.7867324Z { 2024-11-01T16:45:28.7867765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7868344Z "size": 2047, 2024-11-01T16:45:28.7868896Z "digest": "sha256:9113890e17732f4e737f88fd824bf1d95f4990250788e87d7eafca3ca8d3c145" 2024-11-01T16:45:28.7869608Z }, 2024-11-01T16:45:28.7869832Z { 2024-11-01T16:45:28.7870274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7870856Z "size": 7529790, 2024-11-01T16:45:28.7871442Z "digest": "sha256:ad6d4c88cacf37e10c86d50f66a2ba3cc404356243a1399b9986c73beeee06c0" 2024-11-01T16:45:28.7872108Z }, 2024-11-01T16:45:28.7872347Z { 2024-11-01T16:45:28.7872774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7873354Z "size": 165, 2024-11-01T16:45:28.7873917Z "digest": "sha256:760e1aa1b5e32af659fb52bd61597d28609b31878840d1d752e99f3cfd8ce3ec" 2024-11-01T16:45:28.7874571Z }, 2024-11-01T16:45:28.7874807Z { 2024-11-01T16:45:28.7875234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7875889Z "size": 7943, 2024-11-01T16:45:28.7876709Z "digest": "sha256:3c4483bbdce846f42d57f201dfe75cda1f5046ec2ac3bcade9042e5adb8c5079" 2024-11-01T16:45:28.7877455Z }, 2024-11-01T16:45:28.7877724Z { 2024-11-01T16:45:28.7878203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7878854Z "size": 8071, 2024-11-01T16:45:28.7879495Z "digest": "sha256:8764bbbef59040f5d6a8293cfe6e35d1e0178ad129615f69281bcb3b4788b0f0" 2024-11-01T16:45:28.7880227Z }, 2024-11-01T16:45:28.7880487Z { 2024-11-01T16:45:28.7880966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7881616Z "size": 301, 2024-11-01T16:45:28.7882236Z "digest": "sha256:71826d5dd434366bb74c562c6725263f57f398f696093b870a0e1aa0af3834fe" 2024-11-01T16:45:28.7882946Z }, 2024-11-01T16:45:28.7883207Z { 2024-11-01T16:45:28.7883684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7884328Z "size": 7630059, 2024-11-01T16:45:28.7884978Z "digest": "sha256:7b4c7f0a396ed0af23fe3f9a24cdd2783c8558177cb3702476b8529d23144f4a" 2024-11-01T16:45:28.7885900Z }, 2024-11-01T16:45:28.7886164Z { 2024-11-01T16:45:28.7886668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7887304Z "size": 108, 2024-11-01T16:45:28.7887930Z "digest": "sha256:1200fc258863ca7884a699c7e393d80eb77d6dc143b3d42cc2fbba1cb84826c6" 2024-11-01T16:45:28.7888662Z }, 2024-11-01T16:45:28.7889097Z { 2024-11-01T16:45:28.7889588Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7890222Z "size": 54145652, 2024-11-01T16:45:28.7890866Z "digest": "sha256:1b6c965722e8b660c32a130dec4fcc09d3f8e1924d3351990c2317af7dae6911" 2024-11-01T16:45:28.7891731Z }, 2024-11-01T16:45:28.7891997Z { 2024-11-01T16:45:28.7892494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7893129Z "size": 472, 2024-11-01T16:45:28.7893774Z "digest": "sha256:6cf78d9b64805ade5c1afb24be9ca48e9ffa016e6091322be4ede273692a7d82" 2024-11-01T16:45:28.7894522Z }, 2024-11-01T16:45:28.7894784Z { 2024-11-01T16:45:28.7895277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7895933Z "size": 1373229322, 2024-11-01T16:45:28.7896595Z "digest": "sha256:accad7e3a239e2b010d87ef44b7db0fc6939535831f7ac907de46fd089b84fcb" 2024-11-01T16:45:28.7897526Z }, 2024-11-01T16:45:28.7897762Z { 2024-11-01T16:45:28.7898203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7898779Z "size": 103, 2024-11-01T16:45:28.7899338Z "digest": "sha256:b0b6f60fa25319ceba8a48c89bae7353160dafff3324f625d2c62b808da4d98e" 2024-11-01T16:45:28.7899999Z }, 2024-11-01T16:45:28.7900241Z { 2024-11-01T16:45:28.7900683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7901258Z "size": 569, 2024-11-01T16:45:28.7901818Z "digest": "sha256:acbcfa904fd51dbd77c9918823c59fc72b24bc49e9d3e843530877c2120b8db0" 2024-11-01T16:45:28.7902481Z }, 2024-11-01T16:45:28.7902714Z { 2024-11-01T16:45:28.7903155Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7903825Z "size": 36532055, 2024-11-01T16:45:28.7904397Z "digest": "sha256:024bd3c6bf01b2391930e2c9aa2648a121639bc8eadf45eaf325f74eb1a29822" 2024-11-01T16:45:28.7905070Z }, 2024-11-01T16:45:28.7905311Z { 2024-11-01T16:45:28.7905749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7906322Z "size": 111, 2024-11-01T16:45:28.7906891Z "digest": "sha256:9cc11594aa2a6ba9e369c7fdf6f7a27181e53af46336a4e825fb3fa888c2cdbc" 2024-11-01T16:45:28.7907538Z }, 2024-11-01T16:45:28.7907776Z { 2024-11-01T16:45:28.7908220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7908795Z "size": 531, 2024-11-01T16:45:28.7909533Z "digest": "sha256:01fd6f2e9fa39599a7a515a787a896943a1e32b00f54701d01633e1ab2d42449" 2024-11-01T16:45:28.7910210Z }, 2024-11-01T16:45:28.7910444Z { 2024-11-01T16:45:28.7910910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7911521Z "size": 25817, 2024-11-01T16:45:28.7912135Z "digest": "sha256:d06fa63dc0b7603ba664281c70a33befe32c8b1d0fa355cf9a520049165b88ea" 2024-11-01T16:45:28.7912828Z }, 2024-11-01T16:45:28.7913062Z { 2024-11-01T16:45:28.7913526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7914132Z "size": 106, 2024-11-01T16:45:28.7914719Z "digest": "sha256:72612b798350e7abf0b29d04df95850b30c9b7cb9b0b15787b9090a3b18ee673" 2024-11-01T16:45:28.7915396Z }, 2024-11-01T16:45:28.7915864Z { 2024-11-01T16:45:28.7916522Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7917169Z "size": 32, 2024-11-01T16:45:28.7917804Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7918537Z }, 2024-11-01T16:45:28.7918784Z { 2024-11-01T16:45:28.7919280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7919926Z "size": 32, 2024-11-01T16:45:28.7920559Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7921295Z }, 2024-11-01T16:45:28.7921557Z { 2024-11-01T16:45:28.7922031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7922673Z "size": 32, 2024-11-01T16:45:28.7923303Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7924035Z }, 2024-11-01T16:45:28.7924293Z { 2024-11-01T16:45:28.7924767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:45:28.7925493Z "size": 32, 2024-11-01T16:45:28.7926130Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-11-01T16:45:28.7926865Z } 2024-11-01T16:45:28.7927125Z ] 2024-11-01T16:45:28.7927370Z } 2024-11-01T16:45:28.8051598Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:28.8052084Z tag=${ECR_DOCKER_IMAGE##*/} 2024-11-01T16:45:28.8052627Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-11-01T16:45:28.8058682Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:28.8059195Z env: 2024-11-01T16:45:28.8059459Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:28.8060556Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.8061700Z ##[endgroup] 2024-11-01T16:45:28.8088747Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.8145364Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-11-01T16:45:28.8145955Z with: 2024-11-01T16:45:28.8146959Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.8148234Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.8148919Z env: 2024-11-01T16:45:28.8149186Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:28.8149533Z ##[endgroup] 2024-11-01T16:45:28.8169794Z ##[group]Run set -x 2024-11-01T16:45:28.8170138Z set -x 2024-11-01T16:45:28.8170418Z set +e 2024-11-01T16:45:28.8170702Z  2024-11-01T16:45:28.8170968Z login() { 2024-11-01T16:45:28.8171653Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:45:28.8172405Z } 2024-11-01T16:45:28.8172670Z  2024-11-01T16:45:28.8172970Z retry () { 2024-11-01T16:45:28.8173346Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:45:28.8173802Z } 2024-11-01T16:45:28.8174067Z  2024-11-01T16:45:28.8174372Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:45:28.8174790Z  2024-11-01T16:45:28.8175032Z set -e 2024-11-01T16:45:28.8175508Z # ignore output since only exit code is used for conditional 2024-11-01T16:45:28.8176247Z # only pull docker image if it's not available locally 2024-11-01T16:45:28.8177047Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-11-01T16:45:28.8177773Z  retry docker pull "${DOCKER_IMAGE}" 2024-11-01T16:45:28.8178220Z fi 2024-11-01T16:45:28.8183637Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:28.8184134Z env: 2024-11-01T16:45:28.8184402Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:28.8185872Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:28.8187311Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.8187916Z ##[endgroup] 2024-11-01T16:45:28.8212102Z + set +e 2024-11-01T16:45:28.8212733Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.8213466Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:28.8215994Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:45:28.8217449Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:45:29.3681213Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-11-01T16:45:29.3682357Z Configure a credential helper to remove this warning. See 2024-11-01T16:45:29.3683364Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-11-01T16:45:29.3683981Z 2024-11-01T16:45:29.3684140Z Login Succeeded 2024-11-01T16:45:29.3695849Z + set -e 2024-11-01T16:45:29.3697381Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:29.3838923Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:29.3841285Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:29.6006291Z bdd298b12da59246147f016e0693ffd722419941: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-11-01T16:45:29.6007270Z 7a2c55901189: Pulling fs layer 2024-11-01T16:45:29.6007689Z 09d415c238d7: Pulling fs layer 2024-11-01T16:45:29.6008106Z 9fe6e2e61518: Pulling fs layer 2024-11-01T16:45:29.6008758Z 41f16248e682: Pulling fs layer 2024-11-01T16:45:29.6009438Z 95d7b7817039: Pulling fs layer 2024-11-01T16:45:29.6009872Z 8f6c90485347: Pulling fs layer 2024-11-01T16:45:29.6010323Z ab17245097e4: Pulling fs layer 2024-11-01T16:45:29.6010741Z dfecd7e9912b: Pulling fs layer 2024-11-01T16:45:29.6011162Z 464a8f745445: Pulling fs layer 2024-11-01T16:45:29.6011581Z d67c111fa588: Pulling fs layer 2024-11-01T16:45:29.6011988Z 0723abde19c0: Pulling fs layer 2024-11-01T16:45:29.6012385Z 8f6c90485347: Waiting 2024-11-01T16:45:29.6012894Z 353d86cedcdc: Pulling fs layer 2024-11-01T16:45:29.6013294Z ab17245097e4: Waiting 2024-11-01T16:45:29.6013656Z 0299f4e4d528: Pulling fs layer 2024-11-01T16:45:29.6014073Z 86c353c3bc58: Pulling fs layer 2024-11-01T16:45:29.6014474Z 4f10af237965: Pulling fs layer 2024-11-01T16:45:29.6014894Z b7b00fcb07ca: Pulling fs layer 2024-11-01T16:45:29.6015313Z db97392a2fce: Pulling fs layer 2024-11-01T16:45:29.6015724Z 27e2d54f402a: Pulling fs layer 2024-11-01T16:45:29.6016132Z 8269c56845c3: Pulling fs layer 2024-11-01T16:45:29.6016511Z 41f16248e682: Waiting 2024-11-01T16:45:29.6016855Z d67c111fa588: Waiting 2024-11-01T16:45:29.6017217Z 6c7e3cfd6320: Pulling fs layer 2024-11-01T16:45:29.6017632Z 82c15385d36d: Pulling fs layer 2024-11-01T16:45:29.6018027Z 86c353c3bc58: Waiting 2024-11-01T16:45:29.6018357Z 0299f4e4d528: Waiting 2024-11-01T16:45:29.6018695Z 0723abde19c0: Waiting 2024-11-01T16:45:29.6019059Z 5ffae31f204a: Pulling fs layer 2024-11-01T16:45:29.6019465Z 353d86cedcdc: Waiting 2024-11-01T16:45:29.6019835Z 879b9cee6016: Pulling fs layer 2024-11-01T16:45:29.6020347Z ead67a628392: Pulling fs layer 2024-11-01T16:45:29.6020747Z 2f25291ccc90: Pulling fs layer 2024-11-01T16:45:29.6021149Z 5cdf685c64c3: Pulling fs layer 2024-11-01T16:45:29.6021712Z b7b00fcb07ca: Waiting 2024-11-01T16:45:29.6022061Z dfecd7e9912b: Waiting 2024-11-01T16:45:29.6022414Z 4eab1e4b6316: Pulling fs layer 2024-11-01T16:45:29.6038085Z 464a8f745445: Waiting 2024-11-01T16:45:29.6038528Z 8269c56845c3: Waiting 2024-11-01T16:45:29.6038868Z 4f10af237965: Waiting 2024-11-01T16:45:29.6039228Z 27e2d54f402a: Waiting 2024-11-01T16:45:29.6039582Z 95d7b7817039: Waiting 2024-11-01T16:45:29.6039954Z 7caa1481847d: Pulling fs layer 2024-11-01T16:45:29.6040387Z dafa8b6784dd: Pulling fs layer 2024-11-01T16:45:29.6040806Z dfccec371c6f: Pulling fs layer 2024-11-01T16:45:29.6041222Z 879b9cee6016: Waiting 2024-11-01T16:45:29.6041683Z 82c15385d36d: Waiting 2024-11-01T16:45:29.6042037Z e41e511e406b: Pulling fs layer 2024-11-01T16:45:29.6042452Z 7cd186138469: Pulling fs layer 2024-11-01T16:45:29.6042825Z 7caa1481847d: Waiting 2024-11-01T16:45:29.6043163Z 4eab1e4b6316: Waiting 2024-11-01T16:45:29.6043523Z 38aadcbec807: Pulling fs layer 2024-11-01T16:45:29.6043922Z 6c7e3cfd6320: Waiting 2024-11-01T16:45:29.6044278Z f0b80c9d9a16: Pulling fs layer 2024-11-01T16:45:29.6044652Z dafa8b6784dd: Waiting 2024-11-01T16:45:29.6045014Z bad6bd964ba4: Pulling fs layer 2024-11-01T16:45:29.6045424Z a348147f958b: Pulling fs layer 2024-11-01T16:45:29.6045815Z 7cd186138469: Waiting 2024-11-01T16:45:29.6046150Z f0b80c9d9a16: Waiting 2024-11-01T16:45:29.6046481Z 5cdf685c64c3: Waiting 2024-11-01T16:45:29.6046835Z a3dfd1341f5c: Pulling fs layer 2024-11-01T16:45:29.6047245Z c6d6b49273d1: Pulling fs layer 2024-11-01T16:45:29.6047653Z 9dec96f6a1c9: Pulling fs layer 2024-11-01T16:45:29.6048046Z 38aadcbec807: Waiting 2024-11-01T16:45:29.6048414Z bad6bd964ba4: Waiting 2024-11-01T16:45:29.6048754Z 98a2657c4e60: Pulling fs layer 2024-11-01T16:45:29.6049149Z a3dfd1341f5c: Waiting 2024-11-01T16:45:29.6049510Z 955cea5b7f36: Pulling fs layer 2024-11-01T16:45:29.6049915Z f38b3a72f683: Pulling fs layer 2024-11-01T16:45:29.6050300Z e41e511e406b: Waiting 2024-11-01T16:45:29.6050642Z 4f4fb700ef54: Pulling fs layer 2024-11-01T16:45:29.6051027Z 2f25291ccc90: Waiting 2024-11-01T16:45:29.6051387Z c5cb1a39bf9a: Pulling fs layer 2024-11-01T16:45:29.6051799Z 6765d3ee658a: Pulling fs layer 2024-11-01T16:45:29.6052187Z 4f4fb700ef54: Waiting 2024-11-01T16:45:29.6052519Z dfccec371c6f: Waiting 2024-11-01T16:45:29.6052878Z cf81cf49414b: Pulling fs layer 2024-11-01T16:45:29.6053445Z 303676327e27: Pulling fs layer 2024-11-01T16:45:29.6053854Z 09aac48ef483: Pulling fs layer 2024-11-01T16:45:29.6054246Z 09aac48ef483: Waiting 2024-11-01T16:45:29.6054590Z ce24c94fd6ad: Pulling fs layer 2024-11-01T16:45:29.6054979Z c6d6b49273d1: Waiting 2024-11-01T16:45:29.6055321Z 9dec96f6a1c9: Waiting 2024-11-01T16:45:29.6055681Z 8fd476fea7ee: Pulling fs layer 2024-11-01T16:45:29.6056088Z 4cf3da115849: Pulling fs layer 2024-11-01T16:45:29.6056562Z 1de6b248716c: Pulling fs layer 2024-11-01T16:45:29.6056973Z f4eec39b74d5: Pulling fs layer 2024-11-01T16:45:29.6057369Z 4cf3da115849: Waiting 2024-11-01T16:45:29.6057705Z 1de6b248716c: Waiting 2024-11-01T16:45:29.6058047Z 8fd476fea7ee: Waiting 2024-11-01T16:45:29.6058374Z ce24c94fd6ad: Waiting 2024-11-01T16:45:29.6058734Z bc86c61c49d6: Pulling fs layer 2024-11-01T16:45:29.6059119Z 303676327e27: Waiting 2024-11-01T16:45:29.6059451Z f4eec39b74d5: Waiting 2024-11-01T16:45:29.6059799Z 4790041b71a9: Pulling fs layer 2024-11-01T16:45:29.6060176Z f38b3a72f683: Waiting 2024-11-01T16:45:29.6060525Z 9113890e1773: Pulling fs layer 2024-11-01T16:45:29.6060932Z ad6d4c88cacf: Pulling fs layer 2024-11-01T16:45:29.6061340Z 760e1aa1b5e3: Pulling fs layer 2024-11-01T16:45:29.6061745Z 3c4483bbdce8: Pulling fs layer 2024-11-01T16:45:29.6062134Z 8764bbbef590: Pulling fs layer 2024-11-01T16:45:29.6062516Z 9113890e1773: Waiting 2024-11-01T16:45:29.6062864Z 71826d5dd434: Pulling fs layer 2024-11-01T16:45:29.6063270Z 7b4c7f0a396e: Pulling fs layer 2024-11-01T16:45:29.6063659Z ad6d4c88cacf: Waiting 2024-11-01T16:45:29.6064000Z 1200fc258863: Pulling fs layer 2024-11-01T16:45:29.6064390Z 7b4c7f0a396e: Waiting 2024-11-01T16:45:29.6064742Z 1b6c965722e8: Pulling fs layer 2024-11-01T16:45:29.6065157Z 6cf78d9b6480: Pulling fs layer 2024-11-01T16:45:29.6065563Z accad7e3a239: Pulling fs layer 2024-11-01T16:45:29.6065972Z b0b6f60fa253: Pulling fs layer 2024-11-01T16:45:29.6066347Z 1b6c965722e8: Waiting 2024-11-01T16:45:29.6066706Z acbcfa904fd5: Pulling fs layer 2024-11-01T16:45:29.6067122Z 024bd3c6bf01: Pulling fs layer 2024-11-01T16:45:29.6067507Z b0b6f60fa253: Waiting 2024-11-01T16:45:29.6067864Z 9cc11594aa2a: Pulling fs layer 2024-11-01T16:45:29.6068258Z 01fd6f2e9fa3: Pulling fs layer 2024-11-01T16:45:29.6068651Z 9cc11594aa2a: Waiting 2024-11-01T16:45:29.6069008Z d06fa63dc0b7: Pulling fs layer 2024-11-01T16:45:29.6069401Z 01fd6f2e9fa3: Waiting 2024-11-01T16:45:29.6069751Z 72612b798350: Pulling fs layer 2024-11-01T16:45:29.6070132Z d06fa63dc0b7: Waiting 2024-11-01T16:45:29.6070473Z 6cf78d9b6480: Waiting 2024-11-01T16:45:29.6070814Z accad7e3a239: Waiting 2024-11-01T16:45:29.6071151Z 024bd3c6bf01: Waiting 2024-11-01T16:45:29.6071483Z 1200fc258863: Waiting 2024-11-01T16:45:29.8199687Z 09d415c238d7: Verifying Checksum 2024-11-01T16:45:29.8200163Z 09d415c238d7: Download complete 2024-11-01T16:45:29.8975977Z 41f16248e682: Verifying Checksum 2024-11-01T16:45:29.8976460Z 41f16248e682: Download complete 2024-11-01T16:45:29.9336890Z 7a2c55901189: Verifying Checksum 2024-11-01T16:45:29.9337407Z 7a2c55901189: Download complete 2024-11-01T16:45:29.9837403Z 95d7b7817039: Verifying Checksum 2024-11-01T16:45:30.0498323Z ab17245097e4: Download complete 2024-11-01T16:45:30.1143424Z dfecd7e9912b: Verifying Checksum 2024-11-01T16:45:30.1143938Z dfecd7e9912b: Download complete 2024-11-01T16:45:30.1748074Z 464a8f745445: Verifying Checksum 2024-11-01T16:45:30.1748609Z 464a8f745445: Download complete 2024-11-01T16:45:30.2213699Z 9fe6e2e61518: Verifying Checksum 2024-11-01T16:45:30.2214279Z 9fe6e2e61518: Download complete 2024-11-01T16:45:30.2968088Z 0723abde19c0: Download complete 2024-11-01T16:45:30.3600481Z 353d86cedcdc: Verifying Checksum 2024-11-01T16:45:30.3600983Z 353d86cedcdc: Download complete 2024-11-01T16:45:30.6564299Z 7a2c55901189: Pull complete 2024-11-01T16:45:30.8504113Z 09d415c238d7: Pull complete 2024-11-01T16:45:31.6301909Z 9fe6e2e61518: Pull complete 2024-11-01T16:45:31.6499872Z 41f16248e682: Pull complete 2024-11-01T16:45:31.6684349Z 95d7b7817039: Pull complete 2024-11-01T16:45:36.9586049Z 0299f4e4d528: Verifying Checksum 2024-11-01T16:45:36.9586713Z 0299f4e4d528: Download complete 2024-11-01T16:45:37.0325159Z 86c353c3bc58: Verifying Checksum 2024-11-01T16:45:37.0325672Z 86c353c3bc58: Download complete 2024-11-01T16:45:37.1018086Z 4f10af237965: Verifying Checksum 2024-11-01T16:45:37.1018601Z 4f10af237965: Download complete 2024-11-01T16:45:37.1673198Z b7b00fcb07ca: Download complete 2024-11-01T16:45:38.1445604Z db97392a2fce: Verifying Checksum 2024-11-01T16:45:38.1446520Z db97392a2fce: Download complete 2024-11-01T16:45:38.2216386Z 27e2d54f402a: Verifying Checksum 2024-11-01T16:45:38.2217349Z 27e2d54f402a: Download complete 2024-11-01T16:45:38.3164521Z 8269c56845c3: Verifying Checksum 2024-11-01T16:45:38.3165025Z 8269c56845c3: Download complete 2024-11-01T16:45:38.3900072Z 6c7e3cfd6320: Verifying Checksum 2024-11-01T16:45:38.3900658Z 6c7e3cfd6320: Download complete 2024-11-01T16:45:43.8268205Z 8f6c90485347: Verifying Checksum 2024-11-01T16:45:43.8268957Z 8f6c90485347: Download complete 2024-11-01T16:45:43.9041631Z 5ffae31f204a: Verifying Checksum 2024-11-01T16:45:43.9042815Z 5ffae31f204a: Download complete 2024-11-01T16:45:43.9652437Z 879b9cee6016: Download complete 2024-11-01T16:45:44.0367838Z ead67a628392: Verifying Checksum 2024-11-01T16:45:44.0369057Z ead67a628392: Download complete 2024-11-01T16:45:44.1159050Z 2f25291ccc90: Download complete 2024-11-01T16:45:44.2389644Z 5cdf685c64c3: Download complete 2024-11-01T16:45:45.5095844Z 4eab1e4b6316: Verifying Checksum 2024-11-01T16:45:45.5096380Z 4eab1e4b6316: Download complete 2024-11-01T16:45:45.5894964Z 7caa1481847d: Download complete 2024-11-01T16:45:45.6828054Z dafa8b6784dd: Verifying Checksum 2024-11-01T16:45:45.6828598Z dafa8b6784dd: Download complete 2024-11-01T16:45:45.7857026Z dfccec371c6f: Verifying Checksum 2024-11-01T16:45:45.7857561Z dfccec371c6f: Download complete 2024-11-01T16:45:45.8515909Z e41e511e406b: Verifying Checksum 2024-11-01T16:45:45.8516652Z e41e511e406b: Download complete 2024-11-01T16:45:45.9192668Z 7cd186138469: Verifying Checksum 2024-11-01T16:45:45.9193207Z 7cd186138469: Download complete 2024-11-01T16:45:50.2419203Z 38aadcbec807: Verifying Checksum 2024-11-01T16:45:50.3499193Z 38aadcbec807: Download complete 2024-11-01T16:45:50.3500008Z f0b80c9d9a16: Verifying Checksum 2024-11-01T16:45:50.3500777Z f0b80c9d9a16: Download complete 2024-11-01T16:45:50.4334534Z bad6bd964ba4: Verifying Checksum 2024-11-01T16:45:50.4335533Z bad6bd964ba4: Download complete 2024-11-01T16:45:50.8487451Z a348147f958b: Verifying Checksum 2024-11-01T16:45:50.8488275Z a348147f958b: Download complete 2024-11-01T16:45:50.9256851Z a3dfd1341f5c: Verifying Checksum 2024-11-01T16:45:51.0213839Z c6d6b49273d1: Download complete 2024-11-01T16:45:51.3171821Z 9dec96f6a1c9: Download complete 2024-11-01T16:45:51.3941148Z 98a2657c4e60: Verifying Checksum 2024-11-01T16:45:51.3942202Z 98a2657c4e60: Download complete 2024-11-01T16:45:51.4634081Z 955cea5b7f36: Verifying Checksum 2024-11-01T16:45:51.4634628Z 955cea5b7f36: Download complete 2024-11-01T16:45:51.5432181Z f38b3a72f683: Verifying Checksum 2024-11-01T16:45:51.5432896Z f38b3a72f683: Download complete 2024-11-01T16:45:51.5494573Z 4f4fb700ef54: Verifying Checksum 2024-11-01T16:45:51.5495071Z 4f4fb700ef54: Download complete 2024-11-01T16:45:51.6179636Z c5cb1a39bf9a: Download complete 2024-11-01T16:45:51.6869154Z 6765d3ee658a: Verifying Checksum 2024-11-01T16:45:51.6869799Z 6765d3ee658a: Download complete 2024-11-01T16:45:52.1771040Z cf81cf49414b: Verifying Checksum 2024-11-01T16:45:52.1771920Z cf81cf49414b: Download complete 2024-11-01T16:45:52.2503211Z 303676327e27: Verifying Checksum 2024-11-01T16:45:52.2504020Z 303676327e27: Download complete 2024-11-01T16:45:52.3199387Z 09aac48ef483: Download complete 2024-11-01T16:45:52.3938401Z ce24c94fd6ad: Verifying Checksum 2024-11-01T16:45:52.3939232Z ce24c94fd6ad: Download complete 2024-11-01T16:45:52.4852700Z 8fd476fea7ee: Verifying Checksum 2024-11-01T16:45:52.4853533Z 8fd476fea7ee: Download complete 2024-11-01T16:45:54.6967565Z d67c111fa588: Verifying Checksum 2024-11-01T16:45:54.6968515Z d67c111fa588: Download complete 2024-11-01T16:45:54.7652300Z 1de6b248716c: Verifying Checksum 2024-11-01T16:45:54.7652846Z 1de6b248716c: Download complete 2024-11-01T16:45:54.8393310Z f4eec39b74d5: Verifying Checksum 2024-11-01T16:45:54.8393886Z f4eec39b74d5: Download complete 2024-11-01T16:45:54.9164080Z bc86c61c49d6: Verifying Checksum 2024-11-01T16:45:54.9164617Z bc86c61c49d6: Download complete 2024-11-01T16:45:54.9914114Z 4790041b71a9: Download complete 2024-11-01T16:45:55.0624737Z 9113890e1773: Download complete 2024-11-01T16:45:55.2277981Z ad6d4c88cacf: Verifying Checksum 2024-11-01T16:45:55.2278539Z ad6d4c88cacf: Download complete 2024-11-01T16:45:55.2950325Z 760e1aa1b5e3: Verifying Checksum 2024-11-01T16:45:55.2950868Z 760e1aa1b5e3: Download complete 2024-11-01T16:45:55.3640282Z 3c4483bbdce8: Download complete 2024-11-01T16:45:55.4403071Z 8764bbbef590: Download complete 2024-11-01T16:45:55.5091206Z 71826d5dd434: Download complete 2024-11-01T16:45:55.6311806Z 7b4c7f0a396e: Verifying Checksum 2024-11-01T16:45:55.6312392Z 7b4c7f0a396e: Download complete 2024-11-01T16:45:55.6843633Z 1200fc258863: Verifying Checksum 2024-11-01T16:45:55.6844165Z 1200fc258863: Download complete 2024-11-01T16:45:56.3013317Z 1b6c965722e8: Verifying Checksum 2024-11-01T16:45:56.3013863Z 1b6c965722e8: Download complete 2024-11-01T16:45:56.3839428Z 6cf78d9b6480: Verifying Checksum 2024-11-01T16:45:56.3839951Z 6cf78d9b6480: Download complete 2024-11-01T16:45:57.7940042Z 8f6c90485347: Pull complete 2024-11-01T16:45:58.1678467Z ab17245097e4: Pull complete 2024-11-01T16:45:58.6422265Z dfecd7e9912b: Pull complete 2024-11-01T16:45:59.1149251Z 464a8f745445: Pull complete 2024-11-01T16:46:07.6494107Z 82c15385d36d: Verifying Checksum 2024-11-01T16:46:07.6494643Z 82c15385d36d: Download complete 2024-11-01T16:46:07.7278724Z b0b6f60fa253: Verifying Checksum 2024-11-01T16:46:07.7279415Z b0b6f60fa253: Download complete 2024-11-01T16:46:07.8366370Z acbcfa904fd5: Verifying Checksum 2024-11-01T16:46:07.8366918Z acbcfa904fd5: Download complete 2024-11-01T16:46:08.2611186Z 024bd3c6bf01: Verifying Checksum 2024-11-01T16:46:08.2612130Z 024bd3c6bf01: Download complete 2024-11-01T16:46:08.3477641Z 9cc11594aa2a: Download complete 2024-11-01T16:46:08.4158338Z 01fd6f2e9fa3: Verifying Checksum 2024-11-01T16:46:08.4158872Z 01fd6f2e9fa3: Download complete 2024-11-01T16:46:08.4973072Z d06fa63dc0b7: Download complete 2024-11-01T16:46:08.5875355Z 72612b798350: Download complete 2024-11-01T16:46:10.1793592Z accad7e3a239: Verifying Checksum 2024-11-01T16:46:10.1794120Z accad7e3a239: Download complete 2024-11-01T16:46:24.3921537Z d67c111fa588: Pull complete 2024-11-01T16:46:24.8485449Z 0723abde19c0: Pull complete 2024-11-01T16:46:25.3058646Z 353d86cedcdc: Pull complete 2024-11-01T16:46:34.9823619Z 0299f4e4d528: Pull complete 2024-11-01T16:46:35.3717710Z 86c353c3bc58: Pull complete 2024-11-01T16:46:35.6376301Z 4cf3da115849: Verifying Checksum 2024-11-01T16:46:35.6376819Z 4cf3da115849: Download complete 2024-11-01T16:46:35.7652583Z 4f10af237965: Pull complete 2024-11-01T16:46:36.2050879Z b7b00fcb07ca: Pull complete 2024-11-01T16:46:38.4766664Z db97392a2fce: Pull complete 2024-11-01T16:46:38.8711449Z 27e2d54f402a: Pull complete 2024-11-01T16:46:39.2456330Z 8269c56845c3: Pull complete 2024-11-01T16:46:39.6545171Z 6c7e3cfd6320: Pull complete 2024-11-01T16:47:20.6330960Z 82c15385d36d: Pull complete 2024-11-01T16:47:20.8759082Z 5ffae31f204a: Pull complete 2024-11-01T16:47:21.1482348Z 879b9cee6016: Pull complete 2024-11-01T16:47:21.5739431Z ead67a628392: Pull complete 2024-11-01T16:47:22.0680938Z 2f25291ccc90: Pull complete 2024-11-01T16:47:22.4816464Z 5cdf685c64c3: Pull complete 2024-11-01T16:47:25.3198870Z 4eab1e4b6316: Pull complete 2024-11-01T16:47:25.8008156Z 7caa1481847d: Pull complete 2024-11-01T16:47:26.2777730Z dafa8b6784dd: Pull complete 2024-11-01T16:47:26.7267900Z dfccec371c6f: Pull complete 2024-11-01T16:47:27.1936000Z e41e511e406b: Pull complete 2024-11-01T16:47:27.6565621Z 7cd186138469: Pull complete 2024-11-01T16:47:34.9843121Z 38aadcbec807: Pull complete 2024-11-01T16:47:35.4374351Z f0b80c9d9a16: Pull complete 2024-11-01T16:47:35.8289160Z bad6bd964ba4: Pull complete 2024-11-01T16:47:36.7814934Z a348147f958b: Pull complete 2024-11-01T16:47:37.0369803Z a3dfd1341f5c: Pull complete 2024-11-01T16:47:37.2970871Z c6d6b49273d1: Pull complete 2024-11-01T16:47:37.7188906Z 9dec96f6a1c9: Pull complete 2024-11-01T16:47:37.9614966Z 98a2657c4e60: Pull complete 2024-11-01T16:47:38.3967451Z 955cea5b7f36: Pull complete 2024-11-01T16:47:38.7937460Z f38b3a72f683: Pull complete 2024-11-01T16:47:39.2565686Z 4f4fb700ef54: Pull complete 2024-11-01T16:47:39.6577743Z c5cb1a39bf9a: Pull complete 2024-11-01T16:47:40.1003988Z 6765d3ee658a: Pull complete 2024-11-01T16:47:42.1796121Z cf81cf49414b: Pull complete 2024-11-01T16:47:42.6604666Z 303676327e27: Pull complete 2024-11-01T16:47:43.1185582Z 09aac48ef483: Pull complete 2024-11-01T16:47:44.0359485Z ce24c94fd6ad: Pull complete 2024-11-01T16:47:44.4891186Z 8fd476fea7ee: Pull complete 2024-11-01T16:48:49.9583786Z 4cf3da115849: Pull complete 2024-11-01T16:48:50.4000127Z 1de6b248716c: Pull complete 2024-11-01T16:48:50.7774634Z f4eec39b74d5: Pull complete 2024-11-01T16:48:51.6644401Z bc86c61c49d6: Pull complete 2024-11-01T16:48:52.5896880Z 4790041b71a9: Pull complete 2024-11-01T16:48:53.0810032Z 9113890e1773: Pull complete 2024-11-01T16:48:53.6857961Z ad6d4c88cacf: Pull complete 2024-11-01T16:48:54.1422077Z 760e1aa1b5e3: Pull complete 2024-11-01T16:48:54.6314109Z 3c4483bbdce8: Pull complete 2024-11-01T16:48:55.0619932Z 8764bbbef590: Pull complete 2024-11-01T16:48:55.5309978Z 71826d5dd434: Pull complete 2024-11-01T16:48:56.6719553Z 7b4c7f0a396e: Pull complete 2024-11-01T16:48:57.0594105Z 1200fc258863: Pull complete 2024-11-01T16:48:58.9467385Z 1b6c965722e8: Pull complete 2024-11-01T16:48:59.3534786Z 6cf78d9b6480: Pull complete 2024-11-01T16:49:12.0439441Z accad7e3a239: Pull complete 2024-11-01T16:49:12.4947875Z b0b6f60fa253: Pull complete 2024-11-01T16:49:12.9472534Z acbcfa904fd5: Pull complete 2024-11-01T16:49:13.8108599Z 024bd3c6bf01: Pull complete 2024-11-01T16:49:14.2384850Z 9cc11594aa2a: Pull complete 2024-11-01T16:49:14.7086887Z 01fd6f2e9fa3: Pull complete 2024-11-01T16:49:15.2077442Z d06fa63dc0b7: Pull complete 2024-11-01T16:49:15.6358325Z 72612b798350: Pull complete 2024-11-01T16:49:17.5168138Z Digest: sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T16:49:17.6291243Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:17.6768971Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:49:17.6861186Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:17.6862706Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:49:17.6872110Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:49:17.6872655Z env: 2024-11-01T16:49:17.6872942Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:17.6873305Z ##[endgroup] 2024-11-01T16:49:17.7109761Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-11-01T16:49:17.7110339Z with: 2024-11-01T16:49:17.7110628Z driver-version: 550.54.15 2024-11-01T16:49:17.7111105Z env: 2024-11-01T16:49:17.7111369Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:17.7111710Z ##[endgroup] 2024-11-01T16:49:17.7353214Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-11-01T16:49:17.7353837Z with: 2024-11-01T16:49:17.7354107Z timeout_minutes: 10 2024-11-01T16:49:17.7354451Z max_attempts: 3 2024-11-01T16:49:17.7395370Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-11-01T16:49:17.7437366Z retry_wait_seconds: 10 2024-11-01T16:49:17.7437769Z polling_interval_seconds: 1 2024-11-01T16:49:17.7438189Z warning_on_retry: true 2024-11-01T16:49:17.7438574Z continue_on_error: false 2024-11-01T16:49:17.7438933Z env: 2024-11-01T16:49:17.7439221Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:49:17.7439614Z DRIVER_VERSION: 550.54.15 2024-11-01T16:49:17.7440004Z ##[endgroup] 2024-11-01T16:49:17.8266441Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-11-01T16:49:17.8268505Z + pre_install_nvidia_driver_amzn2 2024-11-01T16:49:17.8269225Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-11-01T16:49:18.2175325Z No match for argument: nvidia-driver-latest-dkms 2024-11-01T16:49:18.2175911Z No packages marked for removal. 2024-11-01T16:49:18.2241058Z Dependencies resolved. 2024-11-01T16:49:18.2250755Z Nothing to do. 2024-11-01T16:49:18.2251360Z Complete! 2024-11-01T16:49:18.3279563Z + install_nvidia_driver_common 2024-11-01T16:49:18.3281795Z + echo 'Before installing NVIDIA driver' 2024-11-01T16:49:18.3282297Z + lspci 2024-11-01T16:49:18.3283394Z Before installing NVIDIA driver 2024-11-01T16:49:18.3411941Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:18.3412834Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:18.3414746Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:18.3415711Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:18.3416646Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:18.3417572Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:18.3418443Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.3419268Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.3420281Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.3421097Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.3422063Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:18.3422729Z + lsmod 2024-11-01T16:49:18.3452768Z Module Size Used by 2024-11-01T16:49:18.3453661Z ib_core 454656 0 2024-11-01T16:49:18.3454142Z veth 36864 0 2024-11-01T16:49:18.3454567Z nvidia_modeset 1351680 0 2024-11-01T16:49:18.3455076Z video 65536 1 nvidia_modeset 2024-11-01T16:49:18.3455564Z wmi 36864 1 video 2024-11-01T16:49:18.3455998Z nvidia_uvm 4706304 0 2024-11-01T16:49:18.3456498Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:18.3457670Z drm 602112 1 nvidia 2024-11-01T16:49:18.3458157Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:18.3458780Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:18.3459351Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:18.3459818Z xt_conntrack 16384 1 2024-11-01T16:49:18.3460230Z nft_chain_nat 16384 3 2024-11-01T16:49:18.3460836Z xt_MASQUERADE 20480 1 2024-11-01T16:49:18.3461337Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:18.3461868Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:18.3462527Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:18.3463255Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:18.3463767Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:18.3464245Z xfrm_user 57344 1 2024-11-01T16:49:18.3464670Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:18.3465118Z xt_addrtype 16384 2 2024-11-01T16:49:18.3465530Z nft_compat 20480 4 2024-11-01T16:49:18.3466029Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:18.3466723Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:18.3467338Z br_netfilter 36864 0 2024-11-01T16:49:18.3467770Z bridge 323584 1 br_netfilter 2024-11-01T16:49:18.3468261Z stp 16384 1 bridge 2024-11-01T16:49:18.3468723Z llc 16384 2 bridge,stp 2024-11-01T16:49:18.3469181Z overlay 167936 0 2024-11-01T16:49:18.3469575Z tls 114688 0 2024-11-01T16:49:18.3469964Z nls_ascii 16384 1 2024-11-01T16:49:18.3470349Z nls_cp437 20480 1 2024-11-01T16:49:18.3470749Z vfat 24576 1 2024-11-01T16:49:18.3471144Z sunrpc 692224 1 2024-11-01T16:49:18.3471544Z fat 86016 1 vfat 2024-11-01T16:49:18.3471967Z ena 167936 0 2024-11-01T16:49:18.3472340Z i8042 45056 0 2024-11-01T16:49:18.3472737Z skx_edac_common 28672 0 2024-11-01T16:49:18.3473154Z serio 28672 3 i8042 2024-11-01T16:49:18.3473592Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:18.3474004Z button 24576 0 2024-11-01T16:49:18.3474390Z sch_fq_codel 20480 33 2024-11-01T16:49:18.3474799Z dm_mod 188416 0 2024-11-01T16:49:18.3475194Z fuse 163840 1 2024-11-01T16:49:18.3475584Z loop 36864 0 2024-11-01T16:49:18.3476124Z configfs 57344 1 2024-11-01T16:49:18.3476710Z dax 45056 1 dm_mod 2024-11-01T16:49:18.3477160Z dmi_sysfs 20480 0 2024-11-01T16:49:18.3477580Z crc32_pclmul 16384 0 2024-11-01T16:49:18.3477996Z crc32c_intel 24576 0 2024-11-01T16:49:18.3478411Z efivarfs 24576 1 2024-11-01T16:49:18.3478797Z + modinfo nvidia 2024-11-01T16:49:18.3479668Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:18.3480512Z alias: char-major-195-* 2024-11-01T16:49:18.3481044Z version: 550.54.15 2024-11-01T16:49:18.3481436Z supported: external 2024-11-01T16:49:18.3481813Z license: NVIDIA 2024-11-01T16:49:18.3482221Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:18.3482781Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:18.3483307Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:18.3483838Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:18.3484414Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:18.3484983Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:18.3485723Z depends: i2c-core,drm 2024-11-01T16:49:18.3486127Z retpoline: Y 2024-11-01T16:49:18.3486466Z name: nvidia 2024-11-01T16:49:18.3487196Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:18.3488037Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:18.3488792Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:18.3489494Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:18.3490003Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:18.3490495Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:18.3491137Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:18.3491633Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:18.3492117Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:18.3492733Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:18.3493391Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:18.3493940Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:18.3494427Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:18.3494882Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:18.3495398Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:18.3496008Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:18.3496702Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:18.3497562Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:18.3498236Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:18.3499002Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:18.3499645Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:18.3500263Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:18.3500767Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:18.3501318Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:18.3501872Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:18.3502381Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:18.3502852Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:18.3503344Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:18.3503824Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:18.3504278Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:18.3504779Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:18.3505322Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:18.3505814Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:18.3506310Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:18.3506822Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:18.3507304Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:18.3507819Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:18.3508308Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:18.3508723Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:18.3509198Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:18.3509681Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:18.3510136Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:18.3510626Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:18.3511201Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:18.3511669Z parm: rm_firmware_active:charp 2024-11-01T16:49:18.3512093Z + HAS_NVIDIA_DRIVER=0 2024-11-01T16:49:18.3512464Z ++ command -v nvidia-smi 2024-11-01T16:49:18.3512868Z + '[' -x /usr/bin/nvidia-smi ']' 2024-11-01T16:49:18.3513236Z + set +e 2024-11-01T16:49:18.3513762Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-11-01T16:49:18.3793556Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-11-01T16:49:18.3794086Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:18.3794548Z + '[' 0 -ne 0 ']' 2024-11-01T16:49:18.3794953Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-11-01T16:49:18.3795380Z + HAS_NVIDIA_DRIVER=1 2024-11-01T16:49:18.3796370Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-11-01T16:49:18.3797207Z + set -e 2024-11-01T16:49:18.3797518Z + '[' 1 -eq 0 ']' 2024-11-01T16:49:18.3798188Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-11-01T16:49:18.3799006Z + post_install_nvidia_driver_common 2024-11-01T16:49:18.3799466Z + sudo modprobe nvidia 2024-11-01T16:49:18.5356998Z + echo 'After installing NVIDIA driver' 2024-11-01T16:49:18.5357702Z + lspci 2024-11-01T16:49:18.5358009Z After installing NVIDIA driver 2024-11-01T16:49:18.5480910Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-11-01T16:49:18.5481825Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-11-01T16:49:18.5482948Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-11-01T16:49:18.5483909Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-11-01T16:49:18.5484873Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-11-01T16:49:18.5486010Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-11-01T16:49:18.5486897Z 00:1b.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.5487754Z 00:1c.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.5488591Z 00:1d.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.5489430Z 00:1e.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) 2024-11-01T16:49:18.5490424Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-11-01T16:49:18.5491105Z + lsmod 2024-11-01T16:49:18.5507681Z Module Size Used by 2024-11-01T16:49:18.5508307Z ib_core 454656 0 2024-11-01T16:49:18.5508728Z veth 36864 0 2024-11-01T16:49:18.5509139Z nvidia_modeset 1351680 0 2024-11-01T16:49:18.5509595Z video 65536 1 nvidia_modeset 2024-11-01T16:49:18.5510066Z wmi 36864 1 video 2024-11-01T16:49:18.5510493Z nvidia_uvm 4706304 0 2024-11-01T16:49:18.5510992Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-11-01T16:49:18.5511544Z drm 602112 1 nvidia 2024-11-01T16:49:18.5512033Z drm_panel_orientation_quirks 32768 1 drm 2024-11-01T16:49:18.5512628Z backlight 24576 3 video,drm,nvidia_modeset 2024-11-01T16:49:18.5514692Z i2c_core 106496 2 nvidia,drm 2024-11-01T16:49:18.5515177Z xt_conntrack 16384 1 2024-11-01T16:49:18.5515623Z nft_chain_nat 16384 3 2024-11-01T16:49:18.5516175Z xt_MASQUERADE 20480 1 2024-11-01T16:49:18.5516689Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-11-01T16:49:18.5517253Z nf_conntrack_netlink 57344 0 2024-11-01T16:49:18.5517935Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-11-01T16:49:18.5518675Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-11-01T16:49:18.5519205Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-11-01T16:49:18.5519696Z xfrm_user 57344 1 2024-11-01T16:49:18.5520140Z xfrm_algo 16384 1 xfrm_user 2024-11-01T16:49:18.5520806Z xt_addrtype 16384 2 2024-11-01T16:49:18.5521218Z nft_compat 20480 4 2024-11-01T16:49:18.5521736Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-11-01T16:49:18.5522451Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-11-01T16:49:18.5523093Z br_netfilter 36864 0 2024-11-01T16:49:18.5523554Z bridge 323584 1 br_netfilter 2024-11-01T16:49:18.5524035Z stp 16384 1 bridge 2024-11-01T16:49:18.5524506Z llc 16384 2 bridge,stp 2024-11-01T16:49:18.5524977Z overlay 167936 0 2024-11-01T16:49:18.5525383Z tls 114688 0 2024-11-01T16:49:18.5525790Z nls_ascii 16384 1 2024-11-01T16:49:18.5526196Z nls_cp437 20480 1 2024-11-01T16:49:18.5526585Z vfat 24576 1 2024-11-01T16:49:18.5526993Z sunrpc 692224 1 2024-11-01T16:49:18.5527517Z fat 86016 1 vfat 2024-11-01T16:49:18.5527936Z ena 167936 0 2024-11-01T16:49:18.5528326Z i8042 45056 0 2024-11-01T16:49:18.5528711Z skx_edac_common 28672 0 2024-11-01T16:49:18.5529127Z serio 28672 3 i8042 2024-11-01T16:49:18.5529699Z ghash_clmulni_intel 16384 0 2024-11-01T16:49:18.5530111Z button 24576 0 2024-11-01T16:49:18.5530513Z sch_fq_codel 20480 33 2024-11-01T16:49:18.5530904Z dm_mod 188416 0 2024-11-01T16:49:18.5531299Z fuse 163840 1 2024-11-01T16:49:18.5531688Z loop 36864 0 2024-11-01T16:49:18.5532080Z configfs 57344 1 2024-11-01T16:49:18.5532483Z dax 45056 1 dm_mod 2024-11-01T16:49:18.5532902Z dmi_sysfs 20480 0 2024-11-01T16:49:18.5533303Z crc32_pclmul 16384 0 2024-11-01T16:49:18.5533707Z crc32c_intel 24576 0 2024-11-01T16:49:18.5534105Z efivarfs 24576 1 2024-11-01T16:49:18.5534497Z + modinfo nvidia 2024-11-01T16:49:18.5535327Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-11-01T16:49:18.5536122Z alias: char-major-195-* 2024-11-01T16:49:18.5536537Z version: 550.54.15 2024-11-01T16:49:18.5536915Z supported: external 2024-11-01T16:49:18.5537282Z license: NVIDIA 2024-11-01T16:49:18.5537690Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-11-01T16:49:18.5538225Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-11-01T16:49:18.5538824Z srcversion: 833721318DA517F0C2FEC97 2024-11-01T16:49:18.5539326Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-11-01T16:49:18.5539864Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-11-01T16:49:18.5540512Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-11-01T16:49:18.5541009Z depends: i2c-core,drm 2024-11-01T16:49:18.5541351Z retpoline: Y 2024-11-01T16:49:18.5541651Z name: nvidia 2024-11-01T16:49:18.5542293Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-11-01T16:49:18.5543040Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-11-01T16:49:18.5543713Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-11-01T16:49:18.5544338Z parm: NVreg_ResmanDebugLevel:int 2024-11-01T16:49:18.5544773Z parm: NVreg_RmLogonRC:int 2024-11-01T16:49:18.5545213Z parm: NVreg_ModifyDeviceFiles:int 2024-11-01T16:49:18.5545675Z parm: NVreg_DeviceFileUID:int 2024-11-01T16:49:18.5546111Z parm: NVreg_DeviceFileGID:int 2024-11-01T16:49:18.5546730Z parm: NVreg_DeviceFileMode:int 2024-11-01T16:49:18.5547289Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-11-01T16:49:18.5547910Z parm: NVreg_UsePageAttributeTable:int 2024-11-01T16:49:18.5548430Z parm: NVreg_EnablePCIeGen3:int 2024-11-01T16:49:18.5548892Z parm: NVreg_EnableMSI:int 2024-11-01T16:49:18.5549336Z parm: NVreg_TCEBypassMode:int 2024-11-01T16:49:18.5549886Z parm: NVreg_EnableStreamMemOPs:int 2024-11-01T16:49:18.5550474Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-11-01T16:49:18.5551121Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-11-01T16:49:18.5551746Z parm: NVreg_EnableS0ixPowerManagement:int 2024-11-01T16:49:18.5552426Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:18.5553091Z parm: NVreg_DynamicPowerManagement:int 2024-11-01T16:49:18.5553762Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-11-01T16:49:18.5554425Z parm: NVreg_EnableGpuFirmware:int 2024-11-01T16:49:18.5554961Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-11-01T16:49:18.5555546Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-11-01T16:49:18.5556413Z parm: NVreg_EnableUserNUMAManagement:int 2024-11-01T16:49:18.5556974Z parm: NVreg_MemoryPoolSize:int 2024-11-01T16:49:18.5557509Z parm: NVreg_KMallocHeapMaxSize:int 2024-11-01T16:49:18.5558066Z parm: NVreg_VMallocHeapMaxSize:int 2024-11-01T16:49:18.5558609Z parm: NVreg_IgnoreMMIOCheck:int 2024-11-01T16:49:18.5559117Z parm: NVreg_NvLinkDisable:int 2024-11-01T16:49:18.5559763Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-11-01T16:49:18.5560359Z parm: NVreg_RegisterPCIDriver:int 2024-11-01T16:49:18.5560909Z parm: NVreg_EnableResizableBar:int 2024-11-01T16:49:18.5561465Z parm: NVreg_EnableDbgBreakpoint:int 2024-11-01T16:49:18.5562044Z parm: NVreg_EnableNonblockingOpen:int 2024-11-01T16:49:18.5562606Z parm: NVreg_RegistryDwords:charp 2024-11-01T16:49:18.5563165Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-11-01T16:49:18.5563717Z parm: NVreg_RmMsg:charp 2024-11-01T16:49:18.5564184Z parm: NVreg_GpuBlacklist:charp 2024-11-01T16:49:18.5564721Z parm: NVreg_TemporaryFilePath:charp 2024-11-01T16:49:18.5565257Z parm: NVreg_ExcludedGpus:charp 2024-11-01T16:49:18.5565783Z parm: NVreg_DmaRemapPeerMmio:int 2024-11-01T16:49:18.5566319Z parm: NVreg_RmNvlinkBandwidth:charp 2024-11-01T16:49:18.5566865Z parm: NVreg_ImexChannelCount:int 2024-11-01T16:49:18.5567390Z parm: rm_firmware_active:charp 2024-11-01T16:49:18.5567841Z + set +e 2024-11-01T16:49:18.5568183Z + nvidia-smi 2024-11-01T16:49:18.6030308Z Fri Nov 1 16:49:18 2024 2024-11-01T16:49:18.6031081Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:18.6032058Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:49:18.6032987Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:18.6033933Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:49:18.6034978Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:49:18.6035890Z | | | MIG M. | 2024-11-01T16:49:18.6036679Z |=========================================+========================+======================| 2024-11-01T16:49:18.6207320Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:49:18.6209772Z | N/A 29C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:18.6210511Z | | | N/A | 2024-11-01T16:49:18.6211353Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:18.6377766Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:49:18.6378653Z | N/A 28C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:18.6379587Z | | | N/A | 2024-11-01T16:49:18.6380429Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:18.6557206Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:49:18.6558115Z | N/A 26C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:18.6558862Z | | | N/A | 2024-11-01T16:49:18.6559730Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:18.6721583Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:49:18.6722458Z | N/A 27C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:49:18.6723231Z | | | N/A | 2024-11-01T16:49:18.6724102Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:49:18.6742503Z 2024-11-01T16:49:18.6743627Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:18.6744380Z | Processes: | 2024-11-01T16:49:18.6745201Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:49:18.6746003Z | ID ID Usage | 2024-11-01T16:49:18.6746654Z |=========================================================================================| 2024-11-01T16:49:18.6771953Z | No running processes found | 2024-11-01T16:49:18.6772968Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:49:19.3249759Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-11-01T16:49:19.3437983Z Tesla T4 2024-11-01T16:49:19.3509776Z + NVIDIA_SMI_STATUS=0 2024-11-01T16:49:19.3510397Z + '[' 0 -eq 0 ']' 2024-11-01T16:49:19.3510848Z + echo 'INFO: Ignoring allowed status 0' 2024-11-01T16:49:19.3511322Z + set -e 2024-11-01T16:49:19.3511634Z INFO: Ignoring allowed status 0 2024-11-01T16:49:19.3517604Z == Installing nvidia container toolkit for amzn2023 == 2024-11-01T16:49:19.3521350Z + sudo yum install -y yum-utils 2024-11-01T16:49:19.8214184Z Last metadata expiration check: 0:31:15 ago on Fri Nov 1 16:18:04 2024. 2024-11-01T16:49:19.8473424Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-11-01T16:49:19.8865474Z Dependencies resolved. 2024-11-01T16:49:19.9045420Z Nothing to do. 2024-11-01T16:49:19.9045783Z Complete! 2024-11-01T16:49:20.0292436Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-11-01T16:49:20.0293720Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:20.0295285Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:20.3039238Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-11-01T16:49:20.3738619Z + sudo yum install -y nvidia-docker2 2024-11-01T16:49:20.9494524Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-11-01T16:49:20.9757682Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-11-01T16:49:21.0146001Z Dependencies resolved. 2024-11-01T16:49:21.0326339Z Nothing to do. 2024-11-01T16:49:21.0328104Z Complete! 2024-11-01T16:49:21.1935385Z + sudo systemctl restart docker 2024-11-01T16:51:00.6841257Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-11-01T16:51:00.7371248Z Fri Nov 1 16:51:00 2024 2024-11-01T16:51:00.7372114Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:00.7374357Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-11-01T16:51:00.7375499Z |-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:00.7376545Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-11-01T16:51:00.7377725Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-11-01T16:51:00.7378525Z | | | MIG M. | 2024-11-01T16:51:00.7379147Z |=========================================+========================+======================| 2024-11-01T16:51:00.7549317Z | 0 Tesla T4 On | 00000000:00:1B.0 Off | 0 | 2024-11-01T16:51:00.7550204Z | N/A 29C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:00.7550939Z | | | N/A | 2024-11-01T16:51:00.7552116Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:00.7715281Z | 1 Tesla T4 On | 00000000:00:1C.0 Off | 0 | 2024-11-01T16:51:00.7716222Z | N/A 28C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:00.7717142Z | | | N/A | 2024-11-01T16:51:00.7718021Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:00.7891575Z | 2 Tesla T4 On | 00000000:00:1D.0 Off | 0 | 2024-11-01T16:51:00.7892446Z | N/A 27C P8 9W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:00.7893205Z | | | N/A | 2024-11-01T16:51:00.7894087Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:00.8057605Z | 3 Tesla T4 On | 00000000:00:1E.0 Off | 0 | 2024-11-01T16:51:00.8058587Z | N/A 27C P8 10W / 70W | 0MiB / 15360MiB | 0% Default | 2024-11-01T16:51:00.8059335Z | | | N/A | 2024-11-01T16:51:00.8060192Z +-----------------------------------------+------------------------+----------------------+ 2024-11-01T16:51:00.8074605Z 2024-11-01T16:51:00.8075524Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:00.8076580Z | Processes: | 2024-11-01T16:51:00.8077431Z | GPU GI CI PID Type Process name GPU Memory | 2024-11-01T16:51:00.8078267Z | ID ID Usage | 2024-11-01T16:51:00.8078929Z |=========================================================================================| 2024-11-01T16:51:00.8103069Z | No running processes found | 2024-11-01T16:51:00.8104179Z +-----------------------------------------------------------------------------------------+ 2024-11-01T16:51:01.9255280Z Command completed after 1 attempt(s). 2024-11-01T16:51:01.9346443Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:51:01.9347253Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:51:01.9347961Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-11-01T16:51:01.9348647Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:01.9358099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:01.9358692Z env: 2024-11-01T16:51:01.9358996Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:01.9359509Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:01.9360049Z ##[endgroup] 2024-11-01T16:51:02.1957900Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:51:02.2136527Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-11-01T16:51:02.2142624Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-11-01T16:51:02.3414128Z Prepare all required actions 2024-11-01T16:51:02.3414670Z Getting action download info 2024-11-01T16:51:02.4628517Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-11-01T16:51:02.6644242Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-11-01T16:51:02.7877732Z ##[group]Run ./.github/actions/download-build-artifacts 2024-11-01T16:51:02.7878292Z with: 2024-11-01T16:51:02.7878638Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:02.7879121Z s3-bucket: gha-artifacts 2024-11-01T16:51:02.7879497Z env: 2024-11-01T16:51:02.7879793Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:02.7880300Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:02.7880851Z ##[endgroup] 2024-11-01T16:51:02.7919806Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:02.7920323Z with: 2024-11-01T16:51:02.7920712Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:02.7921211Z s3-bucket: gha-artifacts 2024-11-01T16:51:02.7921589Z region: us-east-1 2024-11-01T16:51:02.7921917Z env: 2024-11-01T16:51:02.7922211Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:02.7922713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:02.7923263Z ##[endgroup] 2024-11-01T16:51:03.2944918Z (node:119153) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:03.2945787Z 2024-11-01T16:51:03.2946071Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:03.2946893Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:03.2947932Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:03.3614876Z Found 1 objects with prefix pytorch/pytorch/11632514903/linux-focal-cuda11.8-py3.10-gcc9/ 2024-11-01T16:51:03.3616128Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.7989967Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:51:21.7995525Z Artifact download has finished successfully 2024-11-01T16:51:21.8229243Z ##[group]Run unzip -o artifacts.zip 2024-11-01T16:51:21.8229721Z unzip -o artifacts.zip 2024-11-01T16:51:21.8237995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:21.8238572Z env: 2024-11-01T16:51:21.8238876Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:21.8239389Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:21.8239940Z ##[endgroup] 2024-11-01T16:51:21.8282002Z Archive: artifacts.zip 2024-11-01T16:51:21.8283026Z creating: dist/ 2024-11-01T16:51:23.8486982Z inflating: dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:23.8487786Z creating: build/custom_test_artifacts/ 2024-11-01T16:51:23.8488565Z creating: build/custom_test_artifacts/custom-op-build/ 2024-11-01T16:51:23.8489408Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-11-01T16:51:23.8490390Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.8495342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.8496458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.8497580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.8498764Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.8499925Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.8501252Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.8502854Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.8504063Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.8505256Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.8506648Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.8508184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.8509574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.8510947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.8512349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.8513763Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.8515002Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.8516481Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.8559168Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.8561013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.8562846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.8564729Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.8566589Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.8568442Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.8570147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.8571860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.8573576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.8605570Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.8646767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.8648664Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.8650247Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.8651772Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.8653260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.8654767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.8656224Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.8657708Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.8717515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.8781793Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.8783269Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.8784473Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.8785913Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.8786994Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.8788109Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-11-01T16:51:23.8789343Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8790730Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8792077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-11-01T16:51:23.8793330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-11-01T16:51:23.8794625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.8796004Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-11-01T16:51:23.8797534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.8798865Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-11-01T16:51:23.8800172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-11-01T16:51:23.8810081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-11-01T16:51:23.8954907Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-11-01T16:51:23.8956245Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-11-01T16:51:23.8957766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-11-01T16:51:23.8959295Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-11-01T16:51:23.8960747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-11-01T16:51:23.8962096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-11-01T16:51:23.8963498Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-11-01T16:51:23.8964926Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-11-01T16:51:23.8966347Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-11-01T16:51:23.8967871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-11-01T16:51:23.8969228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-11-01T16:51:23.8980328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-11-01T16:51:23.9066251Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-11-01T16:51:23.9067743Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.9069049Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.9070413Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.9071526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.9072612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.9073690Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-11-01T16:51:23.9074696Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-11-01T16:51:23.9075991Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-11-01T16:51:23.9077133Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-11-01T16:51:23.9199000Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-11-01T16:51:23.9264087Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-11-01T16:51:23.9265144Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-11-01T16:51:23.9265977Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-11-01T16:51:23.9266942Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.9270780Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.9271867Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.9272976Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.9274146Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.9275293Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.9276899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.9278262Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.9279485Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.9280685Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.9282101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.9283528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.9284939Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.9286513Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.9288061Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.9289454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.9290688Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.9291867Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.9335536Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.9337302Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.9339066Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.9340861Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.9342624Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.9344502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.9346209Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.9347901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.9349657Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.9381785Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.9423586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.9425314Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.9426879Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.9428388Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.9429859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.9431358Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.9432784Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.9434252Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.9492300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:23.9555902Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:23.9557575Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:23.9558807Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:23.9559870Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:23.9560970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:23.9562115Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-11-01T16:51:23.9563432Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-11-01T16:51:23.9564911Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-11-01T16:51:23.9566342Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-11-01T16:51:23.9567671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-11-01T16:51:23.9569134Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-11-01T16:51:23.9570465Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-11-01T16:51:23.9571799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-11-01T16:51:23.9573129Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-11-01T16:51:23.9574446Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-11-01T16:51:23.9583203Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-11-01T16:51:23.9651888Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-11-01T16:51:23.9653376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:23.9654671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:23.9655828Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-11-01T16:51:23.9656989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-11-01T16:51:23.9658053Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:23.9659119Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-11-01T16:51:23.9660110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-11-01T16:51:23.9661007Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-11-01T16:51:23.9661952Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-11-01T16:51:23.9711078Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-11-01T16:51:23.9712016Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-11-01T16:51:23.9712920Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-11-01T16:51:23.9713976Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:51:23.9718768Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:51:23.9719991Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-11-01T16:51:23.9721232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:51:23.9722533Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:51:23.9723797Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:51:23.9725245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:51:23.9726707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:51:23.9728127Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:51:23.9729392Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:51:23.9730858Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:51:23.9732333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:51:23.9733777Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:51:23.9735226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:51:23.9736707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:51:23.9738190Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:51:23.9739516Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-11-01T16:51:23.9740794Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-11-01T16:51:23.9783842Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-11-01T16:51:23.9785888Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-11-01T16:51:23.9787938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-11-01T16:51:23.9789841Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-11-01T16:51:23.9791716Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-11-01T16:51:23.9793591Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-11-01T16:51:23.9795383Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-11-01T16:51:23.9797491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-11-01T16:51:23.9799338Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-11-01T16:51:23.9829678Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-11-01T16:51:23.9871156Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-11-01T16:51:23.9872976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-11-01T16:51:23.9874637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-11-01T16:51:23.9876526Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-11-01T16:51:23.9878151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-11-01T16:51:23.9879792Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-11-01T16:51:23.9881367Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-11-01T16:51:23.9882976Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-11-01T16:51:23.9940861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-11-01T16:51:24.0004924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-11-01T16:51:24.0006528Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-11-01T16:51:24.0007941Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:51:24.0009052Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:51:24.0010219Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-11-01T16:51:24.0011426Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-11-01T16:51:24.0012813Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:24.0014349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-11-01T16:51:24.0015833Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-11-01T16:51:24.0017209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-11-01T16:51:24.0018639Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:24.0020264Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-11-01T16:51:24.0021716Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:24.0023154Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-11-01T16:51:24.0024581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-11-01T16:51:24.0026194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-11-01T16:51:24.0143930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-11-01T16:51:24.0145383Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-11-01T16:51:24.0146828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-11-01T16:51:24.0148453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-11-01T16:51:24.0150008Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-11-01T16:51:24.0151474Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-11-01T16:51:24.0153011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-11-01T16:51:24.0154543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-11-01T16:51:24.0156339Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-11-01T16:51:24.0157897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-11-01T16:51:24.0159450Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-11-01T16:51:24.0169576Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-11-01T16:51:24.0228381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-11-01T16:51:24.0230007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:51:24.0231409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:51:24.0232662Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-11-01T16:51:24.0233819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-11-01T16:51:24.0234977Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-11-01T16:51:24.0236390Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-11-01T16:51:24.0237511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-11-01T16:51:24.0238535Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-11-01T16:51:24.0239589Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-11-01T16:51:24.0340395Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-11-01T16:51:24.0384800Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-11-01T16:51:24.0385741Z creating: build/lib/ 2024-11-01T16:51:24.0393667Z inflating: build/lib/libpthreadpool.a 2024-11-01T16:51:24.0394526Z inflating: build/lib/libclog.a 2024-11-01T16:51:24.0403474Z inflating: build/lib/libcpuinfo_internals.a 2024-11-01T16:51:24.0412163Z inflating: build/lib/libcpuinfo.a 2024-11-01T16:51:24.0414521Z inflating: build/lib/libnnpack_reference_layers.a 2024-11-01T16:51:24.0506473Z inflating: build/lib/libprotobuf-lite.a 2024-11-01T16:51:24.0573348Z inflating: build/lib/libgtest.a 2024-11-01T16:51:24.0652675Z inflating: build/lib/libbenchmark.a 2024-11-01T16:51:24.0659159Z inflating: build/lib/libittnotify.a 2024-11-01T16:51:24.1126728Z inflating: build/lib/libprotobuf.a 2024-11-01T16:51:24.1157241Z inflating: build/lib/libtensorpipe_uv.a 2024-11-01T16:51:24.1225504Z inflating: build/lib/libasmjit.a 2024-11-01T16:51:24.1357215Z inflating: build/lib/libgloo.a 2024-11-01T16:51:24.1379809Z inflating: build/lib/libfmt.a 2024-11-01T16:51:24.1380367Z inflating: build/lib/libcaffe2_nvrtc.so 2024-11-01T16:51:24.1484176Z inflating: build/lib/libc10.so 2024-11-01T16:51:24.1485288Z inflating: build/lib/libtorch_global_deps.so 2024-11-01T16:51:24.1507199Z inflating: build/lib/libpytorch_qnnpack.a 2024-11-01T16:51:24.1525466Z inflating: build/lib/libgmock.a 2024-11-01T16:51:24.2043304Z inflating: build/lib/libprotoc.a 2024-11-01T16:51:24.2043852Z inflating: build/lib/libgtest_main.a 2024-11-01T16:51:24.2044402Z inflating: build/lib/libbenchmark_main.a 2024-11-01T16:51:24.2064674Z inflating: build/lib/libnnpack.a 2024-11-01T16:51:25.2413536Z inflating: build/lib/libdnnl.a 2024-11-01T16:51:25.2994728Z inflating: build/lib/libtensorpipe.a 2024-11-01T16:51:25.3056240Z inflating: build/lib/libc10_cuda.so 2024-11-01T16:51:25.4318953Z inflating: build/lib/libfbgemm.a 2024-11-01T16:51:25.4319507Z inflating: build/lib/libgmock_main.a 2024-11-01T16:51:25.4842829Z inflating: build/lib/libkineto.a 2024-11-01T16:51:25.5104416Z inflating: build/lib/libtensorpipe_cuda.a 2024-11-01T16:51:25.5306205Z inflating: build/lib/libXNNPACK.a 2024-11-01T16:51:25.5691232Z inflating: build/lib/libgloo_cuda.a 2024-11-01T16:51:25.5734938Z inflating: build/lib/libonnx_proto.a 2024-11-01T16:51:25.6477281Z inflating: build/lib/libonnx.a 2024-11-01T16:51:28.1873241Z inflating: build/lib/libtorch_cpu.so 2024-11-01T16:51:28.1876441Z inflating: build/lib/libshm.so 2024-11-01T16:51:28.1881666Z inflating: build/lib/libunbox_lib.a 2024-11-01T16:51:29.8987352Z inflating: build/lib/libtorch_cuda.so 2024-11-01T16:51:29.8987973Z inflating: build/lib/libtorch.so 2024-11-01T16:51:29.8989664Z inflating: build/lib/libc10d_cuda_test.so 2024-11-01T16:51:30.8593662Z inflating: build/lib/libtorch_cuda_linalg.so 2024-11-01T16:51:30.8615328Z inflating: build/lib/libjitbackend_test.so 2024-11-01T16:51:30.8686098Z inflating: build/lib/libtorchbind_test.so 2024-11-01T16:51:30.8713477Z inflating: build/lib/libaoti_custom_ops.so 2024-11-01T16:51:30.8740019Z inflating: build/lib/libbackend_with_compiler.so 2024-11-01T16:51:31.0815796Z inflating: build/lib/libtorch_python.so 2024-11-01T16:51:31.0852743Z inflating: build/lib/libnnapi_backend.so 2024-11-01T16:51:31.0853304Z creating: build/bin/ 2024-11-01T16:51:31.0906642Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-11-01T16:51:31.0959401Z inflating: build/bin/c10_DeviceGuard_test 2024-11-01T16:51:31.1015911Z inflating: build/bin/c10_Device_test 2024-11-01T16:51:31.1076562Z inflating: build/bin/c10_DispatchKeySet_test 2024-11-01T16:51:31.1134117Z inflating: build/bin/c10_Scalar_test 2024-11-01T16:51:31.1185572Z inflating: build/bin/c10_StreamGuard_test 2024-11-01T16:51:31.1238892Z inflating: build/bin/c10_SymInt_test 2024-11-01T16:51:31.1296183Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-11-01T16:51:31.1355087Z inflating: build/bin/c10_InlineStreamGuard_test 2024-11-01T16:51:31.1415521Z inflating: build/bin/c10_SizesAndStrides_test 2024-11-01T16:51:31.1488348Z inflating: build/bin/c10_cow_test 2024-11-01T16:51:31.1544658Z inflating: build/bin/c10_Bitset_test 2024-11-01T16:51:31.1595494Z inflating: build/bin/c10_ConstexprCrc_test 2024-11-01T16:51:31.1648030Z inflating: build/bin/c10_DeadlockDetection_test 2024-11-01T16:51:31.1708166Z inflating: build/bin/c10_LeftRight_test 2024-11-01T16:51:31.1761076Z inflating: build/bin/c10_Half_test 2024-11-01T16:51:31.1821124Z inflating: build/bin/c10_Metaprogramming_test 2024-11-01T16:51:31.1875789Z inflating: build/bin/c10_NetworkFlow_test 2024-11-01T16:51:31.1930503Z inflating: build/bin/c10_Synchronized_test 2024-11-01T16:51:31.1987249Z inflating: build/bin/c10_ThreadLocal_test 2024-11-01T16:51:31.2041047Z inflating: build/bin/c10_TypeIndex_test 2024-11-01T16:51:31.2096269Z inflating: build/bin/c10_TypeList_test 2024-11-01T16:51:31.2148837Z inflating: build/bin/c10_TypeTraits_test 2024-11-01T16:51:31.2203935Z inflating: build/bin/c10_accumulate_test 2024-11-01T16:51:31.2263310Z inflating: build/bin/c10_bfloat16_test 2024-11-01T16:51:31.2316905Z inflating: build/bin/c10_bit_cast_test 2024-11-01T16:51:31.2375237Z inflating: build/bin/c10_complex_math_test 2024-11-01T16:51:31.2434370Z inflating: build/bin/c10_complex_test 2024-11-01T16:51:31.2489858Z inflating: build/bin/c10_exception_test 2024-11-01T16:51:31.2543810Z inflating: build/bin/c10_flags_test 2024-11-01T16:51:31.2595890Z inflating: build/bin/c10_generic_math_test 2024-11-01T16:51:31.2764818Z inflating: build/bin/c10_intrusive_ptr_test 2024-11-01T16:51:31.2820410Z inflating: build/bin/c10_irange_test 2024-11-01T16:51:31.2875363Z inflating: build/bin/c10_lazy_test 2024-11-01T16:51:31.2936417Z inflating: build/bin/c10_logging_test 2024-11-01T16:51:31.3016384Z inflating: build/bin/c10_optional_test 2024-11-01T16:51:31.3080205Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-11-01T16:51:31.3138185Z inflating: build/bin/c10_registry_test 2024-11-01T16:51:31.3294412Z inflating: build/bin/c10_small_vector_test 2024-11-01T16:51:31.3348777Z inflating: build/bin/c10_ssize_test 2024-11-01T16:51:31.3403325Z inflating: build/bin/c10_string_util_test 2024-11-01T16:51:31.3467659Z inflating: build/bin/c10_string_view_test 2024-11-01T16:51:31.3520361Z inflating: build/bin/c10_tempfile_test 2024-11-01T16:51:31.3581321Z inflating: build/bin/c10_typeid_test 2024-11-01T16:51:31.3633222Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-11-01T16:51:31.4100296Z inflating: build/bin/protoc-3.13.0.0 2024-11-01T16:51:31.4565518Z inflating: build/bin/protoc 2024-11-01T16:51:31.4623495Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-11-01T16:51:31.4678025Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-11-01T16:51:31.4735275Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-11-01T16:51:31.4790150Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-11-01T16:51:31.4844594Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-11-01T16:51:31.4896561Z inflating: build/bin/c10_cuda_CUDATest 2024-11-01T16:51:31.4952846Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-11-01T16:51:31.5010773Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-11-01T16:51:31.5397407Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-11-01T16:51:31.5797555Z inflating: build/bin/vec_test_all_types_AVX512 2024-11-01T16:51:31.6212421Z inflating: build/bin/vec_test_all_types_AVX2 2024-11-01T16:51:31.6271059Z inflating: build/bin/TCPStoreTest 2024-11-01T16:51:31.6326599Z inflating: build/bin/BackoffTest 2024-11-01T16:51:31.6383586Z inflating: build/bin/HashStoreTest 2024-11-01T16:51:31.6438125Z inflating: build/bin/FileStoreTest 2024-11-01T16:51:31.6454403Z inflating: build/bin/ProcessGroupMPITest 2024-11-01T16:51:31.6457461Z inflating: build/bin/torch_shm_manager 2024-11-01T16:51:31.6516103Z inflating: build/bin/test_edge_op_registration 2024-11-01T16:51:31.6517997Z inflating: build/bin/example_allreduce 2024-11-01T16:51:31.6573537Z inflating: build/bin/Dimname_test 2024-11-01T16:51:31.6651751Z inflating: build/bin/Dict_test 2024-11-01T16:51:31.6718305Z inflating: build/bin/MaybeOwned_test 2024-11-01T16:51:31.6782560Z inflating: build/bin/apply_utils_test 2024-11-01T16:51:31.6843029Z inflating: build/bin/NamedTensor_test 2024-11-01T16:51:31.6906219Z inflating: build/bin/atest 2024-11-01T16:51:31.6973103Z inflating: build/bin/basic 2024-11-01T16:51:31.7031418Z inflating: build/bin/broadcast_test 2024-11-01T16:51:31.7084019Z inflating: build/bin/cpu_allocator_test 2024-11-01T16:51:31.7145801Z inflating: build/bin/cpu_generator_test 2024-11-01T16:51:31.7202260Z inflating: build/bin/cpu_profiling_allocator_test 2024-11-01T16:51:31.7300435Z inflating: build/bin/cpu_rng_test 2024-11-01T16:51:31.7352977Z inflating: build/bin/dispatch_key_set_test 2024-11-01T16:51:31.7406074Z inflating: build/bin/dlconvertor_test 2024-11-01T16:51:31.7464933Z inflating: build/bin/half_test 2024-11-01T16:51:31.7525707Z inflating: build/bin/extension_backend_test 2024-11-01T16:51:31.7579240Z inflating: build/bin/lazy_tensor_test 2024-11-01T16:51:31.7677662Z inflating: build/bin/ivalue_test 2024-11-01T16:51:31.7736975Z inflating: build/bin/math_kernel_test 2024-11-01T16:51:31.7794388Z inflating: build/bin/memory_format_test 2024-11-01T16:51:31.7851181Z inflating: build/bin/memory_overlapping_test 2024-11-01T16:51:31.7907736Z inflating: build/bin/mobile_memory_cleanup 2024-11-01T16:51:31.7959204Z inflating: build/bin/operator_name_test 2024-11-01T16:51:31.8020237Z inflating: build/bin/native_test 2024-11-01T16:51:31.8073868Z inflating: build/bin/operators_test 2024-11-01T16:51:31.8130815Z inflating: build/bin/packedtensoraccessor_test 2024-11-01T16:51:31.8199656Z inflating: build/bin/pow_test 2024-11-01T16:51:31.8253486Z inflating: build/bin/reduce_ops_test 2024-11-01T16:51:31.8314513Z inflating: build/bin/quantized_test 2024-11-01T16:51:31.8367085Z inflating: build/bin/reportMemoryUsage_test 2024-11-01T16:51:31.8447494Z inflating: build/bin/scalar_tensor_test 2024-11-01T16:51:31.8483281Z inflating: build/bin/StorageUtils_test 2024-11-01T16:51:31.8545565Z inflating: build/bin/scalar_test 2024-11-01T16:51:31.8600700Z inflating: build/bin/stride_properties_test 2024-11-01T16:51:31.8681627Z inflating: build/bin/tensor_iterator_test 2024-11-01T16:51:31.8740933Z inflating: build/bin/test_parallel 2024-11-01T16:51:31.8742637Z inflating: build/bin/thread_init_test 2024-11-01T16:51:31.8803653Z inflating: build/bin/type_ptr_test 2024-11-01T16:51:31.8866596Z inflating: build/bin/type_test 2024-11-01T16:51:31.8921162Z inflating: build/bin/undefined_tensor_test 2024-11-01T16:51:31.8922295Z inflating: build/bin/verify_api_visibility 2024-11-01T16:51:31.8977524Z inflating: build/bin/weakref_test 2024-11-01T16:51:31.9051434Z inflating: build/bin/legacy_vmap_test 2024-11-01T16:51:31.9106350Z inflating: build/bin/wrapdim_test 2024-11-01T16:51:31.9170596Z inflating: build/bin/IListRef_test 2024-11-01T16:51:31.9278809Z inflating: build/bin/List_test 2024-11-01T16:51:31.9335188Z inflating: build/bin/xla_tensor_test 2024-11-01T16:51:31.9462035Z inflating: build/bin/kernel_function_legacy_test 2024-11-01T16:51:31.9561760Z inflating: build/bin/kernel_function_test 2024-11-01T16:51:31.9632675Z inflating: build/bin/KernelFunction_test 2024-11-01T16:51:31.9765721Z inflating: build/bin/kernel_lambda_legacy_test 2024-11-01T16:51:31.9831217Z inflating: build/bin/kernel_stackbased_test 2024-11-01T16:51:31.9939600Z inflating: build/bin/kernel_lambda_test 2024-11-01T16:51:32.0039999Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-11-01T16:51:32.0093606Z inflating: build/bin/CppSignature_test 2024-11-01T16:51:32.0152570Z inflating: build/bin/backend_fallback_test 2024-11-01T16:51:32.0203934Z inflating: build/bin/op_allowlist_test 2024-11-01T16:51:32.0270525Z inflating: build/bin/inline_container_test 2024-11-01T16:51:32.0579688Z inflating: build/bin/op_registration_test 2024-11-01T16:51:32.0635367Z inflating: build/bin/cuda_apply_test 2024-11-01T16:51:32.0689075Z inflating: build/bin/cuda_allocator_test 2024-11-01T16:51:32.0752228Z inflating: build/bin/cuda_atomic_ops_test 2024-11-01T16:51:32.0811669Z inflating: build/bin/cuda_caching_host_allocator_test 2024-11-01T16:51:32.0883183Z inflating: build/bin/cuda_complex_math_test 2024-11-01T16:51:32.0936596Z inflating: build/bin/cuda_device_test 2024-11-01T16:51:32.0998046Z inflating: build/bin/cuda_complex_test 2024-11-01T16:51:32.1059930Z inflating: build/bin/cuda_cub_test 2024-11-01T16:51:32.1114302Z inflating: build/bin/cuda_dlconvertor_test 2024-11-01T16:51:32.1166787Z inflating: build/bin/cuda_integer_divider_test 2024-11-01T16:51:32.1227899Z inflating: build/bin/cuda_generator_test 2024-11-01T16:51:32.1296004Z inflating: build/bin/cuda_distributions_test 2024-11-01T16:51:32.1349217Z inflating: build/bin/cuda_half_test 2024-11-01T16:51:32.1401304Z inflating: build/bin/cuda_optional_test 2024-11-01T16:51:32.1456556Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-11-01T16:51:32.1512545Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-11-01T16:51:32.1575528Z inflating: build/bin/cuda_stream_test 2024-11-01T16:51:32.1631275Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-11-01T16:51:32.1682388Z inflating: build/bin/cuda_cudnn_test 2024-11-01T16:51:32.1739465Z inflating: build/bin/cuda_vectorized_test 2024-11-01T16:51:32.1810166Z inflating: build/bin/ProcessGroupGlooTest 2024-11-01T16:51:32.1825699Z inflating: build/bin/tutorial_tensorexpr 2024-11-01T16:51:32.1893265Z inflating: build/bin/ProcessGroupNCCLTest 2024-11-01T16:51:32.1953842Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-11-01T16:51:32.2020127Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-11-01T16:51:32.2075438Z inflating: build/bin/ProcessGroupUCCTest 2024-11-01T16:51:32.2135775Z inflating: build/bin/test_dist_autograd 2024-11-01T16:51:32.2136872Z inflating: build/bin/parallel_benchmark 2024-11-01T16:51:32.2146330Z inflating: build/bin/aot_model_compiler_test 2024-11-01T16:51:32.2221489Z inflating: build/bin/test_cpp_rpc 2024-11-01T16:51:32.2291051Z inflating: build/bin/test_mobile_nnc 2024-11-01T16:51:32.2643988Z inflating: build/bin/test_lazy 2024-11-01T16:51:32.3500550Z inflating: build/bin/test_tensorexpr 2024-11-01T16:51:32.4717169Z inflating: build/bin/test_api 2024-11-01T16:51:32.5311479Z inflating: build/bin/test_jit 2024-11-01T16:51:32.5312259Z creating: .additional_ci_files/ 2024-11-01T16:51:32.5392943Z inflating: .additional_ci_files/test-times.json 2024-11-01T16:51:32.5713741Z inflating: .additional_ci_files/test-class-times.json 2024-11-01T16:51:32.5747641Z ##[group]Run rm artifacts.zip 2024-11-01T16:51:32.5748039Z rm artifacts.zip 2024-11-01T16:51:32.5754279Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.5754795Z env: 2024-11-01T16:51:32.5755064Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.5755522Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.5756303Z ##[endgroup] 2024-11-01T16:51:32.6352607Z ##[group]Run df -H 2024-11-01T16:51:32.6352962Z df -H 2024-11-01T16:51:32.6359209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:32.6359781Z env: 2024-11-01T16:51:32.6360081Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.6360578Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.6361138Z ##[endgroup] 2024-11-01T16:51:32.6402320Z Filesystem Size Used Avail Use% Mounted on 2024-11-01T16:51:32.6402936Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-11-01T16:51:32.6403478Z tmpfs 101G 0 101G 0% /dev/shm 2024-11-01T16:51:32.6404241Z tmpfs 41G 697k 41G 1% /run 2024-11-01T16:51:32.6404798Z /dev/nvme0n1p1 161G 47G 115G 30% / 2024-11-01T16:51:32.6405312Z tmpfs 101G 13k 101G 1% /tmp 2024-11-01T16:51:32.6405862Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-11-01T16:51:32.6406442Z tmpfs 21G 0 21G 0% /run/user/0 2024-11-01T16:51:32.6451408Z Prepare all required actions 2024-11-01T16:51:32.6452024Z Getting action download info 2024-11-01T16:51:32.7929548Z ##[group]Run ./.github/actions/download-td-artifacts 2024-11-01T16:51:32.7930060Z with: 2024-11-01T16:51:32.7930320Z env: 2024-11-01T16:51:32.7930585Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7931073Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7931597Z ##[endgroup] 2024-11-01T16:51:32.7968945Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:51:32.7969429Z with: 2024-11-01T16:51:32.7969701Z name: td_results 2024-11-01T16:51:32.7970034Z s3-bucket: gha-artifacts 2024-11-01T16:51:32.7970389Z region: us-east-1 2024-11-01T16:51:32.7970698Z env: 2024-11-01T16:51:32.7970973Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:32.7971454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:32.7971974Z ##[endgroup] 2024-11-01T16:51:33.2976603Z (node:119180) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:51:33.2977465Z 2024-11-01T16:51:33.2977758Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:51:33.2978585Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:51:33.2979594Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:51:33.3658576Z Found 1 objects with prefix pytorch/pytorch/11632514903/td_results/ 2024-11-01T16:51:33.3659832Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.4337753Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:51:33.4343215Z Artifact download has finished successfully 2024-11-01T16:51:33.4539730Z ##[group]Run mkdir -p .additional_ci_files 2024-11-01T16:51:33.4540253Z mkdir -p .additional_ci_files 2024-11-01T16:51:33.4540987Z mv td_results.json .additional_ci_files/td_results.json 2024-11-01T16:51:33.4548630Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.4549137Z env: 2024-11-01T16:51:33.4549409Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.4549866Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.4550361Z ##[endgroup] 2024-11-01T16:51:33.4656472Z ##[group]Run .github/scripts/parse_ref.py 2024-11-01T16:51:33.4656957Z .github/scripts/parse_ref.py 2024-11-01T16:51:33.4662462Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:33.4662809Z env: 2024-11-01T16:51:33.4663060Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.4663518Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.4664012Z ##[endgroup] 2024-11-01T16:51:33.4961595Z Prepare all required actions 2024-11-01T16:51:33.5008355Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-11-01T16:51:33.5008826Z with: 2024-11-01T16:51:33.5009398Z github-token: *** 2024-11-01T16:51:33.5009699Z env: 2024-11-01T16:51:33.5009962Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5010572Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5011104Z ##[endgroup] 2024-11-01T16:51:33.5030175Z ##[group]Run set -eux 2024-11-01T16:51:33.5030521Z set -eux 2024-11-01T16:51:33.5031164Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-11-01T16:51:33.5037731Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:33.5038289Z env: 2024-11-01T16:51:33.5038585Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:33.5039097Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:33.5039849Z GITHUB_TOKEN: *** 2024-11-01T16:51:33.5040190Z ##[endgroup] 2024-11-01T16:51:33.5066098Z + python3 .github/scripts/get_workflow_job_id.py 11632514903 i-0a5b4d9b9fef8d246 2024-11-01T16:51:35.5813056Z setting job-id=32397034505 2024-11-01T16:51:35.5814276Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.6056812Z Prepare all required actions 2024-11-01T16:51:35.6057376Z Getting action download info 2024-11-01T16:51:35.7433726Z ##[group]Run ./.github/actions/filter-test-configs 2024-11-01T16:51:35.7434223Z with: 2024-11-01T16:51:35.7434704Z github-token: *** 2024-11-01T16:51:35.7436771Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]} 2024-11-01T16:51:35.7439186Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:35.7440085Z env: 2024-11-01T16:51:35.7440383Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.7440897Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.7441452Z ##[endgroup] 2024-11-01T16:51:35.7488946Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:51:35.7489394Z with: 2024-11-01T16:51:35.7489668Z shell: bash 2024-11-01T16:51:35.7489983Z timeout_minutes: 10 2024-11-01T16:51:35.7490326Z max_attempts: 5 2024-11-01T16:51:35.7490644Z retry_wait_seconds: 30 2024-11-01T16:51:35.7491955Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:35.7493347Z polling_interval_seconds: 1 2024-11-01T16:51:35.7493749Z warning_on_retry: true 2024-11-01T16:51:35.7494121Z continue_on_error: false 2024-11-01T16:51:35.7494479Z env: 2024-11-01T16:51:35.7494752Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:35.7495234Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:35.7495953Z GITHUB_TOKEN: *** 2024-11-01T16:51:35.7496281Z ##[endgroup] 2024-11-01T16:51:35.8491899Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:51:36.1040562Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:51:36.1213867Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-11-01T16:51:36.1215875Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-11-01T16:51:36.1345987Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-11-01T16:51:36.1347996Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-11-01T16:51:36.1358008Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-11-01T16:51:36.1364500Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.8.30) 2024-11-01T16:51:36.8310086Z Command completed after 1 attempt(s). 2024-11-01T16:51:36.8372696Z ##[group]Run set -x 2024-11-01T16:51:36.8373043Z set -x 2024-11-01T16:51:36.8373340Z  2024-11-01T16:51:36.8373936Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:36.8374692Z # in runner workspace 2024-11-01T16:51:36.8375266Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-11-01T16:51:36.8381777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:36.8382294Z env: 2024-11-01T16:51:36.8382568Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:36.8383017Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:36.8383524Z ##[endgroup] 2024-11-01T16:51:36.8410761Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-11-01T16:51:36.8665707Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:36.8666417Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:51:36.8667013Z echo "Job name: ${JOB_NAME}" 2024-11-01T16:51:36.8667587Z  2024-11-01T16:51:36.8668177Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:51:36.8668931Z # in runner workspace 2024-11-01T16:51:36.8669560Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-11-01T16:51:36.8670264Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-11-01T16:51:36.8670752Z  --job-name "${JOB_NAME}" \ 2024-11-01T16:51:36.8672695Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" \ 2024-11-01T16:51:36.8674675Z  --selected-test-configs "" \ 2024-11-01T16:51:36.8675155Z  --pr-number "${PR_NUMBER}" \ 2024-11-01T16:51:36.8675706Z  --tag "${TAG}" \ 2024-11-01T16:51:36.8676299Z  --event-name "${EVENT_NAME}" \ 2024-11-01T16:51:36.8676803Z  --schedule "${SCHEDULE}" \ 2024-11-01T16:51:36.8677354Z  --branch "${HEAD_BRANCH}" 2024-11-01T16:51:36.8683554Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:36.8684111Z env: 2024-11-01T16:51:36.8684406Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:36.8684925Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:36.8685886Z GITHUB_TOKEN: *** 2024-11-01T16:51:36.8686686Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:36.8687585Z PR_NUMBER: 138766 2024-11-01T16:51:36.8688015Z TAG: 2024-11-01T16:51:36.8688307Z EVENT_NAME: pull_request 2024-11-01T16:51:36.8688680Z SCHEDULE: 2024-11-01T16:51:36.8688991Z HEAD_BRANCH: 2024-11-01T16:51:36.8689305Z ##[endgroup] 2024-11-01T16:51:36.8713894Z Workflow: pull 2024-11-01T16:51:36.8714920Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.1229943Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-11-01T16:51:37.2959995Z ##[group]Run echo "Filtered matrix:" 2024-11-01T16:51:37.2960494Z echo "Filtered matrix:" 2024-11-01T16:51:37.2962472Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.g4dn.12xlarge.nvidia.gpu"}]}" 2024-11-01T16:51:37.2964422Z  2024-11-01T16:51:37.2964713Z echo 2024-11-01T16:51:37.2965129Z echo "Is the current job unstable? False" 2024-11-01T16:51:37.2965641Z  2024-11-01T16:51:37.2965925Z echo 2024-11-01T16:51:37.2966329Z echo "Is keep-going label set? False" 2024-11-01T16:51:37.2966827Z  2024-11-01T16:51:37.2967109Z echo 2024-11-01T16:51:37.2967431Z echo "Renabled issues? " 2024-11-01T16:51:37.2973472Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.2973982Z env: 2024-11-01T16:51:37.2974252Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.2974707Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.2975203Z ##[endgroup] 2024-11-01T16:51:37.2998704Z Filtered matrix: 2024-11-01T16:51:37.3000612Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.g4dn.12xlarge.nvidia.gpu}]} 2024-11-01T16:51:37.3002360Z 2024-11-01T16:51:37.3002725Z Is the current job unstable? False 2024-11-01T16:51:37.3003049Z 2024-11-01T16:51:37.3003472Z Is keep-going label set? False 2024-11-01T16:51:37.3003777Z 2024-11-01T16:51:37.3003902Z Renabled issues? 2024-11-01T16:51:37.3049296Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.3050034Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:51:37.3055615Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:51:37.3056128Z env: 2024-11-01T16:51:37.3056393Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.3056836Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.3057331Z JOB_TIMEOUT: 360 2024-11-01T16:51:37.3057633Z ##[endgroup] 2024-11-01T16:51:37.3172117Z ##[group]Run # Fetch aws credential from IMDs 2024-11-01T16:51:37.3172737Z # Fetch aws credential from IMDs 2024-11-01T16:51:37.3173381Z eval "$(python3 .github/scripts/get_aws_session_tokens.py)" 2024-11-01T16:51:37.3173970Z set -x 2024-11-01T16:51:37.3174268Z  2024-11-01T16:51:37.3174626Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-11-01T16:51:37.3175209Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-11-01T16:51:37.3175815Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-11-01T16:51:37.3176358Z  TEST_COMMAND=.ci/onnx/test.sh 2024-11-01T16:51:37.3176791Z else 2024-11-01T16:51:37.3177142Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.3177574Z fi 2024-11-01T16:51:37.3177972Z  2024-11-01T16:51:37.3178443Z # detached container should get cleaned up by teardown_ec2_linux 2024-11-01T16:51:37.3179283Z # TODO: Stop building test binaries as part of the build phase 2024-11-01T16:51:37.3179973Z # Used for GPU_FLAG since that doesn't play nice 2024-11-01T16:51:37.3180531Z # shellcheck disable=SC2086,SC2090 2024-11-01T16:51:37.3180998Z container_name=$(docker run \ 2024-11-01T16:51:37.3181417Z  ${GPU_FLAG:-} \ 2024-11-01T16:51:37.3181838Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-11-01T16:51:37.3182319Z  -e BUILD_ENVIRONMENT \ 2024-11-01T16:51:37.3182703Z  -e PR_NUMBER \ 2024-11-01T16:51:37.3183061Z  -e GITHUB_ACTIONS \ 2024-11-01T16:51:37.3183449Z  -e GITHUB_REPOSITORY \ 2024-11-01T16:51:37.3183851Z  -e GITHUB_WORKFLOW \ 2024-11-01T16:51:37.3184234Z  -e GITHUB_JOB \ 2024-11-01T16:51:37.3184577Z  -e GITHUB_RUN_ID \ 2024-11-01T16:51:37.3184959Z  -e GITHUB_RUN_NUMBER \ 2024-11-01T16:51:37.3185731Z  -e GITHUB_RUN_ATTEMPT \ 2024-11-01T16:51:37.3186196Z  -e JOB_ID \ 2024-11-01T16:51:37.3186555Z  -e JOB_NAME \ 2024-11-01T16:51:37.3186925Z  -e BASE_SHA \ 2024-11-01T16:51:37.3187274Z  -e BRANCH \ 2024-11-01T16:51:37.3187626Z  -e SHA1 \ 2024-11-01T16:51:37.3187988Z  -e AWS_DEFAULT_REGION \ 2024-11-01T16:51:37.3188437Z  -e AWS_ACCESS_KEY_ID \ 2024-11-01T16:51:37.3188887Z  -e AWS_SECRET_ACCESS_KEY \ 2024-11-01T16:51:37.3189329Z  -e AWS_SESSION_TOKEN \ 2024-11-01T16:51:37.3189757Z  -e IN_WHEEL_TEST \ 2024-11-01T16:51:37.3190167Z  -e SHARD_NUMBER \ 2024-11-01T16:51:37.3190566Z  -e TEST_CONFIG \ 2024-11-01T16:51:37.3190961Z  -e NUM_TEST_SHARDS \ 2024-11-01T16:51:37.3191388Z  -e REENABLED_ISSUES \ 2024-11-01T16:51:37.3191821Z  -e CONTINUE_THROUGH_ERROR \ 2024-11-01T16:51:37.3192283Z  -e VERBOSE_TEST_LOGS \ 2024-11-01T16:51:37.3192716Z  -e TEST_SHOWLOCALS \ 2024-11-01T16:51:37.3193141Z  -e NO_TEST_TIMEOUT \ 2024-11-01T16:51:37.3193540Z  -e NO_TD \ 2024-11-01T16:51:37.3193887Z  -e TD_DISTRIBUTED \ 2024-11-01T16:51:37.3194295Z  -e PR_LABELS \ 2024-11-01T16:51:37.3194732Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-11-01T16:51:37.3195231Z  -e SCCACHE_BUCKET \ 2024-11-01T16:51:37.3195893Z  -e SCCACHE_REGION \ 2024-11-01T16:51:37.3196484Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-11-01T16:51:37.3196947Z  -e XLA_CUDA \ 2024-11-01T16:51:37.3197393Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-11-01T16:51:37.3197969Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-11-01T16:51:37.3198571Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-11-01T16:51:37.3199162Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-11-01T16:51:37.3199679Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-11-01T16:51:37.3200191Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-11-01T16:51:37.3200820Z  -e DASHBOARD_TAG \ 2024-11-01T16:51:37.3201253Z  -e IS_A100_RUNNER \ 2024-11-01T16:51:37.3201697Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-11-01T16:51:37.3202281Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-11-01T16:51:37.3202905Z  --security-opt seccomp=unconfined \ 2024-11-01T16:51:37.3203440Z  --cap-add=SYS_PTRACE \ 2024-11-01T16:51:37.3203876Z  --ipc=host \ 2024-11-01T16:51:37.3204269Z  --shm-size="${SHM_SIZE}" \ 2024-11-01T16:51:37.3204710Z  --tty \ 2024-11-01T16:51:37.3205037Z  --detach \ 2024-11-01T16:51:37.3205429Z  --name="${container_name}" \ 2024-11-01T16:51:37.3205898Z  --user jenkins \ 2024-11-01T16:51:37.3206444Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-11-01T16:51:37.3207087Z  -w /var/lib/jenkins/workspace \ 2024-11-01T16:51:37.3207678Z  "${DOCKER_IMAGE}" 2024-11-01T16:51:37.3208151Z ) 2024-11-01T16:51:37.3208557Z # Propagate download.pytorch.org IP to container 2024-11-01T16:51:37.3209532Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-11-01T16:51:37.3210567Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-11-01T16:51:37.3211545Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-11-01T16:51:37.3217445Z shell: /usr/bin/bash -e {0} 2024-11-01T16:51:37.3217798Z env: 2024-11-01T16:51:37.3218070Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:51:37.3218526Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:51:37.3219131Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:51:37.3219629Z PR_NUMBER: 138766 2024-11-01T16:51:37.3219953Z GITHUB_REPOSITORY: pytorch/pytorch 2024-11-01T16:51:37.3220364Z GITHUB_WORKFLOW: pull 2024-11-01T16:51:37.3220700Z GITHUB_JOB: test 2024-11-01T16:51:37.3221003Z GITHUB_RUN_ID: 11632514903 2024-11-01T16:51:37.3221361Z GITHUB_RUN_NUMBER: 261361 2024-11-01T16:51:37.3221698Z GITHUB_RUN_ATTEMPT: 1 2024-11-01T16:51:37.3222019Z JOB_ID: 32397034505 2024-11-01T16:51:37.3222731Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:51:37.3223537Z BRANCH: pull/138766 2024-11-01T16:51:37.3223925Z SHA1: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:51:37.3224469Z BASE_SHA: 2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:51:37.3224935Z TEST_CONFIG: distributed 2024-11-01T16:51:37.3225274Z SHARD_NUMBER: 1 2024-11-01T16:51:37.3225573Z NUM_TEST_SHARDS: 3 2024-11-01T16:51:37.3225888Z REENABLED_ISSUES: 2024-11-01T16:51:37.3226220Z CONTINUE_THROUGH_ERROR: False 2024-11-01T16:51:37.3226592Z VERBOSE_TEST_LOGS: False 2024-11-01T16:51:37.3226944Z TEST_SHOWLOCALS: False 2024-11-01T16:51:37.3227282Z NO_TEST_TIMEOUT: False 2024-11-01T16:51:37.3227614Z NO_TD: False 2024-11-01T16:51:37.3227903Z TD_DISTRIBUTED: False 2024-11-01T16:51:37.3228307Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-11-01T16:51:37.3228802Z SCCACHE_REGION: us-east-1 2024-11-01T16:51:37.3229164Z SCCACHE_S3_KEY_PREFIX: pull 2024-11-01T16:51:37.3229521Z SHM_SIZE: 2g 2024-11-01T16:51:37.3230536Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:37.3231742Z XLA_CUDA: 2024-11-01T16:51:37.3232221Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:51:37.3232867Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-11-01T16:51:37.3233309Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-11-01T16:51:37.3233723Z DASHBOARD_TAG: 2024-11-01T16:51:37.3234246Z HUGGING_FACE_HUB_TOKEN: *** 2024-11-01T16:51:37.3234805Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-11-01T16:51:37.3235193Z IS_A100_RUNNER: 0 2024-11-01T16:51:37.3236179Z ARTIFACTS_FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T16:51:37.3237000Z ##[endgroup] 2024-11-01T16:51:37.3450117Z Traceback (most recent call last): 2024-11-01T16:51:37.3451980Z File "/home/ec2-user/actions-runner/_work/pytorch/pytorch/.github/scripts/get_aws_session_tokens.py", line 2, in 2024-11-01T16:51:37.3453060Z import boto3 # type: ignore[import] 2024-11-01T16:51:37.3453670Z ModuleNotFoundError: No module named 'boto3' 2024-11-01T16:51:37.3477391Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-11-01T16:51:37.3478133Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-11-01T16:51:37.3478706Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:51:37.3486120Z +++ nproc --ignore=2 2024-11-01T16:51:37.3504255Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_11632514903 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:51:48.4408494Z + container_name=f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T16:51:48.4411464Z + grep download.pytorch.org /etc/hosts 2024-11-01T16:51:48.4413680Z + docker exec -i f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d sudo bash -c '/bin/cat >> /etc/hosts' 2024-11-01T16:51:48.5509609Z + echo DOCKER_CONTAINER_ID=f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T16:51:48.5514935Z ++ echo dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:51:48.5517128Z + docker exec -t f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d sh -c 'pip install dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-11-01T16:51:48.9440534Z Processing ./dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+gitd1aa4ef) 2024-11-01T16:51:49.8227887Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.16.1) 2024-11-01T16:51:49.8230320Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (4.12.2) 2024-11-01T16:51:49.8232963Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2.8.8) 2024-11-01T16:51:49.8235147Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.1.4) 2024-11-01T16:51:49.8239103Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (2024.10.0) 2024-11-01T16:51:49.8245156Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.13.1) 2024-11-01T16:51:49.8267242Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.3.0) 2024-11-01T16:51:49.8278975Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.3.0) 2024-11-01T16:51:49.8302062Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (1.22.4) 2024-11-01T16:51:49.8703000Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+gitd1aa4ef->torch==2.6.0a0+gitd1aa4ef) (3.0.2) 2024-11-01T16:51:50.2026980Z Installing collected packages: torch 2024-11-01T16:52:01.2279758Z Successfully installed torch-2.6.0a0+gitd1aa4ef 2024-11-01T16:52:01.3002571Z + export TERM=vt100 2024-11-01T16:52:01.3002974Z + TERM=vt100 2024-11-01T16:52:01.3003977Z ++ dirname .ci/pytorch/test.sh 2024-11-01T16:52:01.3012885Z + source .ci/pytorch/common.sh 2024-11-01T16:52:01.3016693Z +++ dirname .ci/pytorch/common.sh 2024-11-01T16:52:01.3024499Z ++ source .ci/pytorch/common_utils.sh 2024-11-01T16:52:01.3026335Z +++ declare -f -t trap_add 2024-11-01T16:52:01.3032380Z ++ set -ex 2024-11-01T16:52:01.3033228Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:01.3033799Z ++ BUILD_TEST_LIBTORCH=0 2024-11-01T16:52:01.3034344Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T16:52:01.3037778Z ++ stat -c %u /var/lib/jenkins/workspace 2024-11-01T16:52:01.3057141Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-11-01T16:52:01.3057652Z + trap_add cleanup_workspace EXIT 2024-11-01T16:52:01.3058103Z + trap_add_cmd=cleanup_workspace 2024-11-01T16:52:01.3059482Z + shift 2024-11-01T16:52:01.3059799Z + for trap_add_name in "$@" 2024-11-01T16:52:01.3062490Z +++ trap -p EXIT 2024-11-01T16:52:01.3064743Z ++ eval 'extract_trap_cmd ' 2024-11-01T16:52:01.3065144Z +++ extract_trap_cmd 2024-11-01T16:52:01.3065522Z +++ printf '%s\n' '' 2024-11-01T16:52:01.3065942Z ++ printf '%s\n' cleanup_workspace 2024-11-01T16:52:01.3067004Z + trap -- ' 2024-11-01T16:52:01.3067368Z cleanup_workspace' EXIT 2024-11-01T16:52:01.3067889Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-11-01T16:52:01.8137085Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-11-01T16:52:01.8156711Z + echo 'Environment variables:' 2024-11-01T16:52:01.8157205Z Environment variables: 2024-11-01T16:52:01.8157562Z + env 2024-11-01T16:52:01.8167017Z INSTALLED_DB=yes 2024-11-01T16:52:01.8167655Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:01.8168083Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:01.8168504Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:01.8169207Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:01.8169857Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:01.8172343Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:01.8173158Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:01.8173857Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:01.8174641Z HOSTNAME=f17f4030e8f4 2024-11-01T16:52:01.8175683Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8176684Z GITHUB_ACTION=__self 2024-11-01T16:52:01.8177068Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:01.8179843Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:01.8182769Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:01.8183350Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:01.8183736Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:01.8184113Z TEST_CONFIG=distributed 2024-11-01T16:52:01.8184486Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:01.8185026Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:01.8185616Z IS_A100_RUNNER=0 2024-11-01T16:52:01.8186228Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:01.8186731Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:01.8187563Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:01.8188030Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:01.8188437Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:01.8188935Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:01.8189480Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:01.8189857Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:01.8190278Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:01.8190716Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:01.8191336Z XLA_CUDA= 2024-11-01T16:52:01.8191814Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:01.8195313Z *** 2024-11-01T16:52:01.8195713Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:01.8196324Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:01.8196730Z GITHUB_ACTIONS=true 2024-11-01T16:52:01.8197098Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:01.8197703Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:01.8198327Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:01.8198875Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:01.8199479Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:01.8200019Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:01.8200857Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:01.8201714Z UCC_HOME=/usr 2024-11-01T16:52:01.8202079Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:01.8202519Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:01.8202871Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:01.8203384Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:01.8203909Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:01.8204362Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:01.8204772Z SHARD_NUMBER=1 2024-11-01T16:52:01.8205084Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:01.8205461Z HOME=/var/lib/jenkins 2024-11-01T16:52:01.8205849Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:01.8206333Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:01.8206856Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:01.8207379Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:01.8207752Z CUDA_VERSION=11.8.0 2024-11-01T16:52:01.8208323Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:01.8208793Z NUM_TEST_SHARDS=3 2024-11-01T16:52:01.8209099Z UCX_HOME=/usr 2024-11-01T16:52:01.8209658Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:01.8210918Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8212380Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:01.8213841Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8215337Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:01.8216080Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:01.8216466Z DASHBOARD_TAG= 2024-11-01T16:52:01.8216768Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:01.8217268Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:01.8217857Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:01.8219121Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8220116Z GITHUB_ACTOR=c00w 2024-11-01T16:52:01.8220467Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:01.8220869Z PR_NUMBER=138766 2024-11-01T16:52:01.8221353Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:01.8221886Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:01.8222375Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:01.8222877Z TERM=vt100 2024-11-01T16:52:01.8223236Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:01.8223685Z INSTALLED_VISION=yes 2024-11-01T16:52:01.8224024Z BRANCH=pull/138766 2024-11-01T16:52:01.8224404Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:01.8224771Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:01.8225205Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:01.8225671Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:01.8226549Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:01.8227530Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:01.8228061Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:01.8228556Z REENABLED_ISSUES= 2024-11-01T16:52:01.8228859Z SHLVL=1 2024-11-01T16:52:01.8229119Z MAX_JOBS=46 2024-11-01T16:52:01.8229440Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:01.8229807Z NVARCH=x86_64 2024-11-01T16:52:01.8230092Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:01.8230568Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:01.8231120Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:01.8231584Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:01.8232329Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:01.8232946Z GITHUB_JOB=test 2024-11-01T16:52:01.8233367Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:01.8233972Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:01.8234529Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:01.8234864Z TD_DISTRIBUTED=False 2024-11-01T16:52:01.8235248Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:01.8235786Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:01.8236424Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:01.8236827Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:01.8237214Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:01.8237586Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:01.8238846Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:01.8240172Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:01.8241051Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T16:52:01.8241881Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:01.8242295Z CI=true 2024-11-01T16:52:01.8242626Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:01.8243093Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:01.8243512Z JOB_ID=32397034505 2024-11-01T16:52:01.8243857Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:01.8244241Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:01.8244637Z GITHUB_ACTION_REF= 2024-11-01T16:52:01.8245139Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:01.8245679Z TEST_SHOWLOCALS=False 2024-11-01T16:52:01.8246047Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:01.8246424Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:01.8247650Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8248813Z NO_TD=False 2024-11-01T16:52:01.8249123Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:01.8249506Z _=/usr/bin/env 2024-11-01T16:52:01.8250026Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-11-01T16:52:01.8418015Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-11-01T16:52:01.8419279Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T16:52:01.8420514Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-11-01T16:52:01.8421699Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-11-01T16:52:01.8422403Z + BUILD_DIR=build 2024-11-01T16:52:01.8422766Z + BUILD_RENAMED_DIR=build_renamed 2024-11-01T16:52:01.8423351Z + BUILD_BIN_DIR=build/bin 2024-11-01T16:52:01.8423732Z + SHARD_NUMBER=1 2024-11-01T16:52:01.8424067Z + NUM_TEST_SHARDS=3 2024-11-01T16:52:01.8424401Z + export VALGRIND=ON 2024-11-01T16:52:01.8424744Z + VALGRIND=ON 2024-11-01T16:52:01.8425245Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-11-01T16:52:01.8425780Z + [[ 0 == \1 ]] 2024-11-01T16:52:01.8426096Z + [[ False == \1 ]] 2024-11-01T16:52:01.8426585Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-11-01T16:52:01.8427169Z ++ realpath build/custom_test_artifacts 2024-11-01T16:52:01.8433077Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-11-01T16:52:01.8433869Z + [[ -n '' ]] 2024-11-01T16:52:01.8434248Z + echo 'Environment variables' 2024-11-01T16:52:01.8434666Z Environment variables 2024-11-01T16:52:01.8434987Z + env 2024-11-01T16:52:01.8442371Z INSTALLED_DB=yes 2024-11-01T16:52:01.8443044Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-11-01T16:52:01.8443765Z NVIDIA_VISIBLE_DEVICES=all 2024-11-01T16:52:01.8444519Z NV_NVML_DEV_VERSION=11.8.86-1 2024-11-01T16:52:01.8445224Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T16:52:01.8445941Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:52:01.8446541Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-11-01T16:52:01.8447136Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:01.8447770Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-11-01T16:52:01.8448313Z HOSTNAME=f17f4030e8f4 2024-11-01T16:52:01.8449450Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8450816Z GITHUB_ACTION=__self 2024-11-01T16:52:01.8451302Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:52:01.8454418Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-11-01T16:52:01.8457413Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-11-01T16:52:01.8457992Z NV_NVTX_VERSION=11.8.86-1 2024-11-01T16:52:01.8458368Z GITHUB_RUN_NUMBER=261361 2024-11-01T16:52:01.8458737Z TEST_CONFIG=distributed 2024-11-01T16:52:01.8459118Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:52:01.8459626Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-11-01T16:52:01.8460074Z IS_A100_RUNNER=0 2024-11-01T16:52:01.8460442Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-11-01T16:52:01.8460924Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-11-01T16:52:01.8461700Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-11-01T16:52:01.8462137Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-11-01T16:52:01.8462542Z GITHUB_TRIGGERING_ACTOR=c00w 2024-11-01T16:52:01.8463019Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-11-01T16:52:01.8463539Z GITHUB_REF_TYPE=branch 2024-11-01T16:52:01.8463904Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-11-01T16:52:01.8464309Z NCCL_VERSION=2.16.2-1 2024-11-01T16:52:01.8464882Z BASE_SHA=2055d3c8dc8b93c52eda6b9c07d716448d9e28c9 2024-11-01T16:52:01.8465375Z XLA_CUDA= 2024-11-01T16:52:01.8465889Z HUGGING_FACE_HUB_TOKEN=*** 2024-11-01T16:52:01.8466310Z *** 2024-11-01T16:52:01.8466601Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-11-01T16:52:01.8467034Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:52:01.8467415Z GITHUB_ACTIONS=true 2024-11-01T16:52:01.8467770Z NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T16:52:01.8468320Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-11-01T16:52:01.8468916Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-11-01T16:52:01.8469446Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:52:01.8470085Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-11-01T16:52:01.8470619Z GITHUB_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:01.8471431Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/138766/merge 2024-11-01T16:52:01.8472162Z UCC_HOME=/usr 2024-11-01T16:52:01.8472522Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-11-01T16:52:01.8472958Z VERBOSE_TEST_LOGS=False 2024-11-01T16:52:01.8473330Z NVIDIA_PRODUCT_NAME=CUDA 2024-11-01T16:52:01.8473842Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-11-01T16:52:01.8474369Z GITHUB_REF=refs/pull/138766/merge 2024-11-01T16:52:01.8474831Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-11-01T16:52:01.8475242Z SHARD_NUMBER=1 2024-11-01T16:52:01.8475557Z GITHUB_REF_PROTECTED=false 2024-11-01T16:52:01.8476206Z HOME=/var/lib/jenkins 2024-11-01T16:52:01.8476612Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:52:01.8477109Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:52:01.8477659Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-11-01T16:52:01.8478196Z SCCACHE_S3_KEY_PREFIX=pull 2024-11-01T16:52:01.8478583Z CUDA_VERSION=11.8.0 2024-11-01T16:52:01.8479074Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-11-01T16:52:01.8479576Z NUM_TEST_SHARDS=3 2024-11-01T16:52:01.8479905Z UCX_HOME=/usr 2024-11-01T16:52:01.8480532Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-11-01T16:52:01.8481859Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8483412Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.g4dn.12xlarge.nvidia.gpu) 2024-11-01T16:52:01.8484961Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8486581Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:52:01.8487373Z GITHUB_EVENT_NAME=pull_request 2024-11-01T16:52:01.8487785Z DASHBOARD_TAG= 2024-11-01T16:52:01.8488113Z GITHUB_RUN_ID=11632514903 2024-11-01T16:52:01.8488623Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-11-01T16:52:01.8489248Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-11-01T16:52:01.8490489Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8491548Z GITHUB_ACTOR=c00w 2024-11-01T16:52:01.8491927Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-11-01T16:52:01.8492334Z PR_NUMBER=138766 2024-11-01T16:52:01.8492668Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:52:01.8493012Z VALGRIND=ON 2024-11-01T16:52:01.8493336Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:52:01.8493836Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:52:01.8494345Z TERM=vt100 2024-11-01T16:52:01.8494704Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-11-01T16:52:01.8495165Z INSTALLED_VISION=yes 2024-11-01T16:52:01.8495517Z BRANCH=pull/138766 2024-11-01T16:52:01.8495911Z SCCACHE_REGION=us-east-1 2024-11-01T16:52:01.8496290Z OPENSSL_ROOT_DIR=/opt/openssl 2024-11-01T16:52:01.8496851Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-11-01T16:52:01.8497315Z CUDA_PATH=/usr/local/cuda 2024-11-01T16:52:01.8498189Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-11-01T16:52:01.8499203Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:52:01.8499743Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-11-01T16:52:01.8500249Z REENABLED_ISSUES= 2024-11-01T16:52:01.8500567Z SHLVL=1 2024-11-01T16:52:01.8500836Z MAX_JOBS=46 2024-11-01T16:52:01.8501165Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-11-01T16:52:01.8501545Z NVARCH=x86_64 2024-11-01T16:52:01.8501838Z GITHUB_ACTOR_ID=486199 2024-11-01T16:52:01.8502322Z GITHUB_WORKFLOW_SHA=368da7510f5d755034681777a4e5ae6d33c07b38 2024-11-01T16:52:01.8502889Z GITHUB_REF_NAME=138766/merge 2024-11-01T16:52:01.8503364Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-11-01T16:52:01.8504213Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:52:01.8504855Z GITHUB_JOB=test 2024-11-01T16:52:01.8505296Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-11-01T16:52:01.8505923Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-11-01T16:52:01.8506497Z NO_TEST_TIMEOUT=False 2024-11-01T16:52:01.8506848Z TD_DISTRIBUTED=False 2024-11-01T16:52:01.8507248Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-11-01T16:52:01.8507726Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:52:01.8508185Z NV_NVPROF_VERSION=11.8.87-1 2024-11-01T16:52:01.8508573Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:52:01.8508947Z OPENSSL_DIR=/opt/openssl 2024-11-01T16:52:01.8509308Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:52:01.8510523Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:01.8511793Z GITHUB_BASE_REF=gh/c00w/2/base 2024-11-01T16:52:01.8512645Z ARTIFACTS_FILE_SUFFIX=test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T16:52:01.8513452Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-11-01T16:52:01.8513847Z CI=true 2024-11-01T16:52:01.8514163Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-11-01T16:52:01.8514615Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:52:01.8515021Z JOB_ID=32397034505 2024-11-01T16:52:01.8515356Z INSTALLED_PROTOBUF=yes 2024-11-01T16:52:01.8515826Z GITHUB_HEAD_REF=gh/c00w/2/head 2024-11-01T16:52:01.8516387Z GITHUB_ACTION_REF= 2024-11-01T16:52:01.8516889Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:52:01.8517432Z TEST_SHOWLOCALS=False 2024-11-01T16:52:01.8517799Z GITHUB_WORKFLOW=pull 2024-11-01T16:52:01.8518176Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:52:01.8519288Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a8fd6fb5-7c88-4e3b-9d4b-eeb2c1259d33 2024-11-01T16:52:01.8520291Z NO_TD=False 2024-11-01T16:52:01.8520624Z SKIP_SCCACHE_INITIALIZATION=1 2024-11-01T16:52:01.8521032Z _=/usr/bin/env 2024-11-01T16:52:01.8521413Z + echo 'Testing pytorch' 2024-11-01T16:52:01.8521794Z Testing pytorch 2024-11-01T16:52:01.8522179Z + export LANG=C.UTF-8 2024-11-01T16:52:01.8522586Z + LANG=C.UTF-8 2024-11-01T16:52:01.8522911Z + PR_NUMBER=138766 2024-11-01T16:52:01.8523278Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-11-01T16:52:01.8523786Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-11-01T16:52:01.8524443Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:01.8524981Z + [[ distributed == \s\l\o\w ]] 2024-11-01T16:52:01.8525633Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-11-01T16:52:01.8526378Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T16:52:01.8526980Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:01.8527627Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:52:01.8528094Z + [[ distributed == *crossref* ]] 2024-11-01T16:52:01.8528650Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-11-01T16:52:01.8529316Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-11-01T16:52:01.8529987Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:01.8530590Z + pip_install --user ninja==1.10.2 2024-11-01T16:52:01.8531185Z + pip install --progress-bar off --user ninja==1.10.2 2024-11-01T16:52:03.5222876Z Collecting ninja==1.10.2 2024-11-01T16:52:03.5417931Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-11-01T16:52:03.5520024Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-11-01T16:52:03.8819710Z Installing collected packages: ninja 2024-11-01T16:52:03.8895528Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-11-01T16:52:03.8897069Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-11-01T16:52:03.8938566Z Successfully installed ninja-1.10.2 2024-11-01T16:52:03.9587140Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:03.9589750Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:03.9591555Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:03.9592099Z + install_tlparse 2024-11-01T16:52:03.9592505Z + pip_install --user tlparse==0.3.25 2024-11-01T16:52:03.9593140Z + pip install --progress-bar off --user tlparse==0.3.25 2024-11-01T16:52:04.3522898Z Collecting tlparse==0.3.25 2024-11-01T16:52:04.3713545Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-11-01T16:52:04.3830590Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-11-01T16:52:04.7273035Z Installing collected packages: tlparse 2024-11-01T16:52:04.7638823Z Successfully installed tlparse-0.3.25 2024-11-01T16:52:04.8299103Z ++ python -m site --user-base 2024-11-01T16:52:04.8558049Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:52:04.8560059Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-11-01T16:52:04.8560780Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-11-01T16:52:04.8561483Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-11-01T16:52:04.8562551Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-11-01T16:52:04.8563860Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-11-01T16:52:04.8564658Z + cd test 2024-11-01T16:52:04.8565284Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-11-01T16:52:06.5718892Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-11-01T16:52:06.5720030Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-11-01T16:52:06.5723892Z + DYNAMO_BENCHMARK_FLAGS=() 2024-11-01T16:52:06.5725101Z + [[ distributed == *pr_time_benchmarks* ]] 2024-11-01T16:52:06.5726074Z + [[ distributed == *dynamo_eager* ]] 2024-11-01T16:52:06.5726972Z + [[ distributed == *aot_eager* ]] 2024-11-01T16:52:06.5727839Z + [[ distributed == *aot_inductor* ]] 2024-11-01T16:52:06.5728814Z + [[ distributed == *inductor* ]] 2024-11-01T16:52:06.5729634Z + [[ distributed == *dynamic* ]] 2024-11-01T16:52:06.5730421Z + [[ distributed == *cpu* ]] 2024-11-01T16:52:06.5731492Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-11-01T16:52:06.5765616Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:06.5766394Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-11-01T16:52:06.5767134Z + cd test 2024-11-01T16:52:06.5768539Z + python -c 'import torch; print(torch.__config__.show())' 2024-11-01T16:52:08.5753671Z PyTorch built with: 2024-11-01T16:52:08.5754472Z - GCC 9.4 2024-11-01T16:52:08.5754840Z - C++ Version: 201703 2024-11-01T16:52:08.5756028Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:08.5757710Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:08.5758508Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:08.5759107Z - LAPACK is enabled (usually provided by MKL) 2024-11-01T16:52:08.5759646Z - NNPACK is enabled 2024-11-01T16:52:08.5760078Z - CPU capability usage: AVX512 2024-11-01T16:52:08.5760549Z - CUDA Runtime 11.8 2024-11-01T16:52:08.5761154Z - NVCC architecture flags: -gencode;arch=compute_75,code=sm_75 2024-11-01T16:52:08.5761787Z - CuDNN 90.1 2024-11-01T16:52:08.5762135Z - Magma 2.6.1 2024-11-01T16:52:08.5771530Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-11-01T16:52:08.5779778Z 2024-11-01T16:52:09.0030826Z + cd test 2024-11-01T16:52:09.0031699Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-11-01T16:52:10.5071004Z ATen/Parallel: 2024-11-01T16:52:10.5071462Z at::get_num_threads() : 24 2024-11-01T16:52:10.5072100Z at::get_num_interop_threads() : 24 2024-11-01T16:52:10.5072595Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:52:10.5073035Z omp_get_max_threads() : 24 2024-11-01T16:52:10.5074311Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:52:10.5075293Z mkl_get_max_threads() : 24 2024-11-01T16:52:10.5076172Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:52:10.5076935Z std::thread::hardware_concurrency() : 48 2024-11-01T16:52:10.5077449Z Environment variables: 2024-11-01T16:52:10.5077823Z OMP_NUM_THREADS : [not set] 2024-11-01T16:52:10.5078247Z MKL_NUM_THREADS : [not set] 2024-11-01T16:52:10.5078664Z ATen parallel backend: OpenMP 2024-11-01T16:52:10.5078965Z 2024-11-01T16:52:10.7487533Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-11-01T16:52:10.7488182Z + [[ distributed == *backward* ]] 2024-11-01T16:52:10.7488653Z + [[ distributed == *xla* ]] 2024-11-01T16:52:10.7489067Z + [[ distributed == *executorch* ]] 2024-11-01T16:52:10.7489579Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-11-01T16:52:10.7490276Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-11-01T16:52:10.7490865Z + [[ distributed == distributed ]] 2024-11-01T16:52:10.7491308Z + test_distributed 2024-11-01T16:52:10.7491764Z + echo 'Testing distributed python tests' 2024-11-01T16:52:10.7492264Z Testing distributed python tests 2024-11-01T16:52:10.7492978Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-11-01T16:52:10.8543119Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T16:52:10.8544447Z import pkg_resources 2024-11-01T16:52:15.9920730Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T16:52:16.0658630Z Ignoring disabled issues: [''] 2024-11-01T16:52:16.0802169Z Found test times from artifacts 2024-11-01T16:52:16.1359650Z Found test times from artifacts 2024-11-01T16:52:16.1382108Z Running 25% of tests based on TD 2024-11-01T16:52:16.1483007Z Running parallel tests on 2 processes 2024-11-01T16:52:16.1484005Z Name: tests to run (est. time: 7.57min) 2024-11-01T16:52:16.1484490Z Serial tests (27): 2024-11-01T16:52:16.1484953Z distributed/_tensor/test_dtensor_compile 1/1 2024-11-01T16:52:16.1485765Z distributed/_tensor/debug/test_op_coverage 1/1 2024-11-01T16:52:16.1486670Z distributed/flight_recorder/test_fr_analysis 1/1 2024-11-01T16:52:16.1487381Z distributed/_tools/test_mod_tracker 1/1 2024-11-01T16:52:16.1487966Z distributed/checkpoint/test_nested_dict 1/1 2024-11-01T16:52:16.1488578Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-11-01T16:52:16.1489256Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-11-01T16:52:16.1489939Z distributed/pipelining/test_schedule_multiproc 1/1 2024-11-01T16:52:16.1490527Z distributed/_tools/test_sac_ilp 1/1 2024-11-01T16:52:16.1491054Z distributed/pipelining/test_backward 1/1 2024-11-01T16:52:16.1491623Z distributed/pipelining/test_transformer 1/1 2024-11-01T16:52:16.1492195Z distributed/_tools/test_sac_estimator 1/1 2024-11-01T16:52:16.1492754Z distributed/checkpoint/test_dedup_tensors 1/1 2024-11-01T16:52:16.1493320Z distributed/pipelining/test_pipe 1/1 2024-11-01T16:52:16.1493851Z distributed/_tensor/test_common_rules 1/1 2024-11-01T16:52:16.1494387Z distributed/_tools/test_mem_tracker 1/1 2024-11-01T16:52:16.1494941Z distributed/_composable/test_contract 1/1 2024-11-01T16:52:16.1495465Z distributed/fsdp/test_fsdp_fx 1/1 2024-11-01T16:52:16.1495946Z distributed/test_symmetric_memory 1/1 2024-11-01T16:52:16.1496458Z distributed/_tensor/test_dtensor 1/1 2024-11-01T16:52:16.1496999Z distributed/_tools/test_runtime_estimator 1/1 2024-11-01T16:52:16.1497541Z distributed/_tensor/test_api 1/1 2024-11-01T16:52:16.1498042Z distributed/_tensor/test_dtensor_ops 1/1 2024-11-01T16:52:16.1498548Z distributed/_tensor/test_init 1/1 2024-11-01T16:52:16.1499183Z distributed/_tensor/test_convolution_ops 1/1 2024-11-01T16:52:16.1499743Z distributed/_tensor/test_math_ops 1/1 2024-11-01T16:52:16.1500296Z distributed/checkpoint/test_fsdp_optim_state 1/1 2024-11-01T16:52:16.1500826Z Parallel tests (0): 2024-11-01T16:52:16.1501205Z Name: excluded (est. time: 102.54min) 2024-11-01T16:52:16.1501630Z Serial tests (53): 2024-11-01T16:52:16.1502117Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-11-01T16:52:16.1502732Z distributed/test_distributed_spawn 4/10 2024-11-01T16:52:16.1503258Z distributed/test_distributed_spawn 7/10 2024-11-01T16:52:16.1503790Z distributed/test_distributed_spawn 10/10 2024-11-01T16:52:16.1504340Z distributed/fsdp/test_fsdp_optim_state 1/1 2024-11-01T16:52:16.1504854Z distributed/fsdp/test_fsdp_core 1/2 2024-11-01T16:52:16.1505327Z distributed/test_c10d_nccl 2/3 2024-11-01T16:52:16.1505826Z distributed/rpc/cuda/test_tensorpipe_agent 1/2 2024-11-01T16:52:16.1506416Z distributed/fsdp/test_fsdp_use_orig_params 1/1 2024-11-01T16:52:16.1506948Z distributed/test_device_mesh 1/1 2024-11-01T16:52:16.1507588Z distributed/_composable/test_composability/test_pp_composability 1/1 2024-11-01T16:52:16.1508335Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-11-01T16:52:16.1508926Z distributed/checkpoint/test_state_dict 1/1 2024-11-01T16:52:16.1509481Z distributed/fsdp/test_fsdp_checkpoint 1/1 2024-11-01T16:52:16.1510038Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-11-01T16:52:16.1510618Z distributed/fsdp/test_fsdp_ignored_modules 1/1 2024-11-01T16:52:16.1511174Z distributed/pipelining/test_schedule 1/1 2024-11-01T16:52:16.1511736Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-11-01T16:52:16.1512591Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-11-01T16:52:16.1513250Z distributed/_tensor/test_matrix_ops 1/1 2024-11-01T16:52:16.1513776Z distributed/_tensor/test_optimizers 1/1 2024-11-01T16:52:16.1514310Z distributed/checkpoint/test_checkpoint 1/1 2024-11-01T16:52:16.1514849Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-11-01T16:52:16.1515355Z distributed/launcher/test_run 1/1 2024-11-01T16:52:16.1515912Z distributed/test_store 1/1 2024-11-01T16:52:16.1516569Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 2024-11-01T16:52:16.1517225Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 2024-11-01T16:52:16.1517960Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-11-01T16:52:16.1518543Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-11-01T16:52:16.1519084Z distributed/fsdp/test_fsdp_overlap 1/1 2024-11-01T16:52:16.1519673Z distributed/tensor/parallel/test_tp_examples 1/1 2024-11-01T16:52:16.1520463Z distributed/_composable/fully_shard/test_fully_shard_model_checkpoint 1/1 2024-11-01T16:52:16.1521245Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-11-01T16:52:16.1521922Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2024-11-01T16:52:16.1522661Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-11-01T16:52:16.1523371Z distributed/elastic/timer/local_timer_example 1/1 2024-11-01T16:52:16.1524051Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2024-11-01T16:52:16.1524671Z distributed/fsdp/test_fsdp_memory 1/1 2024-11-01T16:52:16.1525320Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2024-11-01T16:52:16.1526123Z distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 2024-11-01T16:52:16.1526822Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-11-01T16:52:16.1527459Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 2024-11-01T16:52:16.1528143Z distributed/fsdp/test_fsdp_uneven 1/1 2024-11-01T16:52:16.1528724Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2024-11-01T16:52:16.1529324Z distributed/fsdp/test_fsdp_traversal 1/1 2024-11-01T16:52:16.1548543Z distributed/test_data_parallel 1/1 2024-11-01T16:52:16.1549275Z distributed/rpc/test_share_memory 1/1 2024-11-01T16:52:16.1549933Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2024-11-01T16:52:16.1550614Z distributed/elastic/utils/logging_test 1/1 2024-11-01T16:52:16.1551200Z distributed/elastic/metrics/api_test 1/1 2024-11-01T16:52:16.1551770Z distributed/test_collective_utils 1/1 2024-11-01T16:52:16.1552271Z distributed/rpc/test_faulty_agent 1/1 2024-11-01T16:52:16.1552786Z distributed/elastic/timer/api_test 1/1 2024-11-01T16:52:16.1553270Z Parallel tests (0): 2024-11-01T16:52:16.1554156Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-11-01 16:52:16.154685] 2024-11-01T16:52:16.1554882Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:52:16.1557242Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:52:16.155041] 2024-11-01T16:53:01.1308883Z 2024-11-01T16:53:01.1312926Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_70eb0ddd4dc317b1_.log 2024-11-01T16:53:01.1365048Z Running 28 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph 2024-11-01T16:53:01.1397378Z 2024-11-01T16:53:01.1398088Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-11-01 16:53:01.130897] 2024-11-01T16:53:01.1398884Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:01.1401044Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:01.131282] 2024-11-01T16:53:05.7579081Z 2024-11-01T16:53:05.7581030Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_65ff37a4dd8daa37_.log 2024-11-01T16:53:05.7583239Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-11-01T16:53:05.7584163Z 2024-11-01T16:53:05.7584745Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2024-11-01 16:53:05.757781] 2024-11-01T16:53:05.7585932Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:05.7588125Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:05.758150] 2024-11-01T16:53:10.0341045Z 2024-11-01T16:53:10.0342965Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_71252e0be2d88ebd_.log 2024-11-01T16:53:10.0345231Z Running 1 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event 2024-11-01T16:53:10.0346153Z 2024-11-01T16:53:10.0347155Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-11-01 16:53:10.033974] 2024-11-01T16:53:10.0347914Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:10.0350203Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:10.034335] 2024-11-01T16:53:14.2600370Z 2024-11-01T16:53:14.2603409Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_6aee17f4e6b26b99_.log 2024-11-01T16:53:14.2606719Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-11-01T16:53:14.2608860Z 2024-11-01T16:53:14.2609422Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2024-11-01 16:53:14.260427] 2024-11-01T16:53:14.2610200Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:14.2612268Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:14.260784] 2024-11-01T16:53:18.4364580Z 2024-11-01T16:53:18.4366476Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_34d695b30125bdef_.log 2024-11-01T16:53:18.4369025Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping 2024-11-01T16:53:18.4370377Z 2024-11-01T16:53:18.4370972Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-11-01 16:53:18.436401] 2024-11-01T16:53:18.4371780Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:18.4373910Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:18.436770] 2024-11-01T16:53:22.7127082Z 2024-11-01T16:53:22.7129010Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_20f1d7add8153903_.log 2024-11-01T16:53:22.7131196Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-11-01T16:53:22.7132151Z 2024-11-01T16:53:22.7132781Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-11-01 16:53:22.712676] 2024-11-01T16:53:22.7133630Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:22.7135824Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:22.713045] 2024-11-01T16:53:40.0199583Z 2024-11-01T16:53:40.0202060Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_47dd6aa99b87e5e0_.log 2024-11-01T16:53:40.0228042Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-11-01T16:53:40.0252931Z 2024-11-01T16:53:40.0253549Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-11-01 16:53:40.019971] 2024-11-01T16:53:40.0254362Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:40.0256486Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:40.020350] 2024-11-01T16:53:55.7271583Z 2024-11-01T16:53:55.7274131Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_8be4a7a397a7c2de_.log 2024-11-01T16:53:55.7293510Z Running 22 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-11-01T16:53:55.7328777Z Running 22 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-11-01T16:53:55.7346263Z 2024-11-01T16:53:55.7346771Z Running distributed/_tools/test_sac_ilp 1/1 ... [2024-11-01 16:53:55.727053] 2024-11-01T16:53:55.7347442Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:53:55.7349379Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_ilp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:53:55.727426] 2024-11-01T16:54:08.6231298Z 2024-11-01T16:54:08.6233055Z distributed/_tools/test_sac_ilp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_ilp_1.1_4f00f288e91c013f_.log 2024-11-01T16:54:08.6236850Z Running 4 items in this shard: test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case1, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case2, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case3, test/distributed/_tools/test_sac_ilp.py::TestOptimalCheckpointingPolicy::test_get_optimial_checkpointing_policy_per_module 2024-11-01T16:54:08.6239265Z 2024-11-01T16:54:08.6239805Z Running distributed/pipelining/test_backward 1/1 ... [2024-11-01 16:54:08.622957] 2024-11-01T16:54:08.6240572Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:08.6242797Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:08.623320] 2024-11-01T16:54:12.8987998Z 2024-11-01T16:54:12.8991078Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_bb93c9fecac27a7c_.log 2024-11-01T16:54:12.8997856Z Running 4 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_input, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight, test/distributed/pipelining/test_backward.py::StageBackwardTests::test_stage_backward_weight_multiple_iters 2024-11-01T16:54:12.9002585Z 2024-11-01T16:54:12.9003254Z Running distributed/pipelining/test_transformer 1/1 ... [2024-11-01 16:54:12.899138] 2024-11-01T16:54:12.9004038Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:12.9006452Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:12.899698] 2024-11-01T16:54:17.6771118Z 2024-11-01T16:54:17.6773043Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_6a86bcedc98bbfc6_.log 2024-11-01T16:54:17.6775001Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-11-01T16:54:17.6775775Z 2024-11-01T16:54:17.6776307Z Running distributed/_tools/test_sac_estimator 1/1 ... [2024-11-01 16:54:17.676976] 2024-11-01T16:54:17.6777046Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:17.6779091Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:17.677410] 2024-11-01T16:54:25.2605680Z 2024-11-01T16:54:25.2607492Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_cd373433f0b2e23f_.log 2024-11-01T16:54:25.2610136Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation 2024-11-01T16:54:25.2611610Z 2024-11-01T16:54:25.2612217Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-11-01 16:54:25.260717] 2024-11-01T16:54:25.2612982Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:25.2615080Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:25.261099] 2024-11-01T16:54:29.4372824Z 2024-11-01T16:54:29.4375798Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_2783f4abb9a4cdb9_.log 2024-11-01T16:54:29.4377893Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-11-01T16:54:29.4378753Z 2024-11-01T16:54:29.4379791Z Running distributed/pipelining/test_pipe 1/1 ... [2024-11-01 16:54:29.437656] 2024-11-01T16:54:29.4380517Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:29.4383025Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:29.438055] 2024-11-01T16:54:34.3657703Z 2024-11-01T16:54:34.3660037Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_5d21ba1aff0e62d6_.log 2024-11-01T16:54:34.3662987Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2 2024-11-01T16:54:34.3664840Z 2024-11-01T16:54:34.3665351Z Running distributed/_tensor/test_common_rules 1/1 ... [2024-11-01 16:54:34.365862] 2024-11-01T16:54:34.3666213Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:54:34.3668242Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:34.366263] 2024-11-01T16:55:25.7615280Z 2024-11-01T16:55:25.7617111Z distributed/_tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_common_rules_1.1_a71396e863f56e29_.log 2024-11-01T16:55:25.7626890Z Running 10 items in this shard: test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/_tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2024-11-01T16:55:25.7632921Z 2024-11-01T16:55:25.7633467Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-11-01 16:55:25.761478] 2024-11-01T16:55:25.7634167Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:25.7636413Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:25.761844] 2024-11-01T16:55:30.6391013Z 2024-11-01T16:55:30.6392948Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_4a7e4b6579a470fe_.log 2024-11-01T16:55:30.6396434Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_cuda_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing 2024-11-01T16:55:30.6398582Z 2024-11-01T16:55:30.6399107Z Running distributed/_composable/test_contract 1/1 ... [2024-11-01 16:55:30.639208] 2024-11-01T16:55:30.6399862Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:30.6401992Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:30.639613] 2024-11-01T16:55:34.9156467Z 2024-11-01T16:55:34.9158337Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_6f0a26780690278f_.log 2024-11-01T16:55:34.9162787Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-11-01T16:55:34.9165529Z 2024-11-01T16:55:34.9166170Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2024-11-01 16:55:34.915877] 2024-11-01T16:55:34.9166860Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:34.9168871Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:34.916328] 2024-11-01T16:55:39.1922764Z 2024-11-01T16:55:39.1924520Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_ce11af7453bf1296_.log 2024-11-01T16:55:39.1926442Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracing::test_symbolic_tracing_outputs 2024-11-01T16:55:39.1927291Z 2024-11-01T16:55:39.1927807Z Running distributed/test_symmetric_memory 1/1 ... [2024-11-01 16:55:39.192316] 2024-11-01T16:55:39.1928505Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:39.1930555Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:39.192689] 2024-11-01T16:55:43.2177434Z 2024-11-01T16:55:43.2179207Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_3fd453bd33285a94_.log 2024-11-01T16:55:43.2269642Z Running 103 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_barrier_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_put_signal_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_stream_write_value, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_wait_signal_timeout, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce 2024-11-01T16:55:43.2347597Z 2024-11-01T16:55:43.2348112Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-11-01 16:55:43.217942] 2024-11-01T16:55:43.2348813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:55:43.2350816Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:55:43.218306] 2024-11-01T16:58:31.7266812Z 2024-11-01T16:58:31.7268604Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_d2b1bb80e177cb50_.log 2024-11-01T16:58:31.7288225Z Running 33 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication_for_foreach_ops, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/_tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log 2024-11-01T16:58:31.7305969Z 2024-11-01T16:58:31.7306561Z Running distributed/_tools/test_runtime_estimator 1/1 ... [2024-11-01 16:58:31.726323] 2024-11-01T16:58:31.7307322Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:31.7309391Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_runtime_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:31.726678] 2024-11-01T16:58:47.1271245Z 2024-11-01T16:58:47.1273176Z distributed/_tools/test_runtime_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_runtime_estimator_1.1_3c35bc9ba0a4a720_.log 2024-11-01T16:58:47.1276007Z Running 2 items in this shard: test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_conv_model_runtime, test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_transformer_runtime 2024-11-01T16:58:47.1277816Z 2024-11-01T16:58:47.1278285Z Running distributed/_tensor/test_api 1/1 ... [2024-11-01 16:58:47.127049] 2024-11-01T16:58:47.1278995Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:58:47.1281162Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:58:47.127409] 2024-11-01T16:59:29.0500049Z 2024-11-01T16:59:29.0501731Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_3e511fb06118dc4b_.log 2024-11-01T16:59:29.0507490Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-11-01T16:59:29.0512002Z 2024-11-01T16:59:29.0512522Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-11-01 16:59:29.050129] 2024-11-01T16:59:29.0513248Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:29.0515263Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:29.050513] 2024-11-01T16:59:34.0000534Z 2024-11-01T16:59:34.0002372Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_b29edaca3960e428_.log 2024-11-01T16:59:34.0003818Z Running 0 items in this shard: 2024-11-01T16:59:34.0004112Z 2024-11-01T16:59:34.0005529Z Running distributed/_tensor/test_init 1/1 ... [2024-11-01 16:59:34.000393] 2024-11-01T16:59:34.0006215Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T16:59:34.0010476Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:59:34.000791] 2024-11-01T17:00:11.2132993Z 2024-11-01T17:00:11.2134806Z distributed/_tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_init_1.1_96514d064e56d661_.log 2024-11-01T17:00:11.2139522Z Running 7 items in this shard: test/distributed/_tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/_tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh 2024-11-01T17:00:11.2143064Z 2024-11-01T17:00:11.2143583Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-11-01 17:00:11.213389] 2024-11-01T17:00:11.2144332Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:11.2146650Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:11.213751] 2024-11-01T17:00:24.0615870Z 2024-11-01T17:00:24.0618701Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_5dba3cc5c7caddfc_.log 2024-11-01T17:00:24.0623360Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-11-01T17:00:24.0625442Z 2024-11-01T17:00:24.0626006Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-11-01 17:00:24.061462] 2024-11-01T17:00:24.0626711Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:00:24.0628987Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:00:24.061951] 2024-11-01T17:01:34.3101992Z 2024-11-01T17:01:34.3104090Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_482e7bc859a26c1d_.log 2024-11-01T17:01:34.3112271Z Running 14 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2024-11-01T17:01:34.3120040Z 2024-11-01T17:01:34.3120660Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2024-11-01 17:01:34.310169] 2024-11-01T17:01:34.3121471Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:34.3123660Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:01:34.310543] 2024-11-01T17:01:43.2489168Z 2024-11-01T17:01:43.2491246Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_3997a6debebcc31f_.log 2024-11-01T17:01:43.2493692Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict 2024-11-01T17:01:43.2494752Z 2024-11-01T17:01:43.9589523Z Running test batch 'tests to run' cost 567.81 seconds 2024-11-01T17:01:44.7351509Z 2024-11-01T17:01:44.7351990Z real 9m33.986s 2024-11-01T17:01:44.7352384Z user 30m12.078s 2024-11-01T17:01:44.7352709Z sys 10m26.914s 2024-11-01T17:01:44.7353026Z + assert_git_not_dirty 2024-11-01T17:01:44.7357142Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-11-01T17:01:44.7357882Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-11-01T17:01:44.7358471Z ++ git status --porcelain 2024-11-01T17:01:44.7358916Z ++ grep -v '?? third_party' 2024-11-01T17:01:47.3136028Z ++ true 2024-11-01T17:01:47.3136398Z + git_status= 2024-11-01T17:01:47.3137005Z + [[ -n '' ]] 2024-11-01T17:01:47.3137507Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T17:01:47.3138034Z + [[ 1 == 1 ]] 2024-11-01T17:01:47.3138442Z + echo 'Testing distributed C++ tests' 2024-11-01T17:01:47.3138910Z Testing distributed C++ tests 2024-11-01T17:01:47.3143974Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:01:47.3157193Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:01:47.3165087Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:01:47.3166209Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:01:47.3167165Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-11-01T17:01:47.4197700Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:01:47.4199077Z import pkg_resources 2024-11-01T17:01:52.2757790Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:01:52.2893595Z Found test times from artifacts 2024-11-01T17:01:52.3461202Z Found test times from artifacts 2024-11-01T17:01:52.3478528Z Running 25% of tests based on TD 2024-11-01T17:01:52.3479048Z Running parallel tests on 2 processes 2024-11-01T17:01:52.3479649Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:01:52.3480124Z Serial tests (1): 2024-11-01T17:01:52.3480509Z cpp/FileStoreTest 1/1 2024-11-01T17:01:52.3480906Z Parallel tests (0): 2024-11-01T17:01:52.3481301Z Name: excluded (est. time: 0.0min) 2024-11-01T17:01:52.3481736Z Serial tests (0): 2024-11-01T17:01:52.3482075Z Parallel tests (0): 2024-11-01T17:01:52.3539603Z Running cpp/FileStoreTest 1/1 ... [2024-11-01 17:01:52.353439] 2024-11-01T17:01:52.3540233Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:01:52.3542542Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-a9e4eece8c3f990d.xml', '-x', '--reruns=2'] ... [2024-11-01 17:01:52.353979] 2024-11-01T17:01:55.6780441Z 2024-11-01T17:01:55.6782015Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_8088badde510d210_.log 2024-11-01T17:01:55.6783068Z 2024-11-01T17:01:56.8038377Z Running test batch 'tests to run' cost 4.46 seconds 2024-11-01T17:01:57.5581938Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-11-01T17:01:57.6630373Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:01:57.6631733Z import pkg_resources 2024-11-01T17:02:02.5543755Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:02:02.5680115Z Found test times from artifacts 2024-11-01T17:02:02.6255427Z Found test times from artifacts 2024-11-01T17:02:02.6274792Z Running 25% of tests based on TD 2024-11-01T17:02:02.6275314Z Running parallel tests on 2 processes 2024-11-01T17:02:02.6275978Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:02:02.6276430Z Serial tests (1): 2024-11-01T17:02:02.6276965Z cpp/HashStoreTest 1/1 2024-11-01T17:02:02.6277358Z Parallel tests (0): 2024-11-01T17:02:02.6278022Z Name: excluded (est. time: 0.0min) 2024-11-01T17:02:02.6278460Z Serial tests (0): 2024-11-01T17:02:02.6278805Z Parallel tests (0): 2024-11-01T17:02:02.6327763Z Running cpp/HashStoreTest 1/1 ... [2024-11-01 17:02:02.632538] 2024-11-01T17:02:02.6328522Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:02.6333003Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-c5635d7f753fe5a5.xml', '-x', '--reruns=2'] ... [2024-11-01 17:02:02.633034] 2024-11-01T17:02:05.7064507Z 2024-11-01T17:02:05.7066549Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_c48f9e5e76cdc00d_.log 2024-11-01T17:02:05.7067625Z 2024-11-01T17:02:07.1001182Z Running test batch 'tests to run' cost 4.47 seconds 2024-11-01T17:02:07.8589598Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-11-01T17:02:07.9638813Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:02:07.9640212Z import pkg_resources 2024-11-01T17:02:12.8492756Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:02:12.8633855Z Found test times from artifacts 2024-11-01T17:02:12.9195412Z Found test times from artifacts 2024-11-01T17:02:12.9215875Z Running 25% of tests based on TD 2024-11-01T17:02:12.9216494Z Running parallel tests on 2 processes 2024-11-01T17:02:12.9217012Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:02:12.9217465Z Serial tests (1): 2024-11-01T17:02:12.9217821Z cpp/TCPStoreTest 1/1 2024-11-01T17:02:12.9218205Z Parallel tests (0): 2024-11-01T17:02:12.9218584Z Name: excluded (est. time: 0.0min) 2024-11-01T17:02:12.9219432Z Serial tests (0): 2024-11-01T17:02:12.9219859Z Parallel tests (0): 2024-11-01T17:02:12.9271393Z Running cpp/TCPStoreTest 1/1 ... [2024-11-01 17:02:12.926654] 2024-11-01T17:02:12.9272045Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:12.9274225Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-1a6b111c3db06ac6.xml', '-x', '--reruns=2'] ... [2024-11-01 17:02:12.927128] 2024-11-01T17:02:27.4851150Z 2024-11-01T17:02:27.4852820Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_1eeea72e704cf4c4_.log 2024-11-01T17:02:27.4853902Z 2024-11-01T17:02:28.1828314Z Running test batch 'tests to run' cost 15.26 seconds 2024-11-01T17:02:28.9387037Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-11-01T17:02:28.9387693Z ++ command -v mpiexec 2024-11-01T17:02:28.9391135Z + MPIEXEC=/usr/bin/mpiexec 2024-11-01T17:02:28.9391658Z + [[ -n /usr/bin/mpiexec ]] 2024-11-01T17:02:28.9392744Z + MPICMD='/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-11-01T17:02:28.9394380Z + eval '/usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest' 2024-11-01T17:02:28.9396053Z ++ /usr/bin/mpiexec -np 2 /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupMPITest 2024-11-01T17:02:31.0727632Z Test successful 2024-11-01T17:02:31.0785948Z Test successful 2024-11-01T17:02:31.2319593Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupGlooTest 2024-11-01T17:02:31.3364196Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:02:31.3365570Z import pkg_resources 2024-11-01T17:02:36.2231504Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:02:36.2364927Z Found test times from artifacts 2024-11-01T17:02:36.2925068Z Found test times from artifacts 2024-11-01T17:02:36.2947411Z Running 25% of tests based on TD 2024-11-01T17:02:36.2947934Z Running parallel tests on 2 processes 2024-11-01T17:02:36.2948413Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:02:36.2948869Z Serial tests (1): 2024-11-01T17:02:36.2950628Z cpp/ProcessGroupGlooTest 1/1 2024-11-01T17:02:36.2951065Z Parallel tests (0): 2024-11-01T17:02:36.2951448Z Name: excluded (est. time: 0.0min) 2024-11-01T17:02:36.2951872Z Serial tests (0): 2024-11-01T17:02:36.2952430Z Parallel tests (0): 2024-11-01T17:02:36.3001334Z Running cpp/ProcessGroupGlooTest 1/1 ... [2024-11-01 17:02:36.299897] 2024-11-01T17:02:36.3002059Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:36.3007060Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e15bd1e5ce6b5452.xml', '-x', '--reruns=2'] ... [2024-11-01 17:02:36.300424] 2024-11-01T17:02:44.4864184Z 2024-11-01T17:02:44.4865918Z cpp/ProcessGroupGlooTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupGlooTest_1.1_d7b1089643579378_.log 2024-11-01T17:02:44.4867075Z 2024-11-01T17:02:45.1957274Z Running test batch 'tests to run' cost 8.9 seconds 2024-11-01T17:02:45.9507446Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLTest 2024-11-01T17:02:46.0557664Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:02:46.0559046Z import pkg_resources 2024-11-01T17:02:50.9126611Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:02:50.9267654Z Found test times from artifacts 2024-11-01T17:02:50.9835450Z Found test times from artifacts 2024-11-01T17:02:50.9855003Z Running 25% of tests based on TD 2024-11-01T17:02:50.9855534Z Running parallel tests on 2 processes 2024-11-01T17:02:50.9856032Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:02:50.9856490Z Serial tests (1): 2024-11-01T17:02:50.9856868Z cpp/ProcessGroupNCCLTest 1/1 2024-11-01T17:02:50.9857306Z Parallel tests (0): 2024-11-01T17:02:50.9857852Z Name: excluded (est. time: 0.0min) 2024-11-01T17:02:50.9858282Z Serial tests (0): 2024-11-01T17:02:50.9858629Z Parallel tests (0): 2024-11-01T17:02:50.9911361Z Running cpp/ProcessGroupNCCLTest 1/1 ... [2024-11-01 17:02:50.990596] 2024-11-01T17:02:50.9912267Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:02:50.9914602Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-74d8d16f16535f99.xml', '-x', '--reruns=2'] ... [2024-11-01 17:02:50.991088] 2024-11-01T17:03:11.0077493Z 2024-11-01T17:03:11.0079362Z cpp/ProcessGroupNCCLTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLTest_1.1_aced1331873cbd20_.log 2024-11-01T17:03:11.0080746Z 2024-11-01T17:03:11.7157811Z Running test batch 'tests to run' cost 20.73 seconds 2024-11-01T17:03:12.4832895Z + python test/run_test.py --cpp --verbose -i cpp/ProcessGroupNCCLErrorsTest 2024-11-01T17:03:12.5881311Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:03:12.5882703Z import pkg_resources 2024-11-01T17:03:17.4619929Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:03:17.4758093Z Found test times from artifacts 2024-11-01T17:03:17.5323074Z Found test times from artifacts 2024-11-01T17:03:17.5345645Z Running 25% of tests based on TD 2024-11-01T17:03:17.5346167Z Running parallel tests on 2 processes 2024-11-01T17:03:17.5346658Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:03:17.5348626Z Serial tests (1): 2024-11-01T17:03:17.5349097Z cpp/ProcessGroupNCCLErrorsTest 1/1 2024-11-01T17:03:17.5349585Z Parallel tests (0): 2024-11-01T17:03:17.5349968Z Name: excluded (est. time: 0.0min) 2024-11-01T17:03:17.5350397Z Serial tests (0): 2024-11-01T17:03:17.5350737Z Parallel tests (0): 2024-11-01T17:03:17.5399956Z Running cpp/ProcessGroupNCCLErrorsTest 1/1 ... [2024-11-01 17:03:17.539734] 2024-11-01T17:03:17.5400746Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:03:17.5405066Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-86ab93214f94d5d5.xml', '-x', '--reruns=2'] ... [2024-11-01 17:03:17.540221] 2024-11-01T17:03:30.4928239Z 2024-11-01T17:03:30.4930595Z cpp/ProcessGroupNCCLErrorsTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_5d9e9390eca3ddf3_.log 2024-11-01T17:03:31.2006453Z 2024-11-01T17:03:31.2007321Z Running test batch 'tests to run' cost 13.67 seconds 2024-11-01T17:03:31.9662650Z + [[ 1 == 1 ]] 2024-11-01T17:03:31.9663063Z + test_rpc 2024-11-01T17:03:31.9663662Z + echo 'Testing RPC C++ tests' 2024-11-01T17:03:31.9664087Z Testing RPC C++ tests 2024-11-01T17:03:31.9669139Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:03:31.9681303Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:03:31.9693244Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:03:31.9694159Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-11-01T17:03:32.0742272Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:03:32.0743623Z import pkg_resources 2024-11-01T17:03:36.9420932Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-11-01T17:03:36.9558005Z Found test times from artifacts 2024-11-01T17:03:37.0120926Z Found test times from artifacts 2024-11-01T17:03:37.0142389Z Running 25% of tests based on TD 2024-11-01T17:03:37.0143533Z Running parallel tests on 2 processes 2024-11-01T17:03:37.0144460Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:03:37.0144902Z Serial tests (1): 2024-11-01T17:03:37.0145255Z cpp/test_cpp_rpc 1/1 2024-11-01T17:03:37.0145630Z Parallel tests (0): 2024-11-01T17:03:37.0146026Z Name: excluded (est. time: 0.0min) 2024-11-01T17:03:37.0148722Z Serial tests (0): 2024-11-01T17:03:37.0149116Z Parallel tests (0): 2024-11-01T17:03:37.0196136Z Running cpp/test_cpp_rpc 1/1 ... [2024-11-01 17:03:37.019371] 2024-11-01T17:03:37.0196783Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-11-01T17:03:37.0201464Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-3960f850d6b81f5b.xml', '-x', '--reruns=2'] ... [2024-11-01 17:03:37.019870] 2024-11-01T17:03:43.6019161Z 2024-11-01T17:03:43.6020995Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_8055b88911d89afd_.log 2024-11-01T17:03:44.3077084Z 2024-11-01T17:03:44.3077973Z Running test batch 'tests to run' cost 7.29 seconds 2024-11-01T17:03:45.0627798Z + cleanup_workspace 2024-11-01T17:03:45.0629285Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-11-01T17:03:45.0630573Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-11-01T17:03:45.0631681Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-11-01T17:03:45.0632414Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-11-01T17:03:45.0633348Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-11-01T17:03:45.0634372Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-11-01T17:03:45.0635148Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-11-01T17:03:45.5628609Z ##[group]Run cat test/**/*_toprint.log || true 2024-11-01T17:03:45.5629294Z cat test/**/*_toprint.log || true 2024-11-01T17:03:45.5639917Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:45.5640492Z env: 2024-11-01T17:03:45.5640789Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:45.5641300Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:45.5642169Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:45.5642934Z ##[endgroup] 2024-11-01T17:03:45.5725378Z cat: 'test/**/*_toprint.log': No such file or directory 2024-11-01T17:03:45.5758963Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:03:45.5759486Z kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:03:45.5765750Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:45.5766251Z env: 2024-11-01T17:03:45.5766522Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:45.5766985Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:45.5767769Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:45.5768475Z MONITOR_SCRIPT_PID: 119117 2024-11-01T17:03:45.5768824Z ##[endgroup] 2024-11-01T17:03:45.6012865Z Prepare all required actions 2024-11-01T17:03:45.6013330Z Getting action download info 2024-11-01T17:03:45.8194258Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-11-01T17:03:45.9856840Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-11-01T17:03:45.9857306Z with: 2024-11-01T17:03:45.9857887Z file-suffix: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T17:03:45.9858591Z s3-bucket: gha-artifacts 2024-11-01T17:03:45.9858923Z env: 2024-11-01T17:03:45.9859184Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:45.9859619Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:45.9860392Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:45.9861068Z ##[endgroup] 2024-11-01T17:03:45.9905014Z ##[group]Run # Remove any previous test jsons if they exist 2024-11-01T17:03:45.9905706Z # Remove any previous test jsons if they exist 2024-11-01T17:03:45.9906228Z rm -f test-jsons-*.zip 2024-11-01T17:03:45.9907003Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-11-01T17:03:45.9913777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:45.9914316Z env: 2024-11-01T17:03:45.9914598Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:45.9915081Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:45.9916332Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:45.9917411Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T17:03:45.9918141Z ##[endgroup] 2024-11-01T17:03:45.9972684Z adding: test/test-reports/td_exclusions-035bc4272960d8f5bc01.json (deflated 86%) 2024-11-01T17:03:45.9973846Z adding: test/test-reports/td_exclusions-c4b37a86de388f66dcfd.json (deflated 15%) 2024-11-01T17:03:45.9974971Z adding: test/test-reports/td_exclusions-7a97f62f1c0ef7b8ff33.json (deflated 14%) 2024-11-01T17:03:45.9976071Z adding: test/test-reports/td_exclusions-651aab16b654742452f7.json (deflated 14%) 2024-11-01T17:03:45.9977230Z adding: test/test-reports/td_exclusions-f34afb2ecd9e0fe90b76.json (deflated 12%) 2024-11-01T17:03:45.9978373Z adding: test/test-reports/td_exclusions-7ebbe49e7d8c42ffd655.json (deflated 12%) 2024-11-01T17:03:45.9979478Z adding: test/test-reports/td_exclusions-acd9b269257a82405ca2.json (deflated 11%) 2024-11-01T17:03:45.9980580Z adding: test/test-reports/td_exclusions-ea817b09a56d620d4a8e.json (deflated 18%) 2024-11-01T17:03:46.0033943Z ##[group]Run # Remove any previous test reports if they exist 2024-11-01T17:03:46.0034670Z # Remove any previous test reports if they exist 2024-11-01T17:03:46.0035242Z rm -f test-reports-*.zip 2024-11-01T17:03:46.0036102Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-11-01T17:03:46.0042812Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:46.0043372Z env: 2024-11-01T17:03:46.0043667Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:46.0044176Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:46.0045046Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:46.0046112Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T17:03:46.0046862Z ##[endgroup] 2024-11-01T17:03:46.0101071Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-0b181b7759941ce1.xml (deflated 84%) 2024-11-01T17:03:46.0103526Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-c359fa85753f7b33.xml (deflated 34%) 2024-11-01T17:03:46.0106145Z adding: test/test-reports/python-pytest/distributed.flight_recorder.test_fr_analysis/distributed.flight_recorder.test_fr_analysis-7403ffd63d641d8c.xml (deflated 36%) 2024-11-01T17:03:46.0108460Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-f23d5a6ee2a03066.xml (deflated 65%) 2024-11-01T17:03:46.0110695Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_nested_dict/distributed.checkpoint.test_nested_dict-f8edff064901a619.xml (deflated 51%) 2024-11-01T17:03:46.0113059Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-4a90c2948a93c5cd.xml (deflated 37%) 2024-11-01T17:03:46.0115488Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-178052d7a92c1aba.xml (deflated 89%) 2024-11-01T17:03:46.0118382Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-cfeb69aaea84e83e.xml (deflated 86%) 2024-11-01T17:03:46.0120975Z adding: test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-e984e9ebc2742878.xml (deflated 86%) 2024-11-01T17:03:46.0123328Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_ilp/distributed._tools.test_sac_ilp-21c1cacad1957518.xml (deflated 66%) 2024-11-01T17:03:46.0125529Z adding: test/test-reports/python-pytest/distributed.pipelining.test_backward/distributed.pipelining.test_backward-69852ca4e70d46f3.xml (deflated 68%) 2024-11-01T17:03:46.0128107Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-ae30bb9455c307d7.xml (deflated 39%) 2024-11-01T17:03:46.0130311Z adding: test/test-reports/python-pytest/distributed._tools.test_sac_estimator/distributed._tools.test_sac_estimator-a74afc15ab0b06b6.xml (deflated 51%) 2024-11-01T17:03:46.0132710Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-f10f1822dd4c6de9.xml (deflated 38%) 2024-11-01T17:03:46.0134883Z adding: test/test-reports/python-pytest/distributed.pipelining.test_pipe/distributed.pipelining.test_pipe-2e9e04b0413b2f8b.xml (deflated 65%) 2024-11-01T17:03:46.0136917Z adding: test/test-reports/python-pytest/distributed._tensor.test_common_rules/distributed._tensor.test_common_rules-eaab5e16ac26a0de.xml (deflated 78%) 2024-11-01T17:03:46.0139040Z adding: test/test-reports/python-pytest/distributed._tools.test_mem_tracker/distributed._tools.test_mem_tracker-8ad955ff214daee6.xml (deflated 58%) 2024-11-01T17:03:46.0141237Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-3419ffb67b0f960d.xml (deflated 69%) 2024-11-01T17:03:46.0143244Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_fx/distributed.fsdp.test_fsdp_fx-db9ad7cd22743c02.xml (deflated 37%) 2024-11-01T17:03:46.0145206Z adding: test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-593ee232c0ff07d4.xml (deflated 97%) 2024-11-01T17:03:46.0147151Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-3ca627dbe2cefbeb.xml (deflated 85%) 2024-11-01T17:03:46.0149214Z adding: test/test-reports/python-pytest/distributed._tools.test_runtime_estimator/distributed._tools.test_runtime_estimator-e8edf8202ffba297.xml (deflated 59%) 2024-11-01T17:03:46.0151199Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-2a095d81b88424a8.xml (deflated 76%) 2024-11-01T17:03:46.0153127Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-23824f37c62c5a02.xml (deflated 28%) 2024-11-01T17:03:46.0155040Z adding: test/test-reports/python-pytest/distributed._tensor.test_init/distributed._tensor.test_init-f74299625f53b04e.xml (deflated 72%) 2024-11-01T17:03:46.0157464Z adding: test/test-reports/python-pytest/distributed._tensor.test_convolution_ops/distributed._tensor.test_convolution_ops-53f23882fc3a6591.xml (deflated 51%) 2024-11-01T17:03:46.0159550Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-a236ab3bf22fc2a9.xml (deflated 80%) 2024-11-01T17:03:46.0161754Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_fsdp_optim_state/distributed.checkpoint.test_fsdp_optim_state-0a5661d8bbfb077d.xml (deflated 38%) 2024-11-01T17:03:46.0163662Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-a9e4eece8c3f990d.xml (deflated 60%) 2024-11-01T17:03:46.0165119Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-c5635d7f753fe5a5.xml (deflated 60%) 2024-11-01T17:03:46.0166560Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-1a6b111c3db06ac6.xml (deflated 69%) 2024-11-01T17:03:46.0168148Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-e15bd1e5ce6b5452.xml (deflated 78%) 2024-11-01T17:03:46.0169567Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-74d8d16f16535f99.xml (deflated 74%) 2024-11-01T17:03:46.0170838Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-86ab93214f94d5d5.xml (deflated 76%) 2024-11-01T17:03:46.0172096Z adding: test/test-reports/python-pytest/test.run_test/test.run_test-3960f850d6b81f5b.xml (deflated 64%) 2024-11-01T17:03:46.0199798Z ##[group]Run # Remove any previous usage logs if they exist 2024-11-01T17:03:46.0200534Z # Remove any previous usage logs if they exist 2024-11-01T17:03:46.0201093Z rm -f logs-*.zip 2024-11-01T17:03:46.0201870Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-11-01T17:03:46.0202794Z # so check to see if the file exists first 2024-11-01T17:03:46.0203363Z if [ -f 'usage_log.txt' ]; then 2024-11-01T17:03:46.0203969Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-11-01T17:03:46.0204520Z fi 2024-11-01T17:03:46.0205076Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-11-01T17:03:46.0205992Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-11-01T17:03:46.0206628Z fi 2024-11-01T17:03:46.0212720Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:46.0213282Z env: 2024-11-01T17:03:46.0213566Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:46.0214045Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:46.0214879Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:46.0215917Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T17:03:46.0216639Z ##[endgroup] 2024-11-01T17:03:46.0267220Z adding: usage_log.txt (deflated 92%) 2024-11-01T17:03:46.0307237Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_70eb0ddd4dc317b1_.log (deflated 82%) 2024-11-01T17:03:46.0308910Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_65ff37a4dd8daa37_.log (deflated 55%) 2024-11-01T17:03:46.0310473Z adding: test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_71252e0be2d88ebd_.log (deflated 54%) 2024-11-01T17:03:46.0311944Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_6aee17f4e6b26b99_.log (deflated 63%) 2024-11-01T17:03:46.0313518Z adding: test/test-reports/distributed.checkpoint.test_nested_dict_1.1_34d695b30125bdef_.log (deflated 58%) 2024-11-01T17:03:46.0315128Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_20f1d7add8153903_.log (deflated 55%) 2024-11-01T17:03:46.0317124Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_47dd6aa99b87e5e0_.log (deflated 87%) 2024-11-01T17:03:46.0319124Z adding: test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_8be4a7a397a7c2de_.log (deflated 91%) 2024-11-01T17:03:46.0320618Z adding: test/test-reports/distributed._tools.test_sac_ilp_1.1_4f00f288e91c013f_.log (deflated 62%) 2024-11-01T17:03:46.0322134Z adding: test/test-reports/distributed.pipelining.test_backward_1.1_bb93c9fecac27a7c_.log (deflated 66%) 2024-11-01T17:03:46.0323689Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_6a86bcedc98bbfc6_.log (deflated 54%) 2024-11-01T17:03:46.0325301Z adding: test/test-reports/distributed._tools.test_sac_estimator_1.1_cd373433f0b2e23f_.log (deflated 59%) 2024-11-01T17:03:46.0326878Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_2783f4abb9a4cdb9_.log (deflated 52%) 2024-11-01T17:03:46.0328527Z adding: test/test-reports/distributed.pipelining.test_pipe_1.1_5d21ba1aff0e62d6_.log (deflated 62%) 2024-11-01T17:03:46.0329997Z adding: test/test-reports/distributed._tensor.test_common_rules_1.1_a71396e863f56e29_.log (deflated 81%) 2024-11-01T17:03:46.0331391Z adding: test/test-reports/distributed._tools.test_mem_tracker_1.1_4a7e4b6579a470fe_.log (deflated 62%) 2024-11-01T17:03:46.0332835Z adding: test/test-reports/distributed._composable.test_contract_1.1_6f0a26780690278f_.log (deflated 66%) 2024-11-01T17:03:46.0334193Z adding: test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_ce11af7453bf1296_.log (deflated 53%) 2024-11-01T17:03:46.0335729Z adding: test/test-reports/distributed.test_symmetric_memory_1.1_3fd453bd33285a94_.log (deflated 94%) 2024-11-01T17:03:46.0337953Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_d2b1bb80e177cb50_.log (deflated 88%) 2024-11-01T17:03:46.0339762Z adding: test/test-reports/distributed._tools.test_runtime_estimator_1.1_3c35bc9ba0a4a720_.log (deflated 59%) 2024-11-01T17:03:46.0341115Z adding: test/test-reports/distributed._tensor.test_api_1.1_3e511fb06118dc4b_.log (deflated 78%) 2024-11-01T17:03:46.0342438Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_b29edaca3960e428_.log (deflated 52%) 2024-11-01T17:03:46.0343766Z adding: test/test-reports/distributed._tensor.test_init_1.1_96514d064e56d661_.log (deflated 76%) 2024-11-01T17:03:46.0345124Z adding: test/test-reports/distributed._tensor.test_convolution_ops_1.1_5dba3cc5c7caddfc_.log (deflated 62%) 2024-11-01T17:03:46.0346510Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_482e7bc859a26c1d_.log (deflated 82%) 2024-11-01T17:03:46.0347922Z adding: test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_3997a6debebcc31f_.log (deflated 55%) 2024-11-01T17:03:46.0349246Z adding: test/test-reports/cpp.FileStoreTest_1.1_8088badde510d210_.log (deflated 71%) 2024-11-01T17:03:46.0350477Z adding: test/test-reports/cpp.HashStoreTest_1.1_c48f9e5e76cdc00d_.log (deflated 71%) 2024-11-01T17:03:46.0351569Z adding: test/test-reports/cpp.TCPStoreTest_1.1_1eeea72e704cf4c4_.log (deflated 79%) 2024-11-01T17:03:46.0352712Z adding: test/test-reports/cpp.ProcessGroupGlooTest_1.1_d7b1089643579378_.log (deflated 85%) 2024-11-01T17:03:46.0353920Z adding: test/test-reports/cpp.ProcessGroupNCCLTest_1.1_aced1331873cbd20_.log (deflated 82%) 2024-11-01T17:03:46.0355179Z adding: test/test-reports/cpp.ProcessGroupNCCLErrorsTest_1.1_5d9e9390eca3ddf3_.log (deflated 78%) 2024-11-01T17:03:46.0356644Z adding: test/test-reports/cpp.test_cpp_rpc_1.1_8055b88911d89afd_.log (deflated 78%) 2024-11-01T17:03:46.0383544Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-11-01T17:03:46.0384359Z # Remove any previous debugging artifacts if they exist 2024-11-01T17:03:46.0384967Z rm -f debug-*.zip 2024-11-01T17:03:46.0385740Z if [ -d 'test/debug' ]; then 2024-11-01T17:03:46.0386341Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-11-01T17:03:46.0386875Z fi 2024-11-01T17:03:46.0392785Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:46.0393722Z env: 2024-11-01T17:03:46.0394016Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:46.0394655Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:46.0395525Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:46.0396718Z FILE_SUFFIX: test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505 2024-11-01T17:03:46.0397460Z ##[endgroup] 2024-11-01T17:03:46.0504618Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:03:46.0505087Z with: 2024-11-01T17:03:46.0505374Z s3-bucket: gha-artifacts 2024-11-01T17:03:46.0505811Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:46.0506311Z retention-days: 14 2024-11-01T17:03:46.0506647Z if-no-files-found: warn 2024-11-01T17:03:46.0507008Z path: test-jsons-*.zip 2024-11-01T17:03:46.0507352Z name: artifact 2024-11-01T17:03:46.0507674Z region: us-east-1 2024-11-01T17:03:46.0507971Z env: 2024-11-01T17:03:46.0508230Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:46.0508688Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:46.0509501Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:46.0510211Z ##[endgroup] 2024-11-01T17:03:46.4148231Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:03:46.4148939Z With the provided path, there will be 1 file uploaded 2024-11-01T17:03:46.4149628Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:46.4191252Z Starting upload of test-jsons-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:46.5350485Z Finished upload of test-jsons-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:46.5528028Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:03:46.5528481Z with: 2024-11-01T17:03:46.5528739Z s3-bucket: gha-artifacts 2024-11-01T17:03:46.5529180Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:46.5529659Z retention-days: 14 2024-11-01T17:03:46.5529989Z if-no-files-found: error 2024-11-01T17:03:46.5530351Z path: test-reports-*.zip 2024-11-01T17:03:46.5530691Z name: artifact 2024-11-01T17:03:46.5530985Z region: us-east-1 2024-11-01T17:03:46.5531277Z env: 2024-11-01T17:03:46.5531540Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:46.5531991Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:46.5532766Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:46.5533431Z ##[endgroup] 2024-11-01T17:03:46.9161901Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:03:46.9162607Z With the provided path, there will be 1 file uploaded 2024-11-01T17:03:46.9163337Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:46.9205579Z Starting upload of test-reports-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:47.0717649Z Finished upload of test-reports-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:47.0898339Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:03:47.0898804Z with: 2024-11-01T17:03:47.0899062Z s3-bucket: gha-artifacts 2024-11-01T17:03:47.0899499Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:47.0899981Z retention-days: 14 2024-11-01T17:03:47.0900312Z if-no-files-found: ignore 2024-11-01T17:03:47.0900666Z path: logs-*.zip 2024-11-01T17:03:47.0900951Z name: artifact 2024-11-01T17:03:47.0901244Z region: us-east-1 2024-11-01T17:03:47.0901548Z env: 2024-11-01T17:03:47.0901808Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:47.0902259Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:47.0903042Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:47.0903710Z ##[endgroup] 2024-11-01T17:03:47.4527787Z NOTE: s3-prefix specified, ignoring name parameter 2024-11-01T17:03:47.4528778Z With the provided path, there will be 1 file uploaded 2024-11-01T17:03:47.4529482Z Uploading to s3 prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:47.4571305Z Starting upload of logs-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:47.5819827Z Finished upload of logs-test-distributed-1-3-linux.g4dn.12xlarge.nvidia.gpu_32397034505.zip 2024-11-01T17:03:47.6505540Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-11-01T17:03:47.6506021Z with: 2024-11-01T17:03:47.6506294Z s3-bucket: gha-artifacts 2024-11-01T17:03:47.6506760Z s3-prefix: pytorch/pytorch/11632514903/1/artifact 2024-11-01T17:03:47.6507286Z retention-days: 14 2024-11-01T17:03:47.6507631Z if-no-files-found: ignore 2024-11-01T17:03:47.6508003Z path: debug-*.zip 2024-11-01T17:03:47.6508307Z name: artifact 2024-11-01T17:03:47.6508617Z region: us-east-1 2024-11-01T17:03:47.6508926Z env: 2024-11-01T17:03:47.6509198Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:47.6509675Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:47.6510494Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:47.6511207Z ##[endgroup] 2024-11-01T17:03:48.0107298Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-11-01T17:03:48.0286620Z ##[group]Run # shellcheck disable=SC2156 2024-11-01T17:03:48.0287159Z # shellcheck disable=SC2156 2024-11-01T17:03:48.0288213Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-11-01T17:03:48.0295121Z shell: /usr/bin/bash -e {0} 2024-11-01T17:03:48.0295500Z env: 2024-11-01T17:03:48.0295789Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:48.0296287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:48.0297252Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:48.0298134Z ##[endgroup] 2024-11-01T17:03:48.2727659Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-11-01T17:03:48.2728394Z with: 2024-11-01T17:03:48.2728771Z env: 2024-11-01T17:03:48.2729049Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:48.2729535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:48.2730365Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:48.2731085Z ##[endgroup] 2024-11-01T17:03:48.2753429Z ##[group]Run set -eou pipefail 2024-11-01T17:03:48.2753859Z set -eou pipefail 2024-11-01T17:03:48.2754219Z  2024-11-01T17:03:48.2754779Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-11-01T17:03:48.2755671Z for _ in $(seq 1440); do 2024-11-01T17:03:48.2756459Z  # Break if no ssh session exists anymore 2024-11-01T17:03:48.2757009Z  if [ "$(who)" = "" ]; then 2024-11-01T17:03:48.2757461Z  break 2024-11-01T17:03:48.2757850Z  fi 2024-11-01T17:03:48.2758172Z  echo "." 2024-11-01T17:03:48.2758514Z  sleep 5 2024-11-01T17:03:48.2758855Z done 2024-11-01T17:03:48.2765094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:48.2765669Z env: 2024-11-01T17:03:48.2765965Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:48.2766476Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:48.2767336Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:48.2768428Z ##[endgroup] 2024-11-01T17:03:48.2792394Z Holding runner for 2 hours until all ssh sessions have logged out 2024-11-01T17:03:48.2847284Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:03:48.2848183Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:03:48.2848935Z # shellcheck disable=SC2046 2024-11-01T17:03:48.2849406Z docker stop $(docker ps -q) || true 2024-11-01T17:03:48.2849890Z # Prune all of the docker images 2024-11-01T17:03:48.2850344Z docker system prune -af 2024-11-01T17:03:48.2855901Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:48.2856404Z env: 2024-11-01T17:03:48.2856674Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:03:48.2857128Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-11-01T17:03:48.2857905Z DOCKER_CONTAINER_ID: f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:48.2858589Z ##[endgroup] 2024-11-01T17:03:48.8077910Z f17f4030e8f4 2024-11-01T17:03:49.2089046Z Deleted Containers: 2024-11-01T17:03:49.2089852Z f17f4030e8f4940d070e9b7a3d3c7298fed833ab72060fd1080007beb03a990d 2024-11-01T17:03:49.2090364Z 2024-11-01T17:03:53.9946591Z Deleted Images: 2024-11-01T17:03:53.9949459Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T17:03:53.9952358Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:7e55ca35c7b3a515f97404dcaef8e9d5373f6e358ac63c424d2d80dda1e618f7 2024-11-01T17:03:53.9954078Z deleted: sha256:f4475432de42c25077a2ffe0f2f56efb42124dedca9504f01daf084f9103394b 2024-11-01T17:03:53.9955092Z deleted: sha256:0a3b02914acc05dcfc3fdaea37e1a4336aaa696c207e6e993e87815010457060 2024-11-01T17:03:53.9956186Z deleted: sha256:287a6b6bfe47fa7d50e7409829056a1cf37ad073cb6890163e779734ff296fb2 2024-11-01T17:03:53.9957378Z deleted: sha256:53fa61dc822c6716c85f6363bb6a83cd7541c8879d5f9b93dea1064af2d6463c 2024-11-01T17:03:53.9958717Z deleted: sha256:1a8f08fa472d2fb692f63de804eaaa893862346178152979127e0609eb12d696 2024-11-01T17:03:53.9959729Z deleted: sha256:a1b5599bc63b031b0d8a037d7dd06b8878dc70ed0692585d42de476263f63af3 2024-11-01T17:03:53.9960739Z deleted: sha256:f8784943057bdd6a3eec2f0816741a058296e19c92a8ec86595e225509b9d244 2024-11-01T17:03:53.9961893Z deleted: sha256:f62e2601c7c54fd6b496f2b9162da63bc6078f0b98bcc590c8438df3191843b0 2024-11-01T17:03:53.9963059Z deleted: sha256:429b19ab4950a882a2573ce19266e5e84f52dd115bf27e8a23188bae420b5ec2 2024-11-01T17:03:53.9964052Z deleted: sha256:2a955783b858a5e071968bf711c0c32f0932a3ca65a4421074551c69d0bbf98f 2024-11-01T17:03:53.9965079Z deleted: sha256:de6fbaec1030affac12cf8d0d4a3728e048f86288dd421ff999b7f3ee058f0b6 2024-11-01T17:03:53.9966102Z deleted: sha256:381f67e4868ab208466a6d548b250ab4f4a05944a6b27ff3bbee4afef20473ca 2024-11-01T17:03:53.9967114Z deleted: sha256:8c153bbd827867cdf94b1c4237be87df7196b4b4b2043e4eb08fffa5af35245d 2024-11-01T17:03:53.9968219Z deleted: sha256:0e72c2f47638416e5761d4a2b6723d0c2176a3232a9d65650d6f2774c1b1a7bb 2024-11-01T17:03:53.9969196Z deleted: sha256:1af9aaf89749d8b264269a0da14092f9bb6ec46db7db6ab6c9edb2b7eacc1e36 2024-11-01T17:03:53.9970211Z deleted: sha256:b7fe349ffe9a77b7cd8df359c0d4f447eab8c3c5fe1d1f8474d5421fae0b52e7 2024-11-01T17:03:53.9971253Z deleted: sha256:89d4a9d4808254c489a6774ebe1a7fbe7a921a48e2867b0370db5ae6643ef591 2024-11-01T17:03:53.9972239Z deleted: sha256:a3e0a41d4550f3243e4b20bdb204324bcbcea605d9ecd04e8f3f1af843f57e86 2024-11-01T17:03:53.9973232Z deleted: sha256:019e53db4314f4ba6148551c897d2d50cdceda1a96a4f03554208af0e99de4b8 2024-11-01T17:03:53.9974194Z deleted: sha256:383583432ea93d7b02c0844ca27319cab73bfc40c49d3f0e1e04295ccbcd9662 2024-11-01T17:03:53.9975190Z deleted: sha256:af4dadd55bfef20fdd6e7e3ce936c5930b3a4030c6dd380e41c57ac57fa6d351 2024-11-01T17:03:53.9976204Z deleted: sha256:df578b0221ae1fdd0a5b9b02c806ab3ee607208c6c0e8b75876caff585bc6177 2024-11-01T17:03:53.9977195Z deleted: sha256:29baf54d9e67bc5615a74704ba363598e29ae1314b34fa4ca5c843f2b5fb1265 2024-11-01T17:03:53.9978347Z deleted: sha256:7b401ef5b5879bdc37e726876c721fbbbe5f4f7721c236560a37cccb8f8ddb32 2024-11-01T17:03:53.9979339Z deleted: sha256:3c65fa1a83a01d453087f56b23a9cdda9218e1cdfe05500cbfcb1e753a3ab978 2024-11-01T17:03:53.9980320Z deleted: sha256:008280bac2ca227cafab11733314d6f662de2f70bb060483e181e5634e8a347f 2024-11-01T17:03:53.9981295Z deleted: sha256:5d4a60ee9e3f8449e187468813913fda8f5408c95b59600b5508ac77126c045e 2024-11-01T17:03:53.9982260Z deleted: sha256:f0b9826ef432a31e2e43af6d32313a2a4ae3557a3636f010eb150f5fb5dfbbff 2024-11-01T17:03:53.9983210Z deleted: sha256:7e4497566413d4a3599e7034506c24a6ab96193b4647a7c214f50295649f5291 2024-11-01T17:03:53.9984158Z deleted: sha256:930abad7c169e40236fa09a9608a0ba7e8e9a966686a6133b78bd660bdfb9a59 2024-11-01T17:03:53.9985312Z deleted: sha256:77f2520779046086794a6a4f24626f4f688c329e9cee7de2d22254477299c457 2024-11-01T17:03:53.9986461Z deleted: sha256:67e6ef1843c893be2880063dee3735bf00e40894ef866be07f61b8d59cf3755a 2024-11-01T17:03:53.9987451Z deleted: sha256:2ddf875758f00a6403c9e203e755cc185eca0f9e423849e5700fdb076dfc562d 2024-11-01T17:03:53.9988470Z deleted: sha256:68d0c60c82c5f5afa713dd4784be9baa1f588998b8e108ce0ece68f6e976d122 2024-11-01T17:03:53.9989491Z deleted: sha256:c3d898e584b844147299b6b3eaf2ea2cb63b4e93f3640a7b52baa64e770bb5f3 2024-11-01T17:03:53.9990504Z deleted: sha256:b46bb13fe8c4577e03ee179b7042226178e6cd61eaeb80445a16d25b1bb1f35f 2024-11-01T17:03:53.9991515Z deleted: sha256:42cc15f51fb1309300408cfaf0056e28b11de1a0f6dc033c99a943473619e1fb 2024-11-01T17:03:53.9992513Z deleted: sha256:4418b6a425adc3a7406a74e6b47bd7f3788ad275193db6716e23729b892ce0a8 2024-11-01T17:03:53.9993506Z deleted: sha256:7c58b3261a79180e66c7932cc232845b4dbdaffc908c0f0a2a2cde84950ad553 2024-11-01T17:03:53.9994537Z deleted: sha256:ea3f9a836caadfa9bd915005c979b0febc5334ae0a607daae43e85f77cc72474 2024-11-01T17:03:53.9995581Z deleted: sha256:decc9ca252ac120ee1858488ad90a3f707afd34238019bd1c51e97b4d09c45f6 2024-11-01T17:03:53.9996825Z deleted: sha256:e033e0e1b98c30aaad1bf6c6c2434254968b11262e9c6e46b56eb4ebdb74ab51 2024-11-01T17:03:53.9997845Z deleted: sha256:f8a7f2c4c34a3c04a982d753851c220e47a6f1049b57f2a42b4144d0b6a17b9e 2024-11-01T17:03:53.9998851Z deleted: sha256:88766d7f595185d3bc2ff7a2476931bda2cdac996c0d13db9a95b301b2db75a9 2024-11-01T17:03:53.9999988Z deleted: sha256:5bf533e94db3a248573dcbfdf6a58089e558f5f0f912277788bce3b54545d01c 2024-11-01T17:03:54.0001009Z deleted: sha256:8d77e0f9c2de0b3a7b5bdc2992ad1034b843b2eb67d9f262f43160f55884bab8 2024-11-01T17:03:54.0002036Z deleted: sha256:54bdbdf9f9e33647140f0c0f8c1dc00c526479f42964765338b8f79a1049ee9c 2024-11-01T17:03:54.0003038Z deleted: sha256:2359a12b67991edef98665db87a5192ba02e183ee2232e283e6b58c94f8d3826 2024-11-01T17:03:54.0004032Z deleted: sha256:7727c4d3952195e6aaf87b225a3f67b89642a424451e601618fa1e7b52325cd7 2024-11-01T17:03:54.0005020Z deleted: sha256:6f9f382dd9bcb89f97553031ee64253426902b4910f2556a2bea7881309c376c 2024-11-01T17:03:54.0006013Z deleted: sha256:c35ad5f713dc248c630d50c7520cd35ba1ab936116cf355e419c3fbe74909a04 2024-11-01T17:03:54.0007024Z deleted: sha256:4a5efa02522a748ca3722e8b7c0142d4d2291584b8d195e63da582862a967528 2024-11-01T17:03:54.0008218Z deleted: sha256:a30107d370bc5fa598ef737c0c53419acc51af7ab8ef64d75696fae37670abcb 2024-11-01T17:03:54.0009126Z deleted: sha256:672ec49fd713d9e0cdfb23310a314ebf467b0a33d23f64599816ca4684cb57f5 2024-11-01T17:03:54.0010032Z deleted: sha256:b4de36d9c64ae16e75798019674dcb2fade5e6653e5095c2dfd2846384786772 2024-11-01T17:03:54.0010944Z deleted: sha256:20d847bb7af881db42903f7d4166e9e372334ebdfb3d1f34b8ef669623d0da58 2024-11-01T17:03:54.0011836Z deleted: sha256:c34468395058a30a439e859a7bd5133e780faa762a11e025e0fdeb88627550e4 2024-11-01T17:03:54.0012708Z deleted: sha256:27f5680f66a0d01ec1d74ae03418fcb2e03db96605d55e0d8933a40acf0def6c 2024-11-01T17:03:54.0013613Z deleted: sha256:9a974b67ee0b37916f8a38a233be94f2fa8205ec477fdee5d06905af1cb34a9c 2024-11-01T17:03:54.0014511Z deleted: sha256:a7e98e6ff938d43862e03ca9edb22288389b8a503b60d8b72b26562de329e745 2024-11-01T17:03:54.0015422Z deleted: sha256:f8dfc84a271d94297e3a777f3455db814bebc4c56d31dc2b589320a70c5b3e01 2024-11-01T17:03:54.0016334Z deleted: sha256:cb9ee3913530d2683b184f99aaeaafade4ee059d8d83ca203fe890c9b71a39ce 2024-11-01T17:03:54.0017249Z deleted: sha256:cf8424e79315a3b2ffb71729abf912d62e257dfe818f3c47b09812733723a948 2024-11-01T17:03:54.0018142Z deleted: sha256:22de19c76bb02aa83af513603ea7b11c598a75ea3d120371286253e6ce67f0f0 2024-11-01T17:03:54.0019045Z deleted: sha256:e3bb91f1c5e44b1d35320ec7e5220af244c47d23a888f50508efbb919f0b7b29 2024-11-01T17:03:54.0019943Z deleted: sha256:08c5645ed9673839aa7a6eae9119ab5e407ecce85b3b5466607c5a651c3f2920 2024-11-01T17:03:54.0020864Z deleted: sha256:32c37789f4e3eaa19d0b7d08bfd1b4a1fed982cbeedccde408a3e3e2afaa1f9e 2024-11-01T17:03:54.0021788Z deleted: sha256:8b0aee0f411b6479db05dfa28910475b3cbbb2e08c3b048b78a43939c48c7419 2024-11-01T17:03:54.0022705Z deleted: sha256:2f7d309060abdcfff184b9086398b192de12a03af2a8e018c807cc6fa89ec403 2024-11-01T17:03:54.0023606Z deleted: sha256:fa738ffc39690ea6d3e51036aaa9acff34100a51d635e157817603eb79dee259 2024-11-01T17:03:54.0024511Z deleted: sha256:07a2164b82cf81c6f9f6fc6859a6d498203b5e0fb9ecd9878bf00de3c4c9949d 2024-11-01T17:03:54.0025405Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-11-01T17:03:54.0026322Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-11-01T17:03:54.0027227Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-11-01T17:03:54.0028116Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-11-01T17:03:54.0029023Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-11-01T17:03:54.0029931Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-11-01T17:03:54.0030846Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-11-01T17:03:54.0031801Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-11-01T17:03:54.0032703Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-11-01T17:03:54.0033601Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-11-01T17:03:54.0034545Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-11-01T17:03:54.0035084Z 2024-11-01T17:03:54.0035215Z Total reclaimed space: 34.51GB 2024-11-01T17:03:54.0164774Z Post job cleanup. 2024-11-01T17:03:54.0225751Z Post job cleanup. 2024-11-01T17:03:54.1141702Z [command]/usr/bin/git version 2024-11-01T17:03:54.1187724Z git version 2.40.1 2024-11-01T17:03:54.1233286Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/486f7128-b039-4788-a125-6ed087ad1849' before making global git config changes 2024-11-01T17:03:54.1234772Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T17:03:54.1236712Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-11-01T17:03:54.1276136Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T17:03:54.1311616Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T17:03:54.1630751Z Entering 'android/libs/fbjni' 2024-11-01T17:03:54.1687125Z Entering 'third_party/FP16' 2024-11-01T17:03:54.1743421Z Entering 'third_party/FXdiv' 2024-11-01T17:03:54.1798279Z Entering 'third_party/NNPACK' 2024-11-01T17:03:54.1856392Z Entering 'third_party/NVTX' 2024-11-01T17:03:54.1913524Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:03:54.1970956Z Entering 'third_party/XNNPACK' 2024-11-01T17:03:54.2051021Z Entering 'third_party/benchmark' 2024-11-01T17:03:54.2107133Z Entering 'third_party/composable_kernel' 2024-11-01T17:03:54.2168403Z Entering 'third_party/cpp-httplib' 2024-11-01T17:03:54.2226012Z Entering 'third_party/cpuinfo' 2024-11-01T17:03:54.2281027Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:03:54.2336963Z Entering 'third_party/cutlass' 2024-11-01T17:03:54.2404459Z Entering 'third_party/eigen' 2024-11-01T17:03:54.2463640Z Entering 'third_party/fbgemm' 2024-11-01T17:03:54.2519248Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:03:54.2576467Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:03:54.2633846Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:03:54.2697485Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:03:54.2758222Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:03:54.2817734Z Entering 'third_party/flatbuffers' 2024-11-01T17:03:54.2878250Z Entering 'third_party/fmt' 2024-11-01T17:03:54.2939208Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:03:54.2995601Z Entering 'third_party/gloo' 2024-11-01T17:03:54.3056029Z Entering 'third_party/googletest' 2024-11-01T17:03:54.3112655Z Entering 'third_party/ideep' 2024-11-01T17:03:54.3167684Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:03:54.3234939Z Entering 'third_party/ittapi' 2024-11-01T17:03:54.3293347Z Entering 'third_party/kineto' 2024-11-01T17:03:54.3350819Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:03:54.3406467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:03:54.3466193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:03:54.3527256Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:03:54.3584878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:03:54.3641337Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:03:54.3701381Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:03:54.3758832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:03:54.3842290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:03:54.3878009Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:03:54.3938189Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:03:54.3996335Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:03:54.4056613Z Entering 'third_party/mimalloc' 2024-11-01T17:03:54.4114126Z Entering 'third_party/nccl/nccl' 2024-11-01T17:03:54.4173026Z Entering 'third_party/nlohmann' 2024-11-01T17:03:54.4234890Z Entering 'third_party/onnx' 2024-11-01T17:03:54.4311924Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:03:54.4373622Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:03:54.4435265Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:03:54.4493140Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:03:54.4551226Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:03:54.4606730Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:03:54.4665501Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:03:54.4722562Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:03:54.4779774Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:03:54.4835602Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:03:54.4895365Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:03:54.4953837Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:03:54.5033611Z Entering 'third_party/pocketfft' 2024-11-01T17:03:54.5088769Z Entering 'third_party/protobuf' 2024-11-01T17:03:54.5150089Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:03:54.5204459Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:03:54.5262812Z Entering 'third_party/psimd' 2024-11-01T17:03:54.5319625Z Entering 'third_party/pthreadpool' 2024-11-01T17:03:54.5379490Z Entering 'third_party/pybind11' 2024-11-01T17:03:54.5437663Z Entering 'third_party/python-peachpy' 2024-11-01T17:03:54.5492030Z Entering 'third_party/sleef' 2024-11-01T17:03:54.5550152Z Entering 'third_party/tensorpipe' 2024-11-01T17:03:54.5604669Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:03:54.5659630Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:03:54.5714624Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:03:54.5773322Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:03:54.5832507Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:03:54.5909369Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T17:03:54.5939376Z http.https://github.com/.extraheader 2024-11-01T17:03:54.5947494Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-11-01T17:03:54.5984276Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T17:03:54.6301578Z Entering 'android/libs/fbjni' 2024-11-01T17:03:54.6339926Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6375737Z Entering 'third_party/FP16' 2024-11-01T17:03:54.6415140Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6452739Z Entering 'third_party/FXdiv' 2024-11-01T17:03:54.6488634Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6524004Z Entering 'third_party/NNPACK' 2024-11-01T17:03:54.6560316Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6604211Z Entering 'third_party/NVTX' 2024-11-01T17:03:54.6640284Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6676725Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:03:54.6715673Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6753342Z Entering 'third_party/XNNPACK' 2024-11-01T17:03:54.6790054Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6840028Z Entering 'third_party/benchmark' 2024-11-01T17:03:54.6877300Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6912456Z Entering 'third_party/composable_kernel' 2024-11-01T17:03:54.6950940Z http.https://github.com/.extraheader 2024-11-01T17:03:54.6991380Z Entering 'third_party/cpp-httplib' 2024-11-01T17:03:54.7031656Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7067652Z Entering 'third_party/cpuinfo' 2024-11-01T17:03:54.7106023Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7141463Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:03:54.7179250Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7217446Z Entering 'third_party/cutlass' 2024-11-01T17:03:54.7255480Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7307831Z Entering 'third_party/eigen' 2024-11-01T17:03:54.7346165Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7385032Z Entering 'third_party/fbgemm' 2024-11-01T17:03:54.7424140Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7460083Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:03:54.7496316Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7536858Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:03:54.7573854Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7606893Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:03:54.7645748Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7691894Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:03:54.7729001Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7765519Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:03:54.7802765Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7844668Z Entering 'third_party/flatbuffers' 2024-11-01T17:03:54.7881246Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7919170Z Entering 'third_party/fmt' 2024-11-01T17:03:54.7957392Z http.https://github.com/.extraheader 2024-11-01T17:03:54.7995003Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:03:54.8032549Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8070436Z Entering 'third_party/gloo' 2024-11-01T17:03:54.8109980Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8144830Z Entering 'third_party/googletest' 2024-11-01T17:03:54.8183742Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8220134Z Entering 'third_party/ideep' 2024-11-01T17:03:54.8257418Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8296456Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:03:54.8333912Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8379134Z Entering 'third_party/ittapi' 2024-11-01T17:03:54.8417641Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8455075Z Entering 'third_party/kineto' 2024-11-01T17:03:54.8490575Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8526249Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:03:54.8561693Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8598373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:03:54.8635257Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8678026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:03:54.8716547Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8753843Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:03:54.8790750Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8827829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:03:54.8865126Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8900400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:03:54.8937608Z http.https://github.com/.extraheader 2024-11-01T17:03:54.8977048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:03:54.9015696Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9053654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:03:54.9089732Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9136815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:03:54.9173071Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9209841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:03:54.9245909Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9286562Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:03:54.9322961Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9364502Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:03:54.9401185Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9439547Z Entering 'third_party/mimalloc' 2024-11-01T17:03:54.9478063Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9516767Z Entering 'third_party/nccl/nccl' 2024-11-01T17:03:54.9554922Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9590433Z Entering 'third_party/nlohmann' 2024-11-01T17:03:54.9629567Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9665984Z Entering 'third_party/onnx' 2024-11-01T17:03:54.9704518Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9756323Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:03:54.9793781Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9837887Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:03:54.9875538Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9914033Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:03:54.9952665Z http.https://github.com/.extraheader 2024-11-01T17:03:54.9995921Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:03:55.0033451Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0074864Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:03:55.0113261Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0152775Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:03:55.0188821Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0226203Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:03:55.0263611Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0297032Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:03:55.0335155Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0372129Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:03:55.0409642Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0452988Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:03:55.0489318Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0525621Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:03:55.0561458Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0600916Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:03:55.0637822Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0699975Z Entering 'third_party/pocketfft' 2024-11-01T17:03:55.0737211Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0773156Z Entering 'third_party/protobuf' 2024-11-01T17:03:55.0812887Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0854581Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:03:55.0890155Z http.https://github.com/.extraheader 2024-11-01T17:03:55.0935345Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:03:55.0972052Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1007785Z Entering 'third_party/psimd' 2024-11-01T17:03:55.1045534Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1083888Z Entering 'third_party/pthreadpool' 2024-11-01T17:03:55.1123375Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1163885Z Entering 'third_party/pybind11' 2024-11-01T17:03:55.1201990Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1237725Z Entering 'third_party/python-peachpy' 2024-11-01T17:03:55.1275588Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1311234Z Entering 'third_party/sleef' 2024-11-01T17:03:55.1348622Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1383842Z Entering 'third_party/tensorpipe' 2024-11-01T17:03:55.1422411Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1458205Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:03:55.1493860Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1537239Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:03:55.1574197Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1607150Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:03:55.1645784Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1680499Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:03:55.1718251Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1753191Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:03:55.1788795Z http.https://github.com/.extraheader 2024-11-01T17:03:55.1914743Z A job completed hook has been configured by the self-hosted runner administrator 2024-11-01T17:03:55.1936330Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-11-01T17:03:55.1942097Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:03:55.1942763Z ##[endgroup] 2024-11-01T17:04:01.8837955Z Cleaning up orphan processes