2024-08-20T22:07:31.0890117Z Current runner version: '2.319.1' 2024-08-20T22:07:31.0896652Z Runner name: 'i-01a7d20fbad5d17a0' 2024-08-20T22:07:31.0897433Z Runner group name: 'Default' 2024-08-20T22:07:31.0898297Z Machine name: 'ip-10-0-37-127' 2024-08-20T22:07:31.0914669Z Testing runner upgrade compatibility 2024-08-20T22:07:31.2161485Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:07:31.2163475Z Actions: read 2024-08-20T22:07:31.2164135Z Attestations: read 2024-08-20T22:07:31.2164659Z Checks: read 2024-08-20T22:07:31.2165116Z Contents: read 2024-08-20T22:07:31.2165792Z Deployments: read 2024-08-20T22:07:31.2166325Z Discussions: read 2024-08-20T22:07:31.2166825Z Issues: read 2024-08-20T22:07:31.2167365Z Metadata: read 2024-08-20T22:07:31.2167865Z Packages: read 2024-08-20T22:07:31.2168440Z Pages: read 2024-08-20T22:07:31.2168889Z PullRequests: read 2024-08-20T22:07:31.2169449Z RepositoryProjects: read 2024-08-20T22:07:31.2170127Z SecurityEvents: read 2024-08-20T22:07:31.2170673Z Statuses: read 2024-08-20T22:07:31.2171323Z ##[endgroup] 2024-08-20T22:07:31.2174619Z Secret source: Actions 2024-08-20T22:07:31.2175478Z Prepare workflow directory 2024-08-20T22:07:31.7474963Z Prepare all required actions 2024-08-20T22:07:31.7658213Z Getting action download info 2024-08-20T22:07:31.9432578Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:07:32.4728842Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:07:36.4563096Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:07:36.6422967Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:07:36.8754656Z Getting action download info 2024-08-20T22:07:36.9805868Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:07:37.2214463Z Getting action download info 2024-08-20T22:07:37.3201140Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:07:37.4828866Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:07:37.4830993Z ##[group] Inputs 2024-08-20T22:07:37.4831466Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:07:37.4840874Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:07:37.4851040Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:07:37.4852218Z sync-tag: 2024-08-20T22:07:37.4853199Z timeout-minutes: 240 2024-08-20T22:07:37.4853541Z use-gha: 2024-08-20T22:07:37.4853830Z dashboard-tag: 2024-08-20T22:07:37.4854149Z s3-bucket: gha-artifacts 2024-08-20T22:07:37.4854505Z aws-role-to-assume: 2024-08-20T22:07:37.4854837Z ##[endgroup] 2024-08-20T22:07:37.4855747Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:07:37.5748169Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:07:37.5888672Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:07:37.5900316Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:07:37.5900909Z ##[endgroup] 2024-08-20T22:07:39.4379843Z Runner Type: amz2023.linux.g5.12xlarge.nvidia.gpu 2024-08-20T22:07:39.4380386Z Instance Type: g5.12xlarge 2024-08-20T22:07:39.4381205Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:07:39.4381770Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:07:45.3522396Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:07:45.3522984Z with: 2024-08-20T22:07:45.3523704Z github-secret: *** 2024-08-20T22:07:45.3524618Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:07:45.3525588Z activate-with-label: false 2024-08-20T22:07:45.3525948Z label: with-ssh 2024-08-20T22:07:45.3526275Z remove-existing-keys: true 2024-08-20T22:07:45.3526644Z fail-silently: true 2024-08-20T22:07:45.3526949Z env: 2024-08-20T22:07:45.3527229Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:07:45.3527572Z ##[endgroup] 2024-08-20T22:07:45.4433792Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:07:45.4436454Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:07:45.7517742Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:07:45.8206383Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:07:45.8206992Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:07:45.9137694Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:07:45.9151204Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:07:45.9176814Z Login using: ssh ec2-user@ec2-44-201-112-27.compute-1.amazonaws.com 2024-08-20T22:07:45.9177656Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:07:45.9178545Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:07:45.9313392Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:07:45.9313936Z with: 2024-08-20T22:07:45.9314210Z submodules: recursive 2024-08-20T22:07:45.9314531Z fetch-depth: 0 2024-08-20T22:07:45.9314817Z env: 2024-08-20T22:07:45.9315076Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:07:45.9315400Z ##[endgroup] 2024-08-20T22:07:45.9517125Z ##[group]Run retry () { 2024-08-20T22:07:45.9517509Z retry () { 2024-08-20T22:07:45.9518546Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:07:45.9519100Z } 2024-08-20T22:07:45.9519401Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:07:45.9519819Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:07:45.9520283Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:07:45.9520736Z else 2024-08-20T22:07:45.9521070Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:07:45.9521489Z fi 2024-08-20T22:07:45.9521828Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:07:45.9532992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:07:45.9533502Z env: 2024-08-20T22:07:45.9533781Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:07:45.9534115Z NO_SUDO: 2024-08-20T22:07:45.9534390Z ##[endgroup] 2024-08-20T22:07:45.9565805Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:07:50.6668292Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:07:50.6668739Z with: 2024-08-20T22:07:50.6669078Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:07:50.6669515Z fetch-depth: 0 2024-08-20T22:07:50.6669824Z submodules: recursive 2024-08-20T22:07:50.6670158Z quiet-checkout: true 2024-08-20T22:07:50.6670504Z repository: pytorch/pytorch 2024-08-20T22:07:50.6670991Z token: *** 2024-08-20T22:07:50.6671281Z ssh-strict: true 2024-08-20T22:07:50.6671605Z persist-credentials: true 2024-08-20T22:07:50.6671962Z clean: true 2024-08-20T22:07:50.6672296Z sparse-checkout-cone-mode: true 2024-08-20T22:07:50.6672681Z lfs: false 2024-08-20T22:07:50.6672975Z set-safe-directory: true 2024-08-20T22:07:50.6673309Z env: 2024-08-20T22:07:50.6673584Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:07:50.6673909Z ##[endgroup] 2024-08-20T22:07:50.7651600Z Syncing repository: pytorch/pytorch 2024-08-20T22:07:50.7653044Z ##[group]Getting Git version info 2024-08-20T22:07:50.7653765Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:07:50.7654622Z [command]/usr/bin/git version 2024-08-20T22:07:50.7654978Z git version 2.40.1 2024-08-20T22:07:50.7665025Z ##[endgroup] 2024-08-20T22:07:50.7681308Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/18c03981-d4c2-4ea2-826f-ac31d60aea0c' before making global git config changes 2024-08-20T22:07:50.7682532Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:07:50.7684010Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:07:50.7738273Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:07:50.7742288Z ##[group]Initializing the repository 2024-08-20T22:07:50.7745039Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:07:50.7793777Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:07:50.7794769Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:07:50.7795567Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:07:50.7796141Z hint: 2024-08-20T22:07:50.7796624Z hint: git config --global init.defaultBranch 2024-08-20T22:07:50.7797110Z hint: 2024-08-20T22:07:50.7797629Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:07:50.7798483Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:07:50.7800864Z hint: 2024-08-20T22:07:50.7801223Z hint: git branch -m 2024-08-20T22:07:50.7801983Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:07:50.7806185Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:07:50.7854170Z ##[endgroup] 2024-08-20T22:07:50.7855203Z ##[group]Disabling automatic garbage collection 2024-08-20T22:07:50.7856683Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:07:50.7899893Z ##[endgroup] 2024-08-20T22:07:50.7900414Z ##[group]Setting up auth 2024-08-20T22:07:50.7905762Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:07:50.7953355Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:07:50.8339303Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:07:50.8384271Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:07:50.8763754Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:07:50.8821456Z ##[endgroup] 2024-08-20T22:07:50.8822033Z ##[group]Fetching the repository 2024-08-20T22:07:50.8827634Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:07:53.6539333Z remote: Enumerating objects: 1008452 2024-08-20T22:07:53.6539881Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:07:53.6541690Z remote: Counting objects: 0% (1/1756) 2024-08-20T22:07:53.6542205Z remote: Counting objects: 1% (18/1756) 2024-08-20T22:07:53.6542727Z remote: Counting objects: 2% (36/1756) 2024-08-20T22:07:53.6543695Z remote: Counting objects: 3% (53/1756) 2024-08-20T22:07:53.6544195Z remote: Counting objects: 4% (71/1756) 2024-08-20T22:07:53.6545963Z remote: Counting objects: 5% (88/1756) 2024-08-20T22:07:53.6547095Z remote: Counting objects: 6% (106/1756) 2024-08-20T22:07:53.6548114Z remote: Counting objects: 7% (123/1756) 2024-08-20T22:07:53.6549169Z remote: Counting objects: 8% (141/1756) 2024-08-20T22:07:53.6550171Z remote: Counting objects: 9% (159/1756) 2024-08-20T22:07:53.6551180Z remote: Counting objects: 10% (176/1756) 2024-08-20T22:07:53.6551786Z remote: Counting objects: 11% (194/1756) 2024-08-20T22:07:53.6552292Z remote: Counting objects: 12% (211/1756) 2024-08-20T22:07:53.6552795Z remote: Counting objects: 13% (229/1756) 2024-08-20T22:07:53.6553294Z remote: Counting objects: 14% (246/1756) 2024-08-20T22:07:53.6553798Z remote: Counting objects: 15% (264/1756) 2024-08-20T22:07:53.6554291Z remote: Counting objects: 16% (281/1756) 2024-08-20T22:07:53.6554787Z remote: Counting objects: 17% (299/1756) 2024-08-20T22:07:53.6555273Z remote: Counting objects: 18% (317/1756) 2024-08-20T22:07:53.6555750Z remote: Counting objects: 19% (334/1756) 2024-08-20T22:07:53.6556241Z remote: Counting objects: 20% (352/1756) 2024-08-20T22:07:53.6556758Z remote: Counting objects: 21% (369/1756) 2024-08-20T22:07:53.6557245Z remote: Counting objects: 22% (387/1756) 2024-08-20T22:07:53.6557739Z remote: Counting objects: 23% (404/1756) 2024-08-20T22:07:53.6558228Z remote: Counting objects: 24% (422/1756) 2024-08-20T22:07:53.6558728Z remote: Counting objects: 25% (439/1756) 2024-08-20T22:07:53.6559218Z remote: Counting objects: 26% (457/1756) 2024-08-20T22:07:53.6559706Z remote: Counting objects: 27% (475/1756) 2024-08-20T22:07:53.6560558Z remote: Counting objects: 28% (492/1756) 2024-08-20T22:07:53.6561057Z remote: Counting objects: 29% (510/1756) 2024-08-20T22:07:53.6561550Z remote: Counting objects: 30% (527/1756) 2024-08-20T22:07:53.6562036Z remote: Counting objects: 31% (545/1756) 2024-08-20T22:07:53.6562547Z remote: Counting objects: 32% (562/1756) 2024-08-20T22:07:53.6563040Z remote: Counting objects: 33% (580/1756) 2024-08-20T22:07:53.6563722Z remote: Counting objects: 34% (598/1756) 2024-08-20T22:07:53.6564207Z remote: Counting objects: 35% (615/1756) 2024-08-20T22:07:53.6564700Z remote: Counting objects: 36% (633/1756) 2024-08-20T22:07:53.6565193Z remote: Counting objects: 37% (650/1756) 2024-08-20T22:07:53.6565685Z remote: Counting objects: 38% (668/1756) 2024-08-20T22:07:53.6566169Z remote: Counting objects: 39% (685/1756) 2024-08-20T22:07:53.6566667Z remote: Counting objects: 40% (703/1756) 2024-08-20T22:07:53.6567178Z remote: Counting objects: 41% (720/1756) 2024-08-20T22:07:53.6567674Z remote: Counting objects: 42% (738/1756) 2024-08-20T22:07:53.6568165Z remote: Counting objects: 43% (756/1756) 2024-08-20T22:07:53.6568663Z remote: Counting objects: 44% (773/1756) 2024-08-20T22:07:53.6569159Z remote: Counting objects: 45% (791/1756) 2024-08-20T22:07:53.6569651Z remote: Counting objects: 46% (808/1756) 2024-08-20T22:07:53.6570143Z remote: Counting objects: 47% (826/1756) 2024-08-20T22:07:53.6570643Z remote: Counting objects: 48% (843/1756) 2024-08-20T22:07:53.6571140Z remote: Counting objects: 49% (861/1756) 2024-08-20T22:07:53.6571628Z remote: Counting objects: 50% (878/1756) 2024-08-20T22:07:53.6572161Z remote: Counting objects: 51% (896/1756) 2024-08-20T22:07:53.6572654Z remote: Counting objects: 52% (914/1756) 2024-08-20T22:07:53.6573152Z remote: Counting objects: 53% (931/1756) 2024-08-20T22:07:53.6573657Z remote: Counting objects: 54% (949/1756) 2024-08-20T22:07:53.6574159Z remote: Counting objects: 55% (966/1756) 2024-08-20T22:07:53.6574644Z remote: Counting objects: 56% (984/1756) 2024-08-20T22:07:53.6575152Z remote: Counting objects: 57% (1001/1756) 2024-08-20T22:07:53.6575672Z remote: Counting objects: 58% (1019/1756) 2024-08-20T22:07:53.6576196Z remote: Counting objects: 59% (1037/1756) 2024-08-20T22:07:53.6576720Z remote: Counting objects: 60% (1054/1756) 2024-08-20T22:07:53.6577233Z remote: Counting objects: 61% (1072/1756) 2024-08-20T22:07:53.6577744Z remote: Counting objects: 62% (1089/1756) 2024-08-20T22:07:53.6578277Z remote: Counting objects: 63% (1107/1756) 2024-08-20T22:07:53.6578801Z remote: Counting objects: 64% (1124/1756) 2024-08-20T22:07:53.6579305Z remote: Counting objects: 65% (1142/1756) 2024-08-20T22:07:53.6579829Z remote: Counting objects: 66% (1159/1756) 2024-08-20T22:07:53.6580341Z remote: Counting objects: 67% (1177/1756) 2024-08-20T22:07:53.6580854Z remote: Counting objects: 68% (1195/1756) 2024-08-20T22:07:53.6581479Z remote: Counting objects: 69% (1212/1756) 2024-08-20T22:07:53.6581988Z remote: Counting objects: 70% (1230/1756) 2024-08-20T22:07:53.6582503Z remote: Counting objects: 71% (1247/1756) 2024-08-20T22:07:53.6583012Z remote: Counting objects: 72% (1265/1756) 2024-08-20T22:07:53.6583522Z remote: Counting objects: 73% (1282/1756) 2024-08-20T22:07:53.6584027Z remote: Counting objects: 74% (1300/1756) 2024-08-20T22:07:53.6584529Z remote: Counting objects: 75% (1317/1756) 2024-08-20T22:07:53.6585032Z remote: Counting objects: 76% (1335/1756) 2024-08-20T22:07:53.6585547Z remote: Counting objects: 77% (1353/1756) 2024-08-20T22:07:53.6586055Z remote: Counting objects: 78% (1370/1756) 2024-08-20T22:07:53.6586661Z remote: Counting objects: 79% (1388/1756) 2024-08-20T22:07:53.6587164Z remote: Counting objects: 80% (1405/1756) 2024-08-20T22:07:53.6587676Z remote: Counting objects: 81% (1423/1756) 2024-08-20T22:07:53.6588180Z remote: Counting objects: 82% (1440/1756) 2024-08-20T22:07:53.6588689Z remote: Counting objects: 83% (1458/1756) 2024-08-20T22:07:53.6589186Z remote: Counting objects: 84% (1476/1756) 2024-08-20T22:07:53.6589708Z remote: Counting objects: 85% (1493/1756) 2024-08-20T22:07:53.6590317Z remote: Counting objects: 86% (1511/1756) 2024-08-20T22:07:53.6590826Z remote: Counting objects: 87% (1528/1756) 2024-08-20T22:07:53.6591322Z remote: Counting objects: 88% (1546/1756) 2024-08-20T22:07:53.6591834Z remote: Counting objects: 89% (1563/1756) 2024-08-20T22:07:53.6592340Z remote: Counting objects: 90% (1581/1756) 2024-08-20T22:07:53.6592843Z remote: Counting objects: 91% (1598/1756) 2024-08-20T22:07:53.6593352Z remote: Counting objects: 92% (1616/1756) 2024-08-20T22:07:53.6593865Z remote: Counting objects: 93% (1634/1756) 2024-08-20T22:07:53.6594385Z remote: Counting objects: 94% (1651/1756) 2024-08-20T22:07:53.6594995Z remote: Counting objects: 95% (1669/1756) 2024-08-20T22:07:53.6595500Z remote: Counting objects: 96% (1686/1756) 2024-08-20T22:07:53.6596009Z remote: Counting objects: 97% (1704/1756) 2024-08-20T22:07:53.6596528Z remote: Counting objects: 98% (1721/1756) 2024-08-20T22:07:53.6597034Z remote: Counting objects: 99% (1739/1756) 2024-08-20T22:07:53.6597533Z remote: Counting objects: 100% (1756/1756) 2024-08-20T22:07:53.6598082Z remote: Counting objects: 100% (1756/1756), done. 2024-08-20T22:07:53.6693016Z remote: Compressing objects: 0% (1/837) 2024-08-20T22:07:53.7034117Z remote: Compressing objects: 1% (9/837) 2024-08-20T22:07:53.7300052Z remote: Compressing objects: 2% (17/837) 2024-08-20T22:07:53.7692913Z remote: Compressing objects: 3% (26/837) 2024-08-20T22:07:53.8393754Z remote: Compressing objects: 4% (34/837) 2024-08-20T22:07:53.8813225Z remote: Compressing objects: 5% (42/837) 2024-08-20T22:07:53.9373351Z remote: Compressing objects: 6% (51/837) 2024-08-20T22:07:53.9677921Z remote: Compressing objects: 7% (59/837) 2024-08-20T22:07:54.0037511Z remote: Compressing objects: 8% (67/837) 2024-08-20T22:07:54.0240575Z remote: Compressing objects: 9% (76/837) 2024-08-20T22:07:54.0491191Z remote: Compressing objects: 10% (84/837) 2024-08-20T22:07:54.0687375Z remote: Compressing objects: 11% (93/837) 2024-08-20T22:07:54.0854189Z remote: Compressing objects: 12% (101/837) 2024-08-20T22:07:54.1011535Z remote: Compressing objects: 13% (109/837) 2024-08-20T22:07:54.1127289Z remote: Compressing objects: 14% (118/837) 2024-08-20T22:07:54.1210990Z remote: Compressing objects: 15% (126/837) 2024-08-20T22:07:54.1297362Z remote: Compressing objects: 16% (134/837) 2024-08-20T22:07:54.1352501Z remote: Compressing objects: 17% (143/837) 2024-08-20T22:07:54.1401782Z remote: Compressing objects: 18% (151/837) 2024-08-20T22:07:54.1443129Z remote: Compressing objects: 19% (160/837) 2024-08-20T22:07:54.1472538Z remote: Compressing objects: 20% (168/837) 2024-08-20T22:07:54.1494453Z remote: Compressing objects: 21% (176/837) 2024-08-20T22:07:54.1497533Z remote: Compressing objects: 22% (185/837) 2024-08-20T22:07:54.1502039Z remote: Compressing objects: 23% (193/837) 2024-08-20T22:07:54.1506517Z remote: Compressing objects: 24% (201/837) 2024-08-20T22:07:54.1510980Z remote: Compressing objects: 25% (210/837) 2024-08-20T22:07:54.1518647Z remote: Compressing objects: 26% (218/837) 2024-08-20T22:07:54.1529132Z remote: Compressing objects: 27% (226/837) 2024-08-20T22:07:54.1548216Z remote: Compressing objects: 28% (235/837) 2024-08-20T22:07:54.1551646Z remote: Compressing objects: 29% (243/837) 2024-08-20T22:07:54.1563085Z remote: Compressing objects: 30% (252/837) 2024-08-20T22:07:54.1571969Z remote: Compressing objects: 31% (260/837) 2024-08-20T22:07:54.1579280Z remote: Compressing objects: 32% (268/837) 2024-08-20T22:07:54.1584785Z remote: Compressing objects: 33% (277/837) 2024-08-20T22:07:54.1589543Z remote: Compressing objects: 34% (285/837) 2024-08-20T22:07:54.1596201Z remote: Compressing objects: 35% (293/837) 2024-08-20T22:07:54.1602186Z remote: Compressing objects: 36% (302/837) 2024-08-20T22:07:54.1611335Z remote: Compressing objects: 37% (310/837) 2024-08-20T22:07:54.1616333Z remote: Compressing objects: 38% (319/837) 2024-08-20T22:07:54.1622748Z remote: Compressing objects: 39% (327/837) 2024-08-20T22:07:54.1627767Z remote: Compressing objects: 40% (335/837) 2024-08-20T22:07:54.1635109Z remote: Compressing objects: 41% (344/837) 2024-08-20T22:07:54.1641499Z remote: Compressing objects: 42% (352/837) 2024-08-20T22:07:54.1646307Z remote: Compressing objects: 43% (360/837) 2024-08-20T22:07:54.1652843Z remote: Compressing objects: 44% (369/837) 2024-08-20T22:07:54.1658100Z remote: Compressing objects: 45% (377/837) 2024-08-20T22:07:54.1660870Z remote: Compressing objects: 46% (386/837) 2024-08-20T22:07:54.1664625Z remote: Compressing objects: 47% (394/837) 2024-08-20T22:07:54.1668305Z remote: Compressing objects: 48% (402/837) 2024-08-20T22:07:54.1671420Z remote: Compressing objects: 49% (411/837) 2024-08-20T22:07:54.1673279Z remote: Compressing objects: 50% (419/837) 2024-08-20T22:07:54.1676747Z remote: Compressing objects: 51% (427/837) 2024-08-20T22:07:54.1678897Z remote: Compressing objects: 52% (436/837) 2024-08-20T22:07:54.1680001Z remote: Compressing objects: 53% (444/837) 2024-08-20T22:07:54.1682385Z remote: Compressing objects: 54% (452/837) 2024-08-20T22:07:54.1683855Z remote: Compressing objects: 55% (461/837) 2024-08-20T22:07:54.1684872Z remote: Compressing objects: 56% (469/837) 2024-08-20T22:07:54.1686218Z remote: Compressing objects: 57% (478/837) 2024-08-20T22:07:54.1686746Z remote: Compressing objects: 58% (486/837) 2024-08-20T22:07:54.1687442Z remote: Compressing objects: 59% (494/837) 2024-08-20T22:07:54.1687976Z remote: Compressing objects: 60% (503/837) 2024-08-20T22:07:54.1688490Z remote: Compressing objects: 61% (511/837) 2024-08-20T22:07:54.1689023Z remote: Compressing objects: 62% (519/837) 2024-08-20T22:07:54.1689556Z remote: Compressing objects: 63% (528/837) 2024-08-20T22:07:54.1694079Z remote: Compressing objects: 64% (536/837) 2024-08-20T22:07:54.1700071Z remote: Compressing objects: 65% (545/837) 2024-08-20T22:07:54.1705067Z remote: Compressing objects: 66% (553/837) 2024-08-20T22:07:54.1709492Z remote: Compressing objects: 67% (561/837) 2024-08-20T22:07:54.1713419Z remote: Compressing objects: 68% (570/837) 2024-08-20T22:07:54.1717859Z remote: Compressing objects: 69% (578/837) 2024-08-20T22:07:54.1722020Z remote: Compressing objects: 70% (586/837) 2024-08-20T22:07:54.1724465Z remote: Compressing objects: 71% (595/837) 2024-08-20T22:07:54.1727448Z remote: Compressing objects: 72% (603/837) 2024-08-20T22:07:54.1730833Z remote: Compressing objects: 73% (612/837) 2024-08-20T22:07:54.1733435Z remote: Compressing objects: 74% (620/837) 2024-08-20T22:07:54.1737124Z remote: Compressing objects: 75% (628/837) 2024-08-20T22:07:54.1739161Z remote: Compressing objects: 76% (637/837) 2024-08-20T22:07:54.1741997Z remote: Compressing objects: 77% (645/837) 2024-08-20T22:07:54.1743601Z remote: Compressing objects: 78% (653/837) 2024-08-20T22:07:54.1745523Z remote: Compressing objects: 79% (662/837) 2024-08-20T22:07:54.1747350Z remote: Compressing objects: 80% (670/837) 2024-08-20T22:07:54.1749319Z remote: Compressing objects: 81% (678/837) 2024-08-20T22:07:54.1751594Z remote: Compressing objects: 82% (687/837) 2024-08-20T22:07:54.1753283Z remote: Compressing objects: 83% (695/837) 2024-08-20T22:07:54.1754806Z remote: Compressing objects: 84% (704/837) 2024-08-20T22:07:54.1756058Z remote: Compressing objects: 85% (712/837) 2024-08-20T22:07:54.1758140Z remote: Compressing objects: 86% (720/837) 2024-08-20T22:07:54.1759694Z remote: Compressing objects: 87% (729/837) 2024-08-20T22:07:54.1761233Z remote: Compressing objects: 88% (737/837) 2024-08-20T22:07:54.1763675Z remote: Compressing objects: 89% (745/837) 2024-08-20T22:07:54.1765688Z remote: Compressing objects: 90% (754/837) 2024-08-20T22:07:54.1766849Z remote: Compressing objects: 91% (762/837) 2024-08-20T22:07:54.1768457Z remote: Compressing objects: 92% (771/837) 2024-08-20T22:07:54.1768992Z remote: Compressing objects: 93% (779/837) 2024-08-20T22:07:54.1769512Z remote: Compressing objects: 94% (787/837) 2024-08-20T22:07:54.1770711Z remote: Compressing objects: 95% (796/837) 2024-08-20T22:07:54.1771396Z remote: Compressing objects: 96% (804/837) 2024-08-20T22:07:54.1772888Z remote: Compressing objects: 97% (812/837) 2024-08-20T22:07:54.1774399Z remote: Compressing objects: 98% (821/837) 2024-08-20T22:07:54.1774927Z remote: Compressing objects: 99% (829/837) 2024-08-20T22:07:54.1775921Z remote: Compressing objects: 100% (837/837) 2024-08-20T22:07:54.1776496Z remote: Compressing objects: 100% (837/837), done. 2024-08-20T22:08:17.6858521Z remote: Total 1010208 (delta 1154), reused 1394 (delta 916), pack-reused 1008452 (from 1) 2024-08-20T22:08:36.4459373Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:08:36.4500010Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:36.4506527Z ##[endgroup] 2024-08-20T22:08:36.4510204Z ##[group]Determining the checkout info 2024-08-20T22:08:36.4510879Z ##[endgroup] 2024-08-20T22:08:36.4511413Z ##[group]Checking out the ref 2024-08-20T22:08:36.4512397Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:38.1767264Z ##[endgroup] 2024-08-20T22:08:38.1768454Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:08:38.1770342Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:08:38.1835875Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:08:38.1876549Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:08:38.1923337Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:08:38.1961960Z ##[endgroup] 2024-08-20T22:08:38.1962552Z ##[group]Fetching submodules 2024-08-20T22:08:38.1966125Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:08:38.2368116Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:08:38.2756408Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:08:38.2758478Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:08:38.2761939Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:08:38.2766816Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:08:38.2770736Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:38.2773048Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:08:38.2776780Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:08:38.2780272Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:08:38.2783930Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:08:38.2787564Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:08:38.2791086Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:08:38.2794834Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:08:38.2798610Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:08:38.2802604Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:08:38.2806599Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:08:38.2810897Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:38.2818463Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:08:38.2822976Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:08:38.2826998Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:08:38.2831417Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:08:38.2835754Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:08:38.2840201Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:08:38.2844727Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:08:38.2849276Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:08:38.2853813Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:08:38.2858852Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:08:38.2863502Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:08:38.2868391Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:08:38.2873628Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:08:38.2878821Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:08:38.2884194Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:08:38.2889600Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:08:38.2894539Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:08:38.2903228Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:08:38.2940897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:08:38.6044630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:08:38.7934863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:08:38.9583545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:08:39.2443568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:08:41.0656870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:08:51.3144211Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:08:51.7175677Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:08:52.0788908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:08:52.6616483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:08:53.8964524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:08:55.9683994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:09:01.9298605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:09:03.4222042Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:09:05.0928279Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:09:06.2743793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:09:06.7203907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:09:07.1214271Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:09:08.2021936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:09:08.5975381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:09:08.8897255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:09:10.6001476Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:09:11.3883923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:09:12.0466235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:09:17.7501057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:09:19.5550920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:09:22.6522482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:09:22.8938792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:09:31.7065718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:09:31.8835810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:09:32.1080202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:09:32.9948154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:09:33.2784146Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:09:33.8924361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:09:34.3342571Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:09:34.3502170Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:09:34.3627093Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:09:34.3948178Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:09:34.4399058Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:09:35.6320623Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:09:35.6609373Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:09:35.7147076Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:09:35.8288944Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:09:35.8702003Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:09:36.4972460Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:09:36.7868822Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:09:36.8852361Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:09:36.8873393Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:09:36.8876996Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:09:36.8880292Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:09:36.8883762Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:09:36.8887248Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:09:36.8921154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:09:37.9036733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:09:38.5043181Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:09:40.5861797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:09:41.6269698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:09:42.0459725Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:09:42.1586048Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:09:42.6661287Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:09:42.7386398Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:09:42.7546281Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:09:42.9024439Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:09:42.9489922Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:09:42.9959677Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:09:43.0293707Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:09:43.0838206Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:09:43.1000031Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:09:43.1023092Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:09:43.1053378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:09:55.7267416Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:09:55.7503287Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:09:55.8520714Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:09:55.8542214Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:09:55.8544936Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:09:55.8548857Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:09:55.8581996Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:09:56.3573304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:09:57.5133246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:09:58.6094704Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:09:58.6114958Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:09:58.6118584Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:09:58.6123840Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:09:58.6127374Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:09:58.6131187Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:09:58.6134841Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:09:58.6138679Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:09:58.6142779Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:09:58.6184109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:09:59.4511563Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:09:59.8272333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:10:00.9436743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:10:01.2810688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:10:01.8039751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:10:02.8133428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:10:09.9325386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:10:10.5216568Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:10:10.5463413Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:10:10.5906535Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:10:10.6085229Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:10:10.6105560Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:10.6138091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:10:11.0958134Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:10:11.1200610Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:10:11.1703849Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:10:11.2903114Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:10:11.3107324Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:10:11.3575915Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:10:11.4253014Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:10:11.4725023Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:10:11.5032242Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:10:11.6322004Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:10:12.1372029Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:10:12.1413800Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:12.1417667Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:12.1450983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:10:12.5698097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:10:13.4638051Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:10:13.5063039Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:10:13.5983073Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:10:13.6005686Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:13.6009088Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:13.6012455Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:13.6016142Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:13.6020199Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:13.6024973Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:13.6028438Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:13.6031986Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:13.6065785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:10:14.0363202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:10:15.0785283Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:10:15.3832059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:10:21.0906323Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:10:21.4161123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:10:21.6238123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:10:21.9800967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:10:28.7900764Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:10:28.8397706Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:10:28.8599528Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:10:28.9857995Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:10:29.0024976Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:10:29.0233687Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:10:29.0447686Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:10:29.0472082Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:29.0475456Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:29.0516858Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:10:30.7532594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:10:32.0476970Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:10:32.1020341Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:10:32.7633922Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:10:32.7785016Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:10:33.0981238Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:10:33.1008792Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:33.1012433Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:33.1048792Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:10:33.5289184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:10:34.4918037Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:10:34.5762125Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:10:34.5890377Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:10:34.6055791Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:10:34.6510513Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:10:34.6866506Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:10:34.7363507Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:10:34.7719539Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:10:34.7742749Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:34.7747204Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:34.7748951Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:34.7751199Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:34.7788135Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:10:35.7859977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:10:36.0064386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:10:37.1269498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:10:38.0466902Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:10:38.0674302Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:10:38.1484619Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:10:38.1844744Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:10:38.1864188Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:38.1897449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:10:38.3810655Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:10:38.3864421Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:10:38.4255586Z Entering 'android/libs/fbjni' 2024-08-20T22:10:38.4313100Z Entering 'third_party/FP16' 2024-08-20T22:10:38.4366442Z Entering 'third_party/FXdiv' 2024-08-20T22:10:38.4422675Z Entering 'third_party/NNPACK' 2024-08-20T22:10:38.4477533Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:38.4530981Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:38.4603148Z Entering 'third_party/benchmark' 2024-08-20T22:10:38.4656952Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:38.4714043Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:38.4768963Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:38.4823480Z Entering 'third_party/cutlass' 2024-08-20T22:10:38.4886002Z Entering 'third_party/eigen' 2024-08-20T22:10:38.4947738Z Entering 'third_party/fbgemm' 2024-08-20T22:10:38.4999227Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:38.5050139Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:38.5107907Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:38.5166350Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:38.5216098Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:38.5269108Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:38.5324753Z Entering 'third_party/fmt' 2024-08-20T22:10:38.5380382Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:38.5433910Z Entering 'third_party/gloo' 2024-08-20T22:10:38.5488017Z Entering 'third_party/googletest' 2024-08-20T22:10:38.5549522Z Entering 'third_party/ideep' 2024-08-20T22:10:38.5600475Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:38.5660953Z Entering 'third_party/ittapi' 2024-08-20T22:10:38.5717367Z Entering 'third_party/kineto' 2024-08-20T22:10:38.5774936Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:38.5826434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:38.5884262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:38.5941827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:38.5996889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:38.6047945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:38.6109460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:38.6165169Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:38.6220976Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:38.6273914Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:38.6329644Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:38.6380620Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:38.6435789Z Entering 'third_party/mimalloc' 2024-08-20T22:10:38.6490257Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:38.6545458Z Entering 'third_party/nlohmann' 2024-08-20T22:10:38.6600690Z Entering 'third_party/onnx' 2024-08-20T22:10:38.6668034Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:38.6724647Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:38.6787049Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:38.6846168Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:38.6895255Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:38.6946737Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:38.7001839Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:38.7054894Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:38.7106242Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:38.7157726Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:38.7207752Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:38.7261880Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:38.7320978Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:38.7393600Z Entering 'third_party/pocketfft' 2024-08-20T22:10:38.7446804Z Entering 'third_party/protobuf' 2024-08-20T22:10:38.7500886Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:38.7552108Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:38.7606610Z Entering 'third_party/psimd' 2024-08-20T22:10:38.7663152Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:38.7717933Z Entering 'third_party/pybind11' 2024-08-20T22:10:38.7769750Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:38.7823240Z Entering 'third_party/sleef' 2024-08-20T22:10:38.7876937Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:38.7928476Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:38.7984391Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:38.8040029Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:38.8092388Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:38.8144222Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:38.8222494Z ##[endgroup] 2024-08-20T22:10:38.8225299Z ##[group]Persisting credentials for submodules 2024-08-20T22:10:38.8228313Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:10:38.8617335Z Entering 'android/libs/fbjni' 2024-08-20T22:10:38.8688491Z Entering 'third_party/FP16' 2024-08-20T22:10:38.8762387Z Entering 'third_party/FXdiv' 2024-08-20T22:10:38.8835314Z Entering 'third_party/NNPACK' 2024-08-20T22:10:38.8904013Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:38.8973056Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:38.9056861Z Entering 'third_party/benchmark' 2024-08-20T22:10:38.9126264Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:38.9196587Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:38.9268184Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:38.9337087Z Entering 'third_party/cutlass' 2024-08-20T22:10:38.9414783Z Entering 'third_party/eigen' 2024-08-20T22:10:38.9490748Z Entering 'third_party/fbgemm' 2024-08-20T22:10:38.9565613Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:38.9637415Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:38.9708074Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:38.9781966Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:38.9855132Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:38.9929624Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:39.0001179Z Entering 'third_party/fmt' 2024-08-20T22:10:39.0069375Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:39.0144279Z Entering 'third_party/gloo' 2024-08-20T22:10:39.0216414Z Entering 'third_party/googletest' 2024-08-20T22:10:39.0286749Z Entering 'third_party/ideep' 2024-08-20T22:10:39.0353894Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:39.0429436Z Entering 'third_party/ittapi' 2024-08-20T22:10:39.0498762Z Entering 'third_party/kineto' 2024-08-20T22:10:39.0568700Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:39.0635379Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:39.0707499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:39.0778580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:39.0853986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:39.0924294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:39.0999350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:39.1072628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:39.1145755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:39.1215739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:39.1289818Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:39.1358979Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:39.1431973Z Entering 'third_party/mimalloc' 2024-08-20T22:10:39.1505814Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:39.1578355Z Entering 'third_party/nlohmann' 2024-08-20T22:10:39.1653520Z Entering 'third_party/onnx' 2024-08-20T22:10:39.1740861Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:39.1808840Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:39.1887682Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:39.1956734Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:39.2030245Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:39.2097856Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:39.2167532Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:39.2235724Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:39.2304266Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:39.2371645Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:39.2442336Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:39.2520779Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:39.2600297Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:39.2689932Z Entering 'third_party/pocketfft' 2024-08-20T22:10:39.2757309Z Entering 'third_party/protobuf' 2024-08-20T22:10:39.2836428Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:39.2908390Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:39.2980793Z Entering 'third_party/psimd' 2024-08-20T22:10:39.3053755Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:39.3125623Z Entering 'third_party/pybind11' 2024-08-20T22:10:39.3196990Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:39.3267492Z Entering 'third_party/sleef' 2024-08-20T22:10:39.3337592Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:39.3405279Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:39.3473737Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:39.3541302Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:39.3612890Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:39.3679322Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:39.3778883Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:10:39.4170378Z Entering 'android/libs/fbjni' 2024-08-20T22:10:39.4234713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:10:39.4257041Z Entering 'third_party/FP16' 2024-08-20T22:10:39.4329174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:10:39.4351459Z Entering 'third_party/FXdiv' 2024-08-20T22:10:39.4413914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:10:39.4436850Z Entering 'third_party/NNPACK' 2024-08-20T22:10:39.4499527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:10:39.4524260Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:39.4588818Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:10:39.4610457Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:39.4677907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:10:39.4720657Z Entering 'third_party/benchmark' 2024-08-20T22:10:39.4783476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:10:39.4806441Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:39.4884527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:10:39.4899418Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:39.4964366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:10:39.4988826Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:39.5052719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:10:39.5075226Z Entering 'third_party/cutlass' 2024-08-20T22:10:39.5138346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:10:39.5168335Z Entering 'third_party/eigen' 2024-08-20T22:10:39.5233112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:10:39.5257293Z Entering 'third_party/fbgemm' 2024-08-20T22:10:39.5325218Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:10:39.5346713Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:39.5408181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:10:39.5430400Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:39.5493284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:10:39.5516737Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:39.5579687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:10:39.5607531Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:39.5669200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.5690618Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:39.5757083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:10:39.5781340Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:39.5843575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:10:39.5869426Z Entering 'third_party/fmt' 2024-08-20T22:10:39.5935452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:10:39.5958254Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:39.6023986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:10:39.6044865Z Entering 'third_party/gloo' 2024-08-20T22:10:39.6108557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:10:39.6132211Z Entering 'third_party/googletest' 2024-08-20T22:10:39.6194527Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.6217985Z Entering 'third_party/ideep' 2024-08-20T22:10:39.6286624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:10:39.6306801Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:39.6378410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:10:39.6407546Z Entering 'third_party/ittapi' 2024-08-20T22:10:39.6472491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:10:39.6497271Z Entering 'third_party/kineto' 2024-08-20T22:10:39.6563161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:10:39.6584682Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:39.6648950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:10:39.6669756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:39.6738342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:10:39.6762580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:39.6824618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:10:39.6847011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:39.6911615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:10:39.6934179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:39.6998087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:10:39.7023000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:39.7091254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:10:39.7116132Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:39.7182569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:10:39.7203575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:39.7268172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.7288916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:39.7352223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:10:39.7375258Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:39.7441679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:10:39.7467211Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:39.7531669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:10:39.7553152Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:39.7616725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.7645296Z Entering 'third_party/mimalloc' 2024-08-20T22:10:39.7709111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:10:39.7732359Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:39.7800101Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:10:39.7825100Z Entering 'third_party/nlohmann' 2024-08-20T22:10:39.7890225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:10:39.7914907Z Entering 'third_party/onnx' 2024-08-20T22:10:39.7980032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:10:39.8019698Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:39.8083512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:10:39.8107295Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:39.8172681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:10:39.8201608Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:39.8265954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:10:39.8288676Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:39.8350467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:10:39.8376186Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:39.8442721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.8466942Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:39.8529260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:10:39.8549966Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:39.8613146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:10:39.8636415Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:39.8697392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:10:39.8718831Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:39.8781937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:10:39.8805227Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:39.8868396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:10:39.8892965Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:39.8960836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:10:39.8985860Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:39.9053203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:10:39.9077835Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:39.9142329Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:10:39.9186943Z Entering 'third_party/pocketfft' 2024-08-20T22:10:39.9256647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:10:39.9278495Z Entering 'third_party/protobuf' 2024-08-20T22:10:39.9352513Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:10:39.9375940Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:39.9438378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:10:39.9460649Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:39.9527740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:39.9552887Z Entering 'third_party/psimd' 2024-08-20T22:10:39.9616301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:10:39.9638587Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:39.9710406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:10:39.9735615Z Entering 'third_party/pybind11' 2024-08-20T22:10:39.9803061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:10:39.9825242Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:39.9887171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:10:39.9910181Z Entering 'third_party/sleef' 2024-08-20T22:10:39.9974247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:10:39.9997386Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:40.0062642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:10:40.0086370Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:40.0148467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:10:40.0169750Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:40.0238469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:10:40.0259572Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:40.0322082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:10:40.0343750Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:40.0412011Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:10:40.0432441Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:40.0496876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:10:40.1194608Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:10:40.1589600Z Entering 'android/libs/fbjni' 2024-08-20T22:10:40.1642238Z Entering 'third_party/FP16' 2024-08-20T22:10:40.1696260Z Entering 'third_party/FXdiv' 2024-08-20T22:10:40.1750259Z Entering 'third_party/NNPACK' 2024-08-20T22:10:40.1803424Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:40.1864614Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:40.1935371Z Entering 'third_party/benchmark' 2024-08-20T22:10:40.1987353Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:40.2041047Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:40.2098490Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:40.2151968Z Entering 'third_party/cutlass' 2024-08-20T22:10:40.2213646Z Entering 'third_party/eigen' 2024-08-20T22:10:40.2270978Z Entering 'third_party/fbgemm' 2024-08-20T22:10:40.2324427Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:40.2375572Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:40.2428880Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:40.2487982Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:40.2539670Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:40.2597446Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:40.2656635Z Entering 'third_party/fmt' 2024-08-20T22:10:40.2708975Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:40.2768101Z Entering 'third_party/gloo' 2024-08-20T22:10:40.2823152Z Entering 'third_party/googletest' 2024-08-20T22:10:40.2876700Z Entering 'third_party/ideep' 2024-08-20T22:10:40.2929201Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:40.2989290Z Entering 'third_party/ittapi' 2024-08-20T22:10:40.3042791Z Entering 'third_party/kineto' 2024-08-20T22:10:40.3095108Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:40.3147614Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:40.3205310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:40.3258560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:40.3313685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:40.3364148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:40.3426357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:40.3477836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:40.3531098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:40.3586017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:40.3645710Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:40.3703181Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:40.3757865Z Entering 'third_party/mimalloc' 2024-08-20T22:10:40.3813915Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:40.3870419Z Entering 'third_party/nlohmann' 2024-08-20T22:10:40.3926163Z Entering 'third_party/onnx' 2024-08-20T22:10:40.3993033Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:40.4050178Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:40.4111391Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:40.4166074Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:40.4216943Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:40.4272888Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:40.4328774Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:40.4382122Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:40.4434139Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:40.4486215Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:40.4535931Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:40.4594381Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:40.4649653Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:40.4723918Z Entering 'third_party/pocketfft' 2024-08-20T22:10:40.4776994Z Entering 'third_party/protobuf' 2024-08-20T22:10:40.4858309Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:40.4914866Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:40.4970641Z Entering 'third_party/psimd' 2024-08-20T22:10:40.5026044Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:40.5079306Z Entering 'third_party/pybind11' 2024-08-20T22:10:40.5133180Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:40.5188215Z Entering 'third_party/sleef' 2024-08-20T22:10:40.5248182Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:40.5304139Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:40.5355905Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:40.5408411Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:40.5460399Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:40.5511222Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:40.5588757Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:10:40.5977096Z Entering 'android/libs/fbjni' 2024-08-20T22:10:40.6035585Z Entering 'third_party/FP16' 2024-08-20T22:10:40.6089324Z Entering 'third_party/FXdiv' 2024-08-20T22:10:40.6146527Z Entering 'third_party/NNPACK' 2024-08-20T22:10:40.6200899Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:40.6261516Z Entering 'third_party/XNNPACK' 2024-08-20T22:10:40.6335647Z Entering 'third_party/benchmark' 2024-08-20T22:10:40.6389562Z Entering 'third_party/cpp-httplib' 2024-08-20T22:10:40.6443853Z Entering 'third_party/cpuinfo' 2024-08-20T22:10:40.6501975Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:10:40.6559611Z Entering 'third_party/cutlass' 2024-08-20T22:10:40.6626460Z Entering 'third_party/eigen' 2024-08-20T22:10:40.6682686Z Entering 'third_party/fbgemm' 2024-08-20T22:10:40.6735797Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:10:40.6787765Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:10:40.6839314Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:10:40.6899312Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:10:40.6953539Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:10:40.7008374Z Entering 'third_party/flatbuffers' 2024-08-20T22:10:40.7066750Z Entering 'third_party/fmt' 2024-08-20T22:10:40.7119852Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:40.7175316Z Entering 'third_party/gloo' 2024-08-20T22:10:40.7228804Z Entering 'third_party/googletest' 2024-08-20T22:10:40.7286450Z Entering 'third_party/ideep' 2024-08-20T22:10:40.7339650Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:10:40.7399919Z Entering 'third_party/ittapi' 2024-08-20T22:10:40.7462097Z Entering 'third_party/kineto' 2024-08-20T22:10:40.7516957Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:10:40.7568073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:10:40.7628080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:10:40.7681832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:10:40.7735173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:10:40.7787524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:10:40.7849223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:10:40.7900608Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:10:40.7952654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:10:40.8005551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:10:40.8061256Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:10:40.8112524Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:10:40.8170037Z Entering 'third_party/mimalloc' 2024-08-20T22:10:40.8225619Z Entering 'third_party/nccl/nccl' 2024-08-20T22:10:40.8281730Z Entering 'third_party/nlohmann' 2024-08-20T22:10:40.8337579Z Entering 'third_party/onnx' 2024-08-20T22:10:40.8405660Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:10:40.8464854Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:10:40.8522780Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:10:40.8578533Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:10:40.8635438Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:10:40.8688633Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:10:40.8747323Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:10:40.8799941Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:10:40.8851608Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:10:40.8903807Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:10:40.8953524Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:10:40.9007257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:10:40.9063333Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:10:40.9136857Z Entering 'third_party/pocketfft' 2024-08-20T22:10:40.9191041Z Entering 'third_party/protobuf' 2024-08-20T22:10:40.9246847Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:10:40.9298520Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:10:40.9355501Z Entering 'third_party/psimd' 2024-08-20T22:10:40.9409679Z Entering 'third_party/pthreadpool' 2024-08-20T22:10:40.9463392Z Entering 'third_party/pybind11' 2024-08-20T22:10:40.9520409Z Entering 'third_party/python-peachpy' 2024-08-20T22:10:40.9574624Z Entering 'third_party/sleef' 2024-08-20T22:10:40.9629773Z Entering 'third_party/tensorpipe' 2024-08-20T22:10:40.9685741Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:10:40.9737361Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:10:40.9788810Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:10:40.9840655Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:10:40.9895157Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:10:40.9980009Z ##[endgroup] 2024-08-20T22:10:41.0031613Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:10:41.0070808Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:10:41.0277531Z Prepare all required actions 2024-08-20T22:10:41.0278001Z Getting action download info 2024-08-20T22:10:41.2117476Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:10:41.2117895Z env: 2024-08-20T22:10:41.2118418Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:41.2118843Z ##[endgroup] 2024-08-20T22:10:41.2178981Z ##[group]Run set -euo pipefail 2024-08-20T22:10:41.2179423Z set -euo pipefail 2024-08-20T22:10:41.2179820Z function get_ec2_metadata() { 2024-08-20T22:10:41.2180370Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:10:41.2181378Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:10:41.2182154Z  category=$1 2024-08-20T22:10:41.2182709Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:10:41.2183351Z  runner_name_str=i-01a7d20fbad5d17a0 2024-08-20T22:10:41.2183848Z  if [[ -f /.inarc ]]; then 2024-08-20T22:10:41.2184351Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:10:41.2184933Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:10:41.2185630Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:10:41.2186235Z  else 2024-08-20T22:10:41.2186736Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:10:41.2187327Z  fi 2024-08-20T22:10:41.2187630Z } 2024-08-20T22:10:41.2187993Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:10:41.2188592Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:10:41.2189247Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:10:41.2189833Z echo "system info $(uname -a)" 2024-08-20T22:10:41.2200306Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:41.2200801Z env: 2024-08-20T22:10:41.2201081Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:41.2201425Z ##[endgroup] 2024-08-20T22:10:41.2300195Z ami-id: ami-06c68f701d8090592 2024-08-20T22:10:41.2364479Z instance-id: i-01a7d20fbad5d17a0 2024-08-20T22:10:41.2429153Z instance-type: g5.12xlarge 2024-08-20T22:10:41.2443709Z system info Linux ip-10-0-37-127.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:10:41.2468083Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:10:41.2468976Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:10:41.2477589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:41.2478269Z env: 2024-08-20T22:10:41.2478557Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:41.2478904Z ##[endgroup] 2024-08-20T22:10:41.2553881Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:10:41.2554493Z if systemctl is-active --quiet docker; then 2024-08-20T22:10:41.2555047Z  echo "Docker daemon is running..."; 2024-08-20T22:10:41.2555504Z else 2024-08-20T22:10:41.2555996Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:10:41.2556588Z fi 2024-08-20T22:10:41.2564951Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:41.2565436Z env: 2024-08-20T22:10:41.2565708Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:41.2566050Z ##[endgroup] 2024-08-20T22:10:41.2648618Z Docker daemon is running... 2024-08-20T22:10:41.2701698Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:10:41.2702266Z with: 2024-08-20T22:10:41.2702548Z shell: bash 2024-08-20T22:10:41.2702834Z timeout_minutes: 5 2024-08-20T22:10:41.2703155Z max_attempts: 3 2024-08-20T22:10:41.2703468Z retry_wait_seconds: 30 2024-08-20T22:10:41.2706576Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:10:41.2709525Z polling_interval_seconds: 1 2024-08-20T22:10:41.2709890Z warning_on_retry: true 2024-08-20T22:10:41.2710235Z continue_on_error: false 2024-08-20T22:10:41.2710579Z env: 2024-08-20T22:10:41.2710849Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:41.2711190Z AWS_RETRY_MODE: standard 2024-08-20T22:10:41.2711536Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:10:41.2711879Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:10:41.2712259Z ##[endgroup] 2024-08-20T22:10:42.4534433Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:10:42.4535386Z Configure a credential helper to remove this warning. See 2024-08-20T22:10:42.4536213Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:10:42.4536729Z 2024-08-20T22:10:42.4539819Z Login Succeeded 2024-08-20T22:10:43.3284724Z Command completed after 1 attempt(s). 2024-08-20T22:10:43.3351906Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:10:43.3352611Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:10:43.3353250Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:10:43.3363309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:43.3363800Z env: 2024-08-20T22:10:43.3364075Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.3364425Z ##[endgroup] 2024-08-20T22:10:43.3468824Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:43.3469594Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:10:43.3470184Z # shellcheck disable=SC2046 2024-08-20T22:10:43.3470648Z docker stop $(docker ps -q) || true 2024-08-20T22:10:43.3471176Z # Prune all of the docker images 2024-08-20T22:10:43.3471629Z docker system prune -af 2024-08-20T22:10:43.3480677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:43.3481172Z env: 2024-08-20T22:10:43.3481446Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.3481784Z ##[endgroup] 2024-08-20T22:10:43.3852315Z "docker stop" requires at least 1 argument. 2024-08-20T22:10:43.3853393Z See 'docker stop --help'. 2024-08-20T22:10:43.3853642Z 2024-08-20T22:10:43.3853872Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:10:43.3854242Z 2024-08-20T22:10:43.3854402Z Stop one or more running containers 2024-08-20T22:10:43.4062858Z Total reclaimed space: 0B 2024-08-20T22:10:43.4110425Z ##[group]Run set +e 2024-08-20T22:10:43.4110791Z set +e 2024-08-20T22:10:43.4111098Z set -x 2024-08-20T22:10:43.4111395Z  2024-08-20T22:10:43.4111715Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:10:43.4112505Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:10:43.4113579Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:10:43.4114329Z # one is returned at random 2024-08-20T22:10:43.4114869Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:10:43.4115370Z  2024-08-20T22:10:43.4115715Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:10:43.4116326Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:10:43.4117289Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:10:43.4117834Z  2024-08-20T22:10:43.4118448Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:10:43.4118997Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:10:43.4119506Z  exit 1 2024-08-20T22:10:43.4119811Z  fi 2024-08-20T22:10:43.4120107Z fi 2024-08-20T22:10:43.4120394Z  2024-08-20T22:10:43.4120752Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:10:43.4121259Z  # Clean up any old records first 2024-08-20T22:10:43.4121768Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:10:43.4122221Z fi 2024-08-20T22:10:43.4122490Z  2024-08-20T22:10:43.4122927Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:10:43.4123466Z cat /etc/hosts 2024-08-20T22:10:43.4132870Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:43.4133358Z env: 2024-08-20T22:10:43.4133643Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.4133979Z ##[endgroup] 2024-08-20T22:10:43.4187845Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:10:43.4194604Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:10:43.4195438Z ++ tail -n1 2024-08-20T22:10:43.4428708Z + RESOLVED_IP=18.160.10.76 2024-08-20T22:10:43.4429169Z + '[' -z 18.160.10.76 ']' 2024-08-20T22:10:43.4429599Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:10:43.4443077Z 108.138.64.61 download.pytorch.org 2024-08-20T22:10:43.4445413Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T22:10:43.5822725Z + echo '18.160.10.76 download.pytorch.org' 2024-08-20T22:10:43.5823249Z + sudo tee -a /etc/hosts 2024-08-20T22:10:43.6145350Z 18.160.10.76 download.pytorch.org 2024-08-20T22:10:43.6167974Z + cat /etc/hosts 2024-08-20T22:10:43.6179083Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:10:43.6192268Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:10:43.6193028Z 18.160.10.76 download.pytorch.org 2024-08-20T22:10:43.6353888Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:10:43.6354480Z with: 2024-08-20T22:10:43.6355570Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6356778Z docker-build-dir: .ci/docker 2024-08-20T22:10:43.6357158Z working-directory: . 2024-08-20T22:10:43.6357616Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:43.6358135Z force-push: false 2024-08-20T22:10:43.6358442Z env: 2024-08-20T22:10:43.6358722Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.6359062Z ##[endgroup] 2024-08-20T22:10:43.6382135Z ##[group]Run set -ex 2024-08-20T22:10:43.6382701Z set -ex 2024-08-20T22:10:43.6383167Z  2024-08-20T22:10:43.6383820Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:10:43.6384945Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:10:43.6385911Z # job could then download the pre-built image as usual 2024-08-20T22:10:43.6386745Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:10:43.6387580Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6388363Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6389021Z  2024-08-20T22:10:43.6389719Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:10:43.6390439Z  exit 0 2024-08-20T22:10:43.6390853Z else 2024-08-20T22:10:43.6391402Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6391953Z fi 2024-08-20T22:10:43.6392358Z  2024-08-20T22:10:43.6392935Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:10:43.6393890Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:10:43.6394781Z  # use it as it is, but first let's extract the tag 2024-08-20T22:10:43.6395745Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:10:43.6396532Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6397353Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6398066Z else 2024-08-20T22:10:43.6398622Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:10:43.6399337Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6400381Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6401240Z fi 2024-08-20T22:10:43.6412316Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:43.6412921Z env: 2024-08-20T22:10:43.6413356Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.6413886Z REPO_NAME: pytorch 2024-08-20T22:10:43.6415097Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6416439Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:10:43.6417007Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:43.6417609Z ##[endgroup] 2024-08-20T22:10:43.6449345Z + [[ ! -d .ci/docker ]] 2024-08-20T22:10:43.6449960Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:10:43.6450413Z + echo skip=false 2024-08-20T22:10:43.6452832Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:10:43.6457759Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6459092Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:10:43.6484396Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6485258Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6486795Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6521346Z ##[group]Run set +e 2024-08-20T22:10:43.6522006Z set +e 2024-08-20T22:10:43.6522479Z set -x 2024-08-20T22:10:43.6522829Z  2024-08-20T22:10:43.6523233Z login() { 2024-08-20T22:10:43.6524052Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:10:43.6524945Z } 2024-08-20T22:10:43.6525332Z  2024-08-20T22:10:43.6525745Z retry () { 2024-08-20T22:10:43.6526279Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:10:43.6526932Z } 2024-08-20T22:10:43.6527389Z  2024-08-20T22:10:43.6527796Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:10:43.6528321Z  2024-08-20T22:10:43.6528966Z # Check if image already exists, if it does then skip building it 2024-08-20T22:10:43.6529789Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:10:43.6530357Z  exit 0 2024-08-20T22:10:43.6530836Z fi 2024-08-20T22:10:43.6531234Z  2024-08-20T22:10:43.6531854Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:10:43.6532829Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:10:43.6533756Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:10:43.6534542Z  # if we're on the base branch then use the parent commit 2024-08-20T22:10:43.6535350Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:10:43.6535851Z else 2024-08-20T22:10:43.6536429Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:10:43.6537314Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:10:43.6537925Z fi 2024-08-20T22:10:43.6538296Z  2024-08-20T22:10:43.6538775Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:10:43.6539383Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6540018Z  2024-08-20T22:10:43.6540747Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:10:43.6541798Z  exit 0 2024-08-20T22:10:43.6542290Z fi 2024-08-20T22:10:43.6542685Z  2024-08-20T22:10:43.6543203Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:10:43.6544353Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:10:43.6545268Z  exit 1 2024-08-20T22:10:43.6545706Z fi 2024-08-20T22:10:43.6546120Z  2024-08-20T22:10:43.6546725Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:10:43.6547807Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:10:43.6548844Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:10:43.6549970Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:10:43.6551187Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:10:43.6552149Z fi 2024-08-20T22:10:43.6552559Z  2024-08-20T22:10:43.6553022Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:10:43.6561522Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:43.6562112Z env: 2024-08-20T22:10:43.6562595Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:43.6563022Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:10:43.6563584Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:43.6565133Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6566478Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:43.6567347Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:43.6567920Z ##[endgroup] 2024-08-20T22:10:43.6598350Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:43.6599121Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:43.6601509Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:10:43.6602722Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:44.2104896Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:10:44.2106585Z Configure a credential helper to remove this warning. See 2024-08-20T22:10:44.2108256Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:10:44.2109431Z 2024-08-20T22:10:44.2109831Z Login Succeeded 2024-08-20T22:10:44.2132318Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:44.4334337Z { 2024-08-20T22:10:44.4334893Z "schemaVersion": 2, 2024-08-20T22:10:44.4335607Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:10:44.4336276Z "config": { 2024-08-20T22:10:44.4336924Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:10:44.4337837Z "size": 48711, 2024-08-20T22:10:44.4338495Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:10:44.4339552Z }, 2024-08-20T22:10:44.4339916Z "layers": [ 2024-08-20T22:10:44.4340409Z { 2024-08-20T22:10:44.4340984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4345021Z "size": 28580681, 2024-08-20T22:10:44.4345918Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:10:44.4346856Z }, 2024-08-20T22:10:44.4347472Z { 2024-08-20T22:10:44.4348139Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4349106Z "size": 7943451, 2024-08-20T22:10:44.4351027Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:10:44.4351962Z }, 2024-08-20T22:10:44.4353008Z { 2024-08-20T22:10:44.4353913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4354896Z "size": 55728572, 2024-08-20T22:10:44.4355840Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:10:44.4356855Z }, 2024-08-20T22:10:44.4357355Z { 2024-08-20T22:10:44.4358240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4359156Z "size": 186, 2024-08-20T22:10:44.4360092Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:10:44.4361112Z }, 2024-08-20T22:10:44.4361690Z { 2024-08-20T22:10:44.4362343Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4363203Z "size": 6886, 2024-08-20T22:10:44.4364145Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:10:44.4365093Z }, 2024-08-20T22:10:44.4365853Z { 2024-08-20T22:10:44.4366670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4367494Z "size": 1329001756, 2024-08-20T22:10:44.4368344Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:10:44.4369111Z }, 2024-08-20T22:10:44.4369464Z { 2024-08-20T22:10:44.4370021Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4370726Z "size": 62501, 2024-08-20T22:10:44.4371697Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:10:44.4372587Z }, 2024-08-20T22:10:44.4372952Z { 2024-08-20T22:10:44.4373455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4374400Z "size": 1684, 2024-08-20T22:10:44.4375038Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:10:44.4375759Z }, 2024-08-20T22:10:44.4376125Z { 2024-08-20T22:10:44.4376665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4377334Z "size": 1523, 2024-08-20T22:10:44.4378043Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:10:44.4378734Z }, 2024-08-20T22:10:44.4379094Z { 2024-08-20T22:10:44.4379680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4380324Z "size": 2528295403, 2024-08-20T22:10:44.4380999Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:10:44.4381878Z }, 2024-08-20T22:10:44.4382243Z { 2024-08-20T22:10:44.4382743Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4383503Z "size": 86016, 2024-08-20T22:10:44.4384186Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:10:44.4384966Z }, 2024-08-20T22:10:44.4385512Z + exit 0 2024-08-20T22:10:44.4385881Z { 2024-08-20T22:10:44.4386493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4387182Z "size": 1822, 2024-08-20T22:10:44.4387805Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:10:44.4388592Z }, 2024-08-20T22:10:44.4388961Z { 2024-08-20T22:10:44.4389459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4390243Z "size": 246761563, 2024-08-20T22:10:44.4390928Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:10:44.4391721Z }, 2024-08-20T22:10:44.4392120Z { 2024-08-20T22:10:44.4392663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4393332Z "size": 544, 2024-08-20T22:10:44.4394039Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:10:44.4394710Z }, 2024-08-20T22:10:44.4395067Z { 2024-08-20T22:10:44.4395675Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4396337Z "size": 1258, 2024-08-20T22:10:44.4396967Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:10:44.4397764Z }, 2024-08-20T22:10:44.4398118Z { 2024-08-20T22:10:44.4398632Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4399350Z "size": 484, 2024-08-20T22:10:44.4399985Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:10:44.4400754Z }, 2024-08-20T22:10:44.4401091Z { 2024-08-20T22:10:44.4401678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4402509Z "size": 91711766, 2024-08-20T22:10:44.4403183Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:10:44.4403880Z }, 2024-08-20T22:10:44.4404326Z { 2024-08-20T22:10:44.4404874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4405629Z "size": 3390, 2024-08-20T22:10:44.4406351Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:10:44.4407081Z }, 2024-08-20T22:10:44.4407440Z { 2024-08-20T22:10:44.4408000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4408687Z "size": 1909, 2024-08-20T22:10:44.4409328Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:10:44.4410114Z }, 2024-08-20T22:10:44.4410429Z { 2024-08-20T22:10:44.4410975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4411705Z "size": 701, 2024-08-20T22:10:44.4412342Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:10:44.4413131Z }, 2024-08-20T22:10:44.4413587Z { 2024-08-20T22:10:44.4414127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4414748Z "size": 2864254879, 2024-08-20T22:10:44.4415521Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:10:44.4416242Z }, 2024-08-20T22:10:44.4416621Z { 2024-08-20T22:10:44.4417174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4417858Z "size": 380, 2024-08-20T22:10:44.4418922Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:10:44.4419710Z }, 2024-08-20T22:10:44.4420027Z { 2024-08-20T22:10:44.4420581Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4421404Z "size": 12900, 2024-08-20T22:10:44.4422023Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:10:44.4422783Z }, 2024-08-20T22:10:44.4423198Z { 2024-08-20T22:10:44.4423754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4424371Z "size": 803, 2024-08-20T22:10:44.4425067Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:10:44.4425826Z }, 2024-08-20T22:10:44.4426263Z { 2024-08-20T22:10:44.4426821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4427529Z "size": 106, 2024-08-20T22:10:44.4428265Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:10:44.4429010Z }, 2024-08-20T22:10:44.4429481Z { 2024-08-20T22:10:44.4429993Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4430757Z "size": 503, 2024-08-20T22:10:44.4431386Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:10:44.4432125Z }, 2024-08-20T22:10:44.4432592Z { 2024-08-20T22:10:44.4433127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4433737Z "size": 121477267, 2024-08-20T22:10:44.4434489Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:10:44.4435185Z }, 2024-08-20T22:10:44.4435515Z { 2024-08-20T22:10:44.4436089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4436739Z "size": 109, 2024-08-20T22:10:44.4437398Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:10:44.4438126Z }, 2024-08-20T22:10:44.4438480Z { 2024-08-20T22:10:44.4439032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4439757Z "size": 490, 2024-08-20T22:10:44.4440350Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:10:44.4441072Z }, 2024-08-20T22:10:44.4441479Z { 2024-08-20T22:10:44.4441984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4442641Z "size": 297, 2024-08-20T22:10:44.4443318Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:10:44.4444191Z }, 2024-08-20T22:10:44.4444517Z { 2024-08-20T22:10:44.4445133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4445808Z "size": 103, 2024-08-20T22:10:44.4446543Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:10:44.4447248Z }, 2024-08-20T22:10:44.4447602Z { 2024-08-20T22:10:44.4448192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4448886Z "size": 1473, 2024-08-20T22:10:44.4449497Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:10:44.4450269Z }, 2024-08-20T22:10:44.4450648Z { 2024-08-20T22:10:44.4451340Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4452088Z "size": 594729516, 2024-08-20T22:10:44.4452742Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:10:44.4453473Z }, 2024-08-20T22:10:44.4453869Z { 2024-08-20T22:10:44.4454410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4455086Z "size": 164, 2024-08-20T22:10:44.4455792Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:10:44.4456460Z }, 2024-08-20T22:10:44.4456832Z { 2024-08-20T22:10:44.4457436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4458053Z "size": 564, 2024-08-20T22:10:44.4458701Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:10:44.4459466Z }, 2024-08-20T22:10:44.4459821Z { 2024-08-20T22:10:44.4460368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4461154Z "size": 35874203, 2024-08-20T22:10:44.4461901Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:10:44.4462637Z }, 2024-08-20T22:10:44.4463029Z { 2024-08-20T22:10:44.4463607Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4464267Z "size": 104, 2024-08-20T22:10:44.4464951Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:10:44.4465635Z }, 2024-08-20T22:10:44.4465998Z { 2024-08-20T22:10:44.4466592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4467264Z "size": 425, 2024-08-20T22:10:44.4467933Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:10:44.4468744Z }, 2024-08-20T22:10:44.4469114Z { 2024-08-20T22:10:44.4469616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4470352Z "size": 20262183, 2024-08-20T22:10:44.4471049Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:10:44.4471847Z }, 2024-08-20T22:10:44.4472184Z { 2024-08-20T22:10:44.4472726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4473473Z "size": 438, 2024-08-20T22:10:44.4474090Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:10:44.4474834Z }, 2024-08-20T22:10:44.4475257Z { 2024-08-20T22:10:44.4475793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4476413Z "size": 701, 2024-08-20T22:10:44.4477122Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:10:44.4477833Z }, 2024-08-20T22:10:44.4478192Z { 2024-08-20T22:10:44.4478755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4479409Z "size": 141, 2024-08-20T22:10:44.4480056Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:10:44.4480847Z }, 2024-08-20T22:10:44.4481200Z { 2024-08-20T22:10:44.4481756Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4482572Z "size": 135, 2024-08-20T22:10:44.4483170Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:10:44.4483906Z }, 2024-08-20T22:10:44.4484316Z { 2024-08-20T22:10:44.4484879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4485501Z "size": 4838068375, 2024-08-20T22:10:44.4486205Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:10:44.4486943Z }, 2024-08-20T22:10:44.4487303Z { 2024-08-20T22:10:44.4487853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4488556Z "size": 194, 2024-08-20T22:10:44.4489210Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:10:44.4490107Z }, 2024-08-20T22:10:44.4490436Z { 2024-08-20T22:10:44.4490992Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4491780Z "size": 564, 2024-08-20T22:10:44.4492399Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:10:44.4493120Z }, 2024-08-20T22:10:44.4493554Z { 2024-08-20T22:10:44.4494097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4494719Z "size": 43162921, 2024-08-20T22:10:44.4495491Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:10:44.4496207Z }, 2024-08-20T22:10:44.4496664Z { 2024-08-20T22:10:44.4497164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4497834Z "size": 106, 2024-08-20T22:10:44.4498548Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:10:44.4499225Z }, 2024-08-20T22:10:44.4499583Z { 2024-08-20T22:10:44.4500193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4500933Z "size": 1380, 2024-08-20T22:10:44.4501658Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:10:44.4502432Z }, 2024-08-20T22:10:44.4502802Z { 2024-08-20T22:10:44.4503372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4504033Z "size": 701, 2024-08-20T22:10:44.4504673Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:10:44.4505407Z }, 2024-08-20T22:10:44.4505815Z { 2024-08-20T22:10:44.4506304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4506976Z "size": 139, 2024-08-20T22:10:44.4507655Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:10:44.4508344Z }, 2024-08-20T22:10:44.4508725Z { 2024-08-20T22:10:44.4509313Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4509991Z "size": 118, 2024-08-20T22:10:44.4510596Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:10:44.4511373Z }, 2024-08-20T22:10:44.4511744Z { 2024-08-20T22:10:44.4512329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4512947Z "size": 4324672728, 2024-08-20T22:10:44.4513644Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:10:44.4514425Z }, 2024-08-20T22:10:44.4514763Z { 2024-08-20T22:10:44.4515296Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4516018Z "size": 174, 2024-08-20T22:10:44.4516674Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:10:44.4517345Z }, 2024-08-20T22:10:44.4517786Z { 2024-08-20T22:10:44.4518598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4519295Z "size": 907, 2024-08-20T22:10:44.4519991Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:10:44.4521377Z }, 2024-08-20T22:10:44.4521747Z { 2024-08-20T22:10:44.4522363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4522982Z "size": 701, 2024-08-20T22:10:44.4523662Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:10:44.4524446Z }, 2024-08-20T22:10:44.4524771Z { 2024-08-20T22:10:44.4525333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4526047Z "size": 134, 2024-08-20T22:10:44.4526697Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:10:44.4527407Z }, 2024-08-20T22:10:44.4527821Z { 2024-08-20T22:10:44.4528534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4529191Z "size": 32, 2024-08-20T22:10:44.4529849Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:10:44.4530630Z }, 2024-08-20T22:10:44.4531005Z { 2024-08-20T22:10:44.4531600Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4532289Z "size": 158, 2024-08-20T22:10:44.4532943Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:10:44.4533722Z }, 2024-08-20T22:10:44.4534058Z { 2024-08-20T22:10:44.4534594Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4535323Z "size": 1841, 2024-08-20T22:10:44.4535968Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:10:44.4536641Z }, 2024-08-20T22:10:44.4537075Z { 2024-08-20T22:10:44.4537646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4538371Z "size": 7529776, 2024-08-20T22:10:44.4539024Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:10:44.4539751Z }, 2024-08-20T22:10:44.4540174Z { 2024-08-20T22:10:44.4540704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4541437Z "size": 163, 2024-08-20T22:10:44.4542171Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:10:44.4542897Z }, 2024-08-20T22:10:44.4543219Z { 2024-08-20T22:10:44.4543841Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4544542Z "size": 7943, 2024-08-20T22:10:44.4545203Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:10:44.4545953Z }, 2024-08-20T22:10:44.4546327Z { 2024-08-20T22:10:44.4546885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4547579Z "size": 8068, 2024-08-20T22:10:44.4548231Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:10:44.4548978Z }, 2024-08-20T22:10:44.4549399Z { 2024-08-20T22:10:44.4549905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4550598Z "size": 302, 2024-08-20T22:10:44.4551339Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:10:44.4552082Z }, 2024-08-20T22:10:44.4552417Z { 2024-08-20T22:10:44.4553002Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4553672Z "size": 7629638, 2024-08-20T22:10:44.4554340Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:10:44.4555066Z }, 2024-08-20T22:10:44.4555453Z { 2024-08-20T22:10:44.4555989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4556649Z "size": 108, 2024-08-20T22:10:44.4557337Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:10:44.4558067Z }, 2024-08-20T22:10:44.4759033Z { 2024-08-20T22:10:44.4759607Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4760624Z "size": 54145661, 2024-08-20T22:10:44.4761310Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:10:44.4761962Z }, 2024-08-20T22:10:44.4762233Z { 2024-08-20T22:10:44.4762666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4763211Z "size": 473, 2024-08-20T22:10:44.4763733Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:10:44.4764342Z }, 2024-08-20T22:10:44.4764614Z { 2024-08-20T22:10:44.4765095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4765639Z "size": 1374858766, 2024-08-20T22:10:44.4766181Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:10:44.4766966Z }, 2024-08-20T22:10:44.4767206Z { 2024-08-20T22:10:44.4767620Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4768157Z "size": 106, 2024-08-20T22:10:44.4768703Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:10:44.4769312Z }, 2024-08-20T22:10:44.4769552Z { 2024-08-20T22:10:44.4769970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4770508Z "size": 559, 2024-08-20T22:10:44.4771034Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:10:44.4771653Z }, 2024-08-20T22:10:44.4771898Z { 2024-08-20T22:10:44.4772322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4772858Z "size": 46248488, 2024-08-20T22:10:44.4773398Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:10:44.4774005Z }, 2024-08-20T22:10:44.4774251Z { 2024-08-20T22:10:44.4774660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4775194Z "size": 111, 2024-08-20T22:10:44.4775719Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:10:44.4776317Z }, 2024-08-20T22:10:44.4776552Z { 2024-08-20T22:10:44.4776967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4777502Z "size": 32, 2024-08-20T22:10:44.4778026Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:10:44.4778625Z }, 2024-08-20T22:10:44.4778880Z { 2024-08-20T22:10:44.4779301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4779826Z "size": 32, 2024-08-20T22:10:44.4780349Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:10:44.4780968Z }, 2024-08-20T22:10:44.4781293Z { 2024-08-20T22:10:44.4781726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4782271Z "size": 32, 2024-08-20T22:10:44.4782792Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:10:44.4783396Z }, 2024-08-20T22:10:44.4783633Z { 2024-08-20T22:10:44.4784053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:10:44.4784589Z "size": 32, 2024-08-20T22:10:44.4785098Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:10:44.4785704Z } 2024-08-20T22:10:44.4785947Z ] 2024-08-20T22:10:44.4786190Z } 2024-08-20T22:10:44.4891576Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:10:44.4892049Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:10:44.4892585Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:10:44.4903052Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:44.4903560Z env: 2024-08-20T22:10:44.4903844Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:44.4904982Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:44.4906128Z ##[endgroup] 2024-08-20T22:10:44.4942063Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:44.4995771Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:10:44.4996351Z with: 2024-08-20T22:10:44.4997707Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:44.4999015Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:44.4999527Z env: 2024-08-20T22:10:44.4999978Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:44.5000323Z ##[endgroup] 2024-08-20T22:10:44.5017494Z ##[group]Run set -x 2024-08-20T22:10:44.5017861Z set -x 2024-08-20T22:10:44.5018499Z set +e 2024-08-20T22:10:44.5018868Z  2024-08-20T22:10:44.5019207Z login() { 2024-08-20T22:10:44.5019989Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:10:44.5020719Z } 2024-08-20T22:10:44.5021025Z  2024-08-20T22:10:44.5021445Z retry () { 2024-08-20T22:10:44.5021847Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:10:44.5022306Z } 2024-08-20T22:10:44.5022605Z  2024-08-20T22:10:44.5022932Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:10:44.5023353Z  2024-08-20T22:10:44.5023633Z set -e 2024-08-20T22:10:44.5024128Z # ignore output since only exit code is used for conditional 2024-08-20T22:10:44.5024860Z # only pull docker image if it's not available locally 2024-08-20T22:10:44.5025629Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:10:44.5026325Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:10:44.5026778Z fi 2024-08-20T22:10:44.5035969Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:44.5036483Z env: 2024-08-20T22:10:44.5036776Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:44.5037917Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:44.5039225Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:44.5039749Z ##[endgroup] 2024-08-20T22:10:44.5068812Z + set +e 2024-08-20T22:10:44.5069362Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:44.5070014Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:44.5073182Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:10:44.5074378Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:10:45.0434549Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:10:45.0435342Z Configure a credential helper to remove this warning. See 2024-08-20T22:10:45.0436161Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:10:45.0436687Z 2024-08-20T22:10:45.0436825Z Login Succeeded 2024-08-20T22:10:45.0459316Z + set -e 2024-08-20T22:10:45.0460606Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:45.0644266Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:45.0646396Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:45.2909220Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:10:45.2913178Z 7a2c55901189: Pulling fs layer 2024-08-20T22:10:45.2913707Z 224fe954d725: Pulling fs layer 2024-08-20T22:10:45.2914489Z 75722010b82e: Pulling fs layer 2024-08-20T22:10:45.2914886Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:10:45.2915282Z b57676e46aee: Pulling fs layer 2024-08-20T22:10:45.2915750Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:10:45.2916150Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:10:45.2916517Z 0c1227890755: Pulling fs layer 2024-08-20T22:10:45.2916878Z d8d1234baab3: Pulling fs layer 2024-08-20T22:10:45.2917343Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:10:45.2918072Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:10:45.2918737Z b57676e46aee: Waiting 2024-08-20T22:10:45.2919091Z a8c1e85b5e14: Waiting 2024-08-20T22:10:45.2919505Z 0c1227890755: Waiting 2024-08-20T22:10:45.2919916Z a41a8d1c11c8: Waiting 2024-08-20T22:10:45.2920307Z d527cbbb87e3: Waiting 2024-08-20T22:10:45.2920710Z 7ed32bc8e469: Waiting 2024-08-20T22:10:45.2921046Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:10:45.2921536Z d8d1234baab3: Waiting 2024-08-20T22:10:45.2921851Z ec1e7978c1fe: Waiting 2024-08-20T22:10:45.2922188Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:10:45.2922567Z 4373aa743caf: Pulling fs layer 2024-08-20T22:10:45.2922940Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:10:45.2923320Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:10:45.2923699Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:10:45.2924052Z 4373aa743caf: Waiting 2024-08-20T22:10:45.2924363Z 75b2abb16c68: Waiting 2024-08-20T22:10:45.2924690Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:10:45.2925050Z 0bf61114d37b: Waiting 2024-08-20T22:10:45.2925372Z ef1e3ef1e85f: Waiting 2024-08-20T22:10:45.2925708Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:10:45.2926140Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:10:45.2926513Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:10:45.2926885Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:10:45.2927254Z 33b639226eee: Pulling fs layer 2024-08-20T22:10:45.2927609Z 1fef4bd19792: Waiting 2024-08-20T22:10:45.2927941Z d3547bd43146: Pulling fs layer 2024-08-20T22:10:45.2928307Z 5645520256fa: Pulling fs layer 2024-08-20T22:10:45.2928675Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:10:45.2929032Z 34c47d63c5de: Waiting 2024-08-20T22:10:45.2929360Z 385d0421d67c: Pulling fs layer 2024-08-20T22:10:45.2929707Z 33b639226eee: Waiting 2024-08-20T22:10:45.2930044Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:10:45.2930406Z 2df849f72ba4: Waiting 2024-08-20T22:10:45.2930730Z 829ed7d91e38: Waiting 2024-08-20T22:10:45.2931052Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:10:45.2931436Z d3547bd43146: Waiting 2024-08-20T22:10:45.2931801Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:10:45.2932169Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:10:45.2932529Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:10:45.2932887Z 33444a5a7e01: Waiting 2024-08-20T22:10:45.2933210Z 7754ee068b33: Pulling fs layer 2024-08-20T22:10:45.2933568Z 0479fbac3361: Pulling fs layer 2024-08-20T22:10:45.2933921Z 1811ff4d3556: Waiting 2024-08-20T22:10:45.2934249Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:10:45.2934624Z afc429e87b59: Pulling fs layer 2024-08-20T22:10:45.2934975Z 0479fbac3361: Waiting 2024-08-20T22:10:45.2935297Z 019620471c25: Pulling fs layer 2024-08-20T22:10:45.2935662Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:10:45.2936016Z 019620471c25: Waiting 2024-08-20T22:10:45.2936337Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:10:45.2936702Z 560cd6122a38: Pulling fs layer 2024-08-20T22:10:45.2937055Z 27aeb1b2cd5e: Waiting 2024-08-20T22:10:45.2937379Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:10:45.2937733Z 829c0e19f4b7: Waiting 2024-08-20T22:10:45.2938064Z da6a838a66c7: Pulling fs layer 2024-08-20T22:10:45.2938419Z 385d0421d67c: Waiting 2024-08-20T22:10:45.2938749Z cbb79a705353: Pulling fs layer 2024-08-20T22:10:45.2939120Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:10:45.2939488Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:10:45.2939838Z 560cd6122a38: Waiting 2024-08-20T22:10:45.2940152Z da6a838a66c7: Waiting 2024-08-20T22:10:45.2940460Z cbb79a705353: Waiting 2024-08-20T22:10:45.2940788Z 4a0412373305: Pulling fs layer 2024-08-20T22:10:45.2941459Z 302c8c177944: Pulling fs layer 2024-08-20T22:10:45.2941839Z 02970347dff6: Pulling fs layer 2024-08-20T22:10:45.2942187Z 5645520256fa: Waiting 2024-08-20T22:10:45.2942567Z 916c5e858016: Pulling fs layer 2024-08-20T22:10:45.2943004Z f1d6874dee9c: Waiting 2024-08-20T22:10:45.2943342Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:10:45.2943703Z 02970347dff6: Waiting 2024-08-20T22:10:45.2944028Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:10:45.2944397Z f957c1624481: Pulling fs layer 2024-08-20T22:10:45.2944945Z 768c31dca016: Pulling fs layer 2024-08-20T22:10:45.2945298Z f957c1624481: Waiting 2024-08-20T22:10:45.2945604Z 916c5e858016: Waiting 2024-08-20T22:10:45.2945913Z 11fad972ef6c: Waiting 2024-08-20T22:10:45.2946247Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:10:45.2946620Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:10:45.2946975Z c42b3602a1fb: Waiting 2024-08-20T22:10:45.2947288Z 8c32d0ecebbe: Waiting 2024-08-20T22:10:45.2947618Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:10:45.2947986Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:10:45.2948349Z 0f085d4fa279: Waiting 2024-08-20T22:10:45.2948658Z 4a0412373305: Waiting 2024-08-20T22:10:45.2948982Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:10:45.2949354Z b10de02255c0: Pulling fs layer 2024-08-20T22:10:45.2949714Z 768c31dca016: Waiting 2024-08-20T22:10:45.2950028Z b119c4fc6c58: Waiting 2024-08-20T22:10:45.2950351Z e56eeff2db93: Pulling fs layer 2024-08-20T22:10:45.2950717Z a7a5392e2bf6: Waiting 2024-08-20T22:10:45.2951050Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:10:45.2951480Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:10:45.2951831Z 70ac3e38d0fd: Waiting 2024-08-20T22:10:45.2952159Z ee1273153675: Pulling fs layer 2024-08-20T22:10:45.2952539Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:10:45.2952914Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:10:45.2953279Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:10:45.2953641Z 6c1f0aabce7c: Waiting 2024-08-20T22:10:45.2953968Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:10:45.2954319Z 7ebe49c3a23e: Waiting 2024-08-20T22:10:45.2954648Z 37513e9b43d7: Waiting 2024-08-20T22:10:45.2954978Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:10:45.2955348Z d79fd7d52698: Pulling fs layer 2024-08-20T22:10:45.2955695Z b10de02255c0: Waiting 2024-08-20T22:10:45.2956031Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:10:45.2956395Z 5d115b29b5d5: Waiting 2024-08-20T22:10:45.2956723Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:10:45.2957075Z d79fd7d52698: Waiting 2024-08-20T22:10:45.2957396Z 9752d6fbbcab: Waiting 2024-08-20T22:10:45.2957725Z 4f4fb700ef54: Waiting 2024-08-20T22:10:45.4250387Z 224fe954d725: Verifying Checksum 2024-08-20T22:10:45.4250848Z 224fe954d725: Download complete 2024-08-20T22:10:45.4947547Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:10:45.4948463Z d527cbbb87e3: Download complete 2024-08-20T22:10:45.5698943Z b57676e46aee: Verifying Checksum 2024-08-20T22:10:45.6257840Z 7a2c55901189: Verifying Checksum 2024-08-20T22:10:45.6258272Z 7a2c55901189: Download complete 2024-08-20T22:10:45.6927729Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:10:45.6928159Z a41a8d1c11c8: Download complete 2024-08-20T22:10:45.7697480Z 0c1227890755: Download complete 2024-08-20T22:10:45.8316995Z d8d1234baab3: Verifying Checksum 2024-08-20T22:10:45.8317437Z d8d1234baab3: Download complete 2024-08-20T22:10:45.8996290Z 75722010b82e: Verifying Checksum 2024-08-20T22:10:45.8996700Z 75722010b82e: Download complete 2024-08-20T22:10:45.9876856Z ec1e7978c1fe: Download complete 2024-08-20T22:10:46.0547944Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:10:46.0548401Z ac23c5fcc1ef: Download complete 2024-08-20T22:10:46.7261789Z 7a2c55901189: Pull complete 2024-08-20T22:10:47.0284473Z 224fe954d725: Pull complete 2024-08-20T22:10:47.8894165Z 75722010b82e: Pull complete 2024-08-20T22:10:47.9178157Z d527cbbb87e3: Pull complete 2024-08-20T22:10:47.9527314Z b57676e46aee: Pull complete 2024-08-20T22:10:48.5751428Z ca426ffc9d2b: Download complete 2024-08-20T22:10:48.6506835Z 4373aa743caf: Download complete 2024-08-20T22:10:48.7215969Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:10:48.7216722Z 75b2abb16c68: Download complete 2024-08-20T22:10:48.7946997Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:10:48.7947485Z ef1e3ef1e85f: Download complete 2024-08-20T22:10:49.8000863Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:10:49.8001436Z 0bf61114d37b: Download complete 2024-08-20T22:10:49.8624205Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:10:49.8624678Z 2df849f72ba4: Download complete 2024-08-20T22:10:49.9339090Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:10:49.9339858Z 34c47d63c5de: Download complete 2024-08-20T22:10:50.0254294Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:10:50.0255842Z 1fef4bd19792: Download complete 2024-08-20T22:10:58.9166228Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:10:58.9166675Z a8c1e85b5e14: Download complete 2024-08-20T22:10:58.9898506Z b119c4fc6c58: Download complete 2024-08-20T22:10:59.0550187Z 33b639226eee: Download complete 2024-08-20T22:10:59.1272368Z d3547bd43146: Download complete 2024-08-20T22:10:59.1925639Z 5645520256fa: Verifying Checksum 2024-08-20T22:10:59.1926057Z 5645520256fa: Download complete 2024-08-20T22:10:59.2625438Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:10:59.2625849Z 829ed7d91e38: Download complete 2024-08-20T22:11:00.5235450Z 385d0421d67c: Verifying Checksum 2024-08-20T22:11:00.5235929Z 385d0421d67c: Download complete 2024-08-20T22:11:00.5883632Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:11:00.5884050Z 7cbfce999d76: Download complete 2024-08-20T22:11:00.6785067Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:11:00.6785515Z 6658e558ebe4: Download complete 2024-08-20T22:11:00.7479408Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:11:00.7479819Z 33444a5a7e01: Download complete 2024-08-20T22:11:00.8479398Z 7c47122f4e09: Download complete 2024-08-20T22:11:00.9159185Z 1811ff4d3556: Download complete 2024-08-20T22:11:06.9172269Z 7754ee068b33: Verifying Checksum 2024-08-20T22:11:06.9172749Z 7754ee068b33: Download complete 2024-08-20T22:11:06.9918503Z 0479fbac3361: Download complete 2024-08-20T22:11:07.0612234Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:11:07.0613539Z f1d6874dee9c: Download complete 2024-08-20T22:11:07.4836596Z afc429e87b59: Verifying Checksum 2024-08-20T22:11:07.4838142Z afc429e87b59: Download complete 2024-08-20T22:11:07.5513668Z 019620471c25: Verifying Checksum 2024-08-20T22:11:07.5514092Z 019620471c25: Download complete 2024-08-20T22:11:07.6132421Z 27aeb1b2cd5e: Download complete 2024-08-20T22:11:07.8636198Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:11:07.8637018Z 829c0e19f4b7: Download complete 2024-08-20T22:11:07.9287006Z 560cd6122a38: Verifying Checksum 2024-08-20T22:11:08.0161190Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:11:08.0161672Z 5a4eb19d7219: Download complete 2024-08-20T22:11:08.1170403Z da6a838a66c7: Verifying Checksum 2024-08-20T22:11:08.1170882Z da6a838a66c7: Download complete 2024-08-20T22:11:11.1600651Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:11:11.1601098Z 7ed32bc8e469: Download complete 2024-08-20T22:11:11.2367934Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:11:11.2368398Z 4d4fa6802ab3: Download complete 2024-08-20T22:11:11.3086165Z 0f085d4fa279: Download complete 2024-08-20T22:11:11.7306529Z a8c1e85b5e14: Pull complete 2024-08-20T22:11:11.7967646Z 4a0412373305: Verifying Checksum 2024-08-20T22:11:11.7968073Z 4a0412373305: Download complete 2024-08-20T22:11:11.8759449Z 302c8c177944: Download complete 2024-08-20T22:11:11.9394002Z 02970347dff6: Verifying Checksum 2024-08-20T22:11:11.9394470Z 02970347dff6: Download complete 2024-08-20T22:11:12.0479970Z 916c5e858016: Verifying Checksum 2024-08-20T22:11:12.0480416Z 916c5e858016: Download complete 2024-08-20T22:11:12.1121354Z c42b3602a1fb: Download complete 2024-08-20T22:11:12.1751877Z a41a8d1c11c8: Pull complete 2024-08-20T22:11:12.6848644Z 0c1227890755: Pull complete 2024-08-20T22:11:13.0703578Z d8d1234baab3: Pull complete 2024-08-20T22:11:18.7395296Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:11:18.7395771Z 6efb96fc87ea: Download complete 2024-08-20T22:11:18.8132889Z f957c1624481: Verifying Checksum 2024-08-20T22:11:18.8133747Z f957c1624481: Download complete 2024-08-20T22:11:18.8914054Z 768c31dca016: Download complete 2024-08-20T22:11:18.9578972Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:11:18.9579429Z 8c32d0ecebbe: Download complete 2024-08-20T22:11:18.9666629Z 4f4fb700ef54: Download complete 2024-08-20T22:11:19.0403138Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:11:19.1165581Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:11:19.1166095Z 37513e9b43d7: Download complete 2024-08-20T22:11:19.2407887Z a7a5392e2bf6: Download complete 2024-08-20T22:11:19.3145989Z b10de02255c0: Verifying Checksum 2024-08-20T22:11:19.3146487Z b10de02255c0: Download complete 2024-08-20T22:11:19.3793766Z e56eeff2db93: Verifying Checksum 2024-08-20T22:11:19.3794260Z e56eeff2db93: Download complete 2024-08-20T22:11:19.4386661Z 70ac3e38d0fd: Download complete 2024-08-20T22:11:19.5170452Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:11:19.5170941Z eb09cbf3e416: Download complete 2024-08-20T22:11:19.6380718Z ee1273153675: Verifying Checksum 2024-08-20T22:11:19.6381350Z ee1273153675: Download complete 2024-08-20T22:11:19.7063132Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:11:19.7063649Z e22979f9e0ee: Download complete 2024-08-20T22:11:20.2956801Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:11:20.2957364Z 6c1f0aabce7c: Download complete 2024-08-20T22:11:20.3561275Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:11:20.3561808Z 7ebe49c3a23e: Download complete 2024-08-20T22:11:34.1503616Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:11:34.1504211Z 5d115b29b5d5: Download complete 2024-08-20T22:11:34.2095287Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:11:34.2095820Z 7f2d7a04fbe6: Download complete 2024-08-20T22:11:34.2739357Z d79fd7d52698: Verifying Checksum 2024-08-20T22:11:34.2739893Z d79fd7d52698: Download complete 2024-08-20T22:11:34.7787885Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:11:34.7788426Z 9752d6fbbcab: Download complete 2024-08-20T22:11:34.8489266Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:11:34.8489807Z 83808b2eff4d: Download complete 2024-08-20T22:11:42.9552246Z 7ed32bc8e469: Pull complete 2024-08-20T22:11:43.4362726Z ec1e7978c1fe: Pull complete 2024-08-20T22:11:43.8528420Z ac23c5fcc1ef: Pull complete 2024-08-20T22:11:52.8674253Z ca426ffc9d2b: Pull complete 2024-08-20T22:11:53.2634501Z 4373aa743caf: Pull complete 2024-08-20T22:11:53.7185885Z 75b2abb16c68: Pull complete 2024-08-20T22:11:54.1817300Z ef1e3ef1e85f: Pull complete 2024-08-20T22:11:55.4089210Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:11:55.4089847Z 11fad972ef6c: Download complete 2024-08-20T22:11:56.5509043Z cbb79a705353: Verifying Checksum 2024-08-20T22:11:56.5509522Z cbb79a705353: Download complete 2024-08-20T22:11:56.9333309Z 0bf61114d37b: Pull complete 2024-08-20T22:11:57.2371811Z 2df849f72ba4: Pull complete 2024-08-20T22:11:57.6342248Z 34c47d63c5de: Pull complete 2024-08-20T22:11:58.0345149Z 1fef4bd19792: Pull complete 2024-08-20T22:12:58.2010816Z 6efb96fc87ea: Pull complete 2024-08-20T22:12:58.4780469Z b119c4fc6c58: Pull complete 2024-08-20T22:12:58.7922575Z 33b639226eee: Pull complete 2024-08-20T22:12:59.0489840Z d3547bd43146: Pull complete 2024-08-20T22:12:59.3197282Z 5645520256fa: Pull complete 2024-08-20T22:12:59.6554411Z 829ed7d91e38: Pull complete 2024-08-20T22:13:03.4341808Z 385d0421d67c: Pull complete 2024-08-20T22:13:03.8851228Z 7cbfce999d76: Pull complete 2024-08-20T22:13:04.3570932Z 6658e558ebe4: Pull complete 2024-08-20T22:13:04.8298950Z 33444a5a7e01: Pull complete 2024-08-20T22:13:05.2929665Z 7c47122f4e09: Pull complete 2024-08-20T22:13:05.8118665Z 1811ff4d3556: Pull complete 2024-08-20T22:13:16.7081617Z 7754ee068b33: Pull complete 2024-08-20T22:13:17.0666611Z 0479fbac3361: Pull complete 2024-08-20T22:13:17.5180140Z f1d6874dee9c: Pull complete 2024-08-20T22:13:18.6799569Z afc429e87b59: Pull complete 2024-08-20T22:13:19.1270214Z 019620471c25: Pull complete 2024-08-20T22:13:19.5759764Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:13:20.2749252Z 829c0e19f4b7: Pull complete 2024-08-20T22:13:20.5453512Z 560cd6122a38: Pull complete 2024-08-20T22:13:21.4487541Z 5a4eb19d7219: Pull complete 2024-08-20T22:13:21.9286424Z da6a838a66c7: Pull complete 2024-08-20T22:14:00.8878912Z cbb79a705353: Pull complete 2024-08-20T22:14:01.3364891Z 4d4fa6802ab3: Pull complete 2024-08-20T22:14:01.7844208Z 0f085d4fa279: Pull complete 2024-08-20T22:14:04.8821381Z 4a0412373305: Pull complete 2024-08-20T22:14:05.3192477Z 302c8c177944: Pull complete 2024-08-20T22:14:05.6496063Z 02970347dff6: Pull complete 2024-08-20T22:14:06.4971190Z 916c5e858016: Pull complete 2024-08-20T22:14:06.9512719Z c42b3602a1fb: Pull complete 2024-08-20T22:15:30.2841629Z 11fad972ef6c: Pull complete 2024-08-20T22:15:30.7626026Z f957c1624481: Pull complete 2024-08-20T22:15:31.2142348Z 768c31dca016: Pull complete 2024-08-20T22:15:31.8304551Z 8c32d0ecebbe: Pull complete 2024-08-20T22:15:32.0545678Z 4f4fb700ef54: Pull complete 2024-08-20T22:15:32.5078195Z 78ee77eddef2: Pull complete 2024-08-20T22:15:32.9784942Z 37513e9b43d7: Pull complete 2024-08-20T22:15:33.5052619Z a7a5392e2bf6: Pull complete 2024-08-20T22:15:33.9931995Z b10de02255c0: Pull complete 2024-08-20T22:15:34.5063859Z e56eeff2db93: Pull complete 2024-08-20T22:15:34.9574356Z 70ac3e38d0fd: Pull complete 2024-08-20T22:15:35.3565864Z eb09cbf3e416: Pull complete 2024-08-20T22:15:37.2150795Z ee1273153675: Pull complete 2024-08-20T22:15:37.6815059Z e22979f9e0ee: Pull complete 2024-08-20T22:15:40.2156576Z 6c1f0aabce7c: Pull complete 2024-08-20T22:15:40.7405508Z 7ebe49c3a23e: Pull complete 2024-08-20T22:15:54.1666505Z 5d115b29b5d5: Pull complete 2024-08-20T22:15:54.1806025Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:15:54.1959857Z d79fd7d52698: Pull complete 2024-08-20T22:15:54.8491660Z 9752d6fbbcab: Pull complete 2024-08-20T22:15:54.8622792Z 83808b2eff4d: Pull complete 2024-08-20T22:15:54.9268967Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:15:54.9292775Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:54.9315269Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:54.9378492Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:15:54.9379409Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:15:54.9390435Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:54.9390930Z env: 2024-08-20T22:15:54.9391211Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:54.9391557Z ##[endgroup] 2024-08-20T22:15:54.9571165Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:15:54.9571705Z with: 2024-08-20T22:15:54.9571983Z driver-version: 550.54.15 2024-08-20T22:15:54.9572315Z env: 2024-08-20T22:15:54.9572594Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:54.9572978Z ##[endgroup] 2024-08-20T22:15:54.9614456Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:54.9615021Z with: 2024-08-20T22:15:54.9615302Z timeout_minutes: 10 2024-08-20T22:15:54.9615630Z max_attempts: 3 2024-08-20T22:15:54.9648047Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:15:54.9684679Z retry_wait_seconds: 10 2024-08-20T22:15:54.9685050Z polling_interval_seconds: 1 2024-08-20T22:15:54.9685419Z warning_on_retry: true 2024-08-20T22:15:54.9685769Z continue_on_error: false 2024-08-20T22:15:54.9686111Z env: 2024-08-20T22:15:54.9686395Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:54.9686745Z DRIVER_VERSION: 550.54.15 2024-08-20T22:15:54.9687100Z ##[endgroup] 2024-08-20T22:15:55.0518375Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:15:55.0519182Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:15:55.0522611Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:15:55.3590436Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:15:55.3591041Z No packages marked for removal. 2024-08-20T22:15:55.3655880Z Dependencies resolved. 2024-08-20T22:15:55.3665105Z Nothing to do. 2024-08-20T22:15:55.3665590Z Complete! 2024-08-20T22:15:55.3975953Z + install_nvidia_driver_common 2024-08-20T22:15:55.3980089Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:15:55.3980594Z + lspci 2024-08-20T22:15:55.3980956Z Before installing NVIDIA driver 2024-08-20T22:15:55.4098209Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:15:55.4099047Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:15:55.4100005Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:15:55.4100800Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:15:55.4101706Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:15:55.4102764Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:15:55.4103576Z 00:1b.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.4104465Z 00:1c.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.4105262Z 00:1d.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.4106705Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.4107663Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:15:55.4108249Z + lsmod 2024-08-20T22:15:55.4152369Z Module Size Used by 2024-08-20T22:15:55.4152966Z ib_core 454656 0 2024-08-20T22:15:55.4153474Z veth 36864 0 2024-08-20T22:15:55.4153962Z nvidia_modeset 1351680 0 2024-08-20T22:15:55.4154372Z video 65536 1 nvidia_modeset 2024-08-20T22:15:55.4154813Z wmi 36864 1 video 2024-08-20T22:15:55.4155211Z nvidia_uvm 4706304 0 2024-08-20T22:15:55.4155665Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-08-20T22:15:55.4156156Z drm 602112 1 nvidia 2024-08-20T22:15:55.4156599Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:15:55.4157247Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:15:55.4157905Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:15:55.4158416Z xt_conntrack 16384 1 2024-08-20T22:15:55.4158794Z nft_chain_nat 16384 3 2024-08-20T22:15:55.4159171Z xt_MASQUERADE 20480 1 2024-08-20T22:15:55.4159623Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:15:55.4160111Z nf_conntrack_netlink 57344 0 2024-08-20T22:15:55.4160873Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:15:55.4161651Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:15:55.4162236Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:15:55.4162845Z xfrm_user 57344 1 2024-08-20T22:15:55.4163356Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:15:55.4165587Z xt_addrtype 16384 2 2024-08-20T22:15:55.4165972Z nft_compat 20480 4 2024-08-20T22:15:55.4166419Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:15:55.4167032Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:15:55.4167569Z br_netfilter 36864 0 2024-08-20T22:15:55.4167974Z bridge 307200 1 br_netfilter 2024-08-20T22:15:55.4168404Z stp 16384 1 bridge 2024-08-20T22:15:55.4168825Z llc 16384 2 bridge,stp 2024-08-20T22:15:55.4169247Z overlay 167936 0 2024-08-20T22:15:55.4169637Z tls 114688 0 2024-08-20T22:15:55.4169998Z nls_ascii 16384 1 2024-08-20T22:15:55.4170363Z nls_cp437 20480 1 2024-08-20T22:15:55.4170726Z vfat 24576 1 2024-08-20T22:15:55.4171085Z fat 86016 1 vfat 2024-08-20T22:15:55.4171506Z ena 167936 0 2024-08-20T22:15:55.4172006Z ghash_clmulni_intel 16384 0 2024-08-20T22:15:55.4172555Z ptp 36864 1 ena 2024-08-20T22:15:55.4173079Z pps_core 24576 1 ptp 2024-08-20T22:15:55.4173607Z aesni_intel 393216 0 2024-08-20T22:15:55.4174088Z i8042 45056 0 2024-08-20T22:15:55.4174574Z serio 28672 3 i8042 2024-08-20T22:15:55.4175105Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:15:55.4175667Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:15:55.4176168Z button 24576 0 2024-08-20T22:15:55.4176534Z sunrpc 692224 1 2024-08-20T22:15:55.4177006Z sch_fq_codel 20480 33 2024-08-20T22:15:55.4177502Z dm_mod 188416 0 2024-08-20T22:15:55.4177963Z fuse 163840 1 2024-08-20T22:15:55.4178409Z dax 45056 1 dm_mod 2024-08-20T22:15:55.4178818Z loop 36864 0 2024-08-20T22:15:55.4179194Z configfs 57344 1 2024-08-20T22:15:55.4179573Z dmi_sysfs 20480 0 2024-08-20T22:15:55.4179940Z crc32_pclmul 16384 0 2024-08-20T22:15:55.4180321Z crc32c_intel 24576 0 2024-08-20T22:15:55.4180980Z efivarfs 24576 1 2024-08-20T22:15:55.4181357Z + modinfo nvidia 2024-08-20T22:15:55.4182092Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:15:55.4182810Z alias: char-major-195-* 2024-08-20T22:15:55.4183198Z version: 550.54.15 2024-08-20T22:15:55.4183549Z supported: external 2024-08-20T22:15:55.4183894Z license: NVIDIA 2024-08-20T22:15:55.4184277Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:15:55.4184761Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:15:55.4185223Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:15:55.4185685Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:15:55.4186188Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:15:55.4186687Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:15:55.4187185Z depends: i2c-core,drm 2024-08-20T22:15:55.4187544Z retpoline: Y 2024-08-20T22:15:55.4187862Z name: nvidia 2024-08-20T22:15:55.4188493Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:15:55.4189193Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:15:55.4189839Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:15:55.4190438Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:15:55.4190886Z parm: NVreg_RmLogonRC:int 2024-08-20T22:15:55.4191319Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:15:55.4191780Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:15:55.4192224Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:15:55.4192670Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:15:55.4193194Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:15:55.4193854Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:15:55.4194336Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:15:55.4194771Z parm: NVreg_EnableMSI:int 2024-08-20T22:15:55.4195191Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:15:55.4195651Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:15:55.4196185Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:15:55.4196766Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:15:55.4197318Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:15:55.4197927Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:55.4198530Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:15:55.4199145Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:55.4199728Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:15:55.4200230Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:15:55.4200772Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:15:55.4201313Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:15:55.4201818Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:15:55.4202288Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:15:55.4202824Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:15:55.4203296Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:15:55.4203751Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:15:55.4204261Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:15:55.4204798Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:15:55.4205278Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:15:55.4205771Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:15:55.4206278Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:15:55.4206769Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:15:55.4207276Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:15:55.4207774Z parm: NVreg_RmMsg:charp 2024-08-20T22:15:55.4208195Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:15:55.4208806Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:15:55.4209279Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:15:55.4209744Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:15:55.4210228Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:15:55.4210707Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:15:55.4211165Z parm: rm_firmware_active:charp 2024-08-20T22:15:55.4211589Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:15:55.4211980Z ++ command -v nvidia-smi 2024-08-20T22:15:55.4212387Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:15:55.4212824Z + set +e 2024-08-20T22:15:55.4213338Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:15:55.4491220Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:15:55.4492176Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:15:55.4492825Z + '[' 0 -ne 0 ']' 2024-08-20T22:15:55.4493192Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:15:55.4493568Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:15:55.4494280Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:15:55.4494983Z + set -e 2024-08-20T22:15:55.4495295Z + '[' 1 -eq 0 ']' 2024-08-20T22:15:55.4495865Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:15:55.4496544Z + post_install_nvidia_driver_common 2024-08-20T22:15:55.4500037Z + sudo modprobe nvidia 2024-08-20T22:15:55.5813706Z + echo 'After installing NVIDIA driver' 2024-08-20T22:15:55.5814766Z + lspci 2024-08-20T22:15:55.5815305Z After installing NVIDIA driver 2024-08-20T22:15:55.5923377Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:15:55.5924352Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:15:55.5925445Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:15:55.5926748Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:15:55.5927591Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:15:55.5928381Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:15:55.5929110Z 00:1b.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.5929782Z 00:1c.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.5930433Z 00:1d.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.5931092Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:15:55.5931889Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:15:55.5932473Z + lsmod 2024-08-20T22:15:55.5959656Z Module Size Used by 2024-08-20T22:15:55.5960213Z ib_core 454656 0 2024-08-20T22:15:55.5960712Z veth 36864 0 2024-08-20T22:15:55.5961185Z nvidia_modeset 1351680 0 2024-08-20T22:15:55.5961697Z video 65536 1 nvidia_modeset 2024-08-20T22:15:55.5962230Z wmi 36864 1 video 2024-08-20T22:15:55.5962620Z nvidia_uvm 4706304 0 2024-08-20T22:15:55.5963088Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-08-20T22:15:55.5963562Z drm 602112 1 nvidia 2024-08-20T22:15:55.5963998Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:15:55.5964524Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:15:55.5965027Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:15:55.5965445Z xt_conntrack 16384 1 2024-08-20T22:15:55.5965819Z nft_chain_nat 16384 3 2024-08-20T22:15:55.5966192Z xt_MASQUERADE 20480 1 2024-08-20T22:15:55.5966635Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:15:55.5967115Z nf_conntrack_netlink 57344 0 2024-08-20T22:15:55.5967695Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:15:55.5968329Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:15:55.5969007Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:15:55.5969439Z xfrm_user 57344 1 2024-08-20T22:15:55.5969826Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:15:55.5970237Z xt_addrtype 16384 2 2024-08-20T22:15:55.5970602Z nft_compat 20480 4 2024-08-20T22:15:55.5971050Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:15:55.5971652Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:15:55.5972186Z br_netfilter 36864 0 2024-08-20T22:15:55.5972632Z bridge 307200 1 br_netfilter 2024-08-20T22:15:55.5973065Z stp 16384 1 bridge 2024-08-20T22:15:55.5973478Z llc 16384 2 bridge,stp 2024-08-20T22:15:55.5973895Z overlay 167936 0 2024-08-20T22:15:55.5974253Z tls 114688 0 2024-08-20T22:15:55.5974616Z nls_ascii 16384 1 2024-08-20T22:15:55.5974992Z nls_cp437 20480 1 2024-08-20T22:15:55.5975356Z vfat 24576 1 2024-08-20T22:15:55.5975714Z fat 86016 1 vfat 2024-08-20T22:15:55.5976098Z ena 167936 0 2024-08-20T22:15:55.5976468Z ghash_clmulni_intel 16384 0 2024-08-20T22:15:55.5976839Z ptp 36864 1 ena 2024-08-20T22:15:55.5977232Z pps_core 24576 1 ptp 2024-08-20T22:15:55.5977627Z aesni_intel 393216 0 2024-08-20T22:15:55.5977992Z i8042 45056 0 2024-08-20T22:15:55.5978354Z serio 28672 3 i8042 2024-08-20T22:15:55.5978774Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:15:55.5979319Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:15:55.5979986Z button 24576 0 2024-08-20T22:15:55.5980678Z sunrpc 692224 1 2024-08-20T22:15:55.5981346Z sch_fq_codel 20480 33 2024-08-20T22:15:55.5981915Z dm_mod 188416 0 2024-08-20T22:15:55.5982453Z fuse 163840 1 2024-08-20T22:15:55.5982928Z dax 45056 1 dm_mod 2024-08-20T22:15:55.5983477Z loop 36864 0 2024-08-20T22:15:55.5983971Z configfs 57344 1 2024-08-20T22:15:55.5984472Z dmi_sysfs 20480 0 2024-08-20T22:15:55.5984995Z crc32_pclmul 16384 0 2024-08-20T22:15:55.5985471Z crc32c_intel 24576 0 2024-08-20T22:15:55.5985960Z efivarfs 24576 1 2024-08-20T22:15:55.5986474Z + modinfo nvidia 2024-08-20T22:15:55.5987505Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:15:55.5988558Z alias: char-major-195-* 2024-08-20T22:15:55.5989078Z version: 550.54.15 2024-08-20T22:15:55.5989546Z supported: external 2024-08-20T22:15:55.5990014Z license: NVIDIA 2024-08-20T22:15:55.5990489Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:15:55.5991094Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:15:55.5991654Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:15:55.5992243Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:15:55.5992872Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:15:55.5993362Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:15:55.5993890Z depends: i2c-core,drm 2024-08-20T22:15:55.5994235Z retpoline: Y 2024-08-20T22:15:55.5994541Z name: nvidia 2024-08-20T22:15:55.5995139Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:15:55.5995820Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:15:55.5996437Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:15:55.5997021Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:15:55.5997463Z parm: NVreg_RmLogonRC:int 2024-08-20T22:15:55.5997885Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:15:55.5998322Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:15:55.5998898Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:15:55.5999329Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:15:55.5999841Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:15:55.6000384Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:15:55.6000853Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:15:55.6001275Z parm: NVreg_EnableMSI:int 2024-08-20T22:15:55.6001673Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:15:55.6002124Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:15:55.6002693Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:15:55.6003269Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:15:55.6003809Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:15:55.6004405Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:55.6004998Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:15:55.6005644Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:15:55.6006286Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:15:55.6006764Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:15:55.6007279Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:15:55.6007810Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:15:55.6008292Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:15:55.6008749Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:15:55.6009231Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:15:55.6009688Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:15:55.6010118Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:15:55.6010612Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:15:55.6011220Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:15:55.6011689Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:15:55.6012161Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:15:55.6012649Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:15:55.6013127Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:15:55.6013604Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:15:55.6014073Z parm: NVreg_RmMsg:charp 2024-08-20T22:15:55.6014482Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:15:55.6014948Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:15:55.6015427Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:15:55.6016018Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:15:55.6016608Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:15:55.6017182Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:15:55.6017735Z parm: rm_firmware_active:charp 2024-08-20T22:15:55.6018495Z + set +e 2024-08-20T22:15:55.6018927Z + nvidia-smi 2024-08-20T22:15:55.6467623Z Tue Aug 20 22:15:55 2024 2024-08-20T22:15:55.6468488Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:55.6469435Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:55.6470222Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:55.6471010Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:55.6471892Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:55.6472578Z | | | MIG M. | 2024-08-20T22:15:55.6473117Z |=========================================+========================+======================| 2024-08-20T22:15:55.6712059Z | 0 NVIDIA A10G On | 00000000:00:1B.0 Off | 0 | 2024-08-20T22:15:55.6713215Z | 0% 25C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:55.6713875Z | | | N/A | 2024-08-20T22:15:55.6714600Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:55.6935849Z | 1 NVIDIA A10G On | 00000000:00:1C.0 Off | 0 | 2024-08-20T22:15:55.6936805Z | 0% 24C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:55.6937459Z | | | N/A | 2024-08-20T22:15:55.6938176Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:55.7160301Z | 2 NVIDIA A10G On | 00000000:00:1D.0 Off | 0 | 2024-08-20T22:15:55.7161260Z | 0% 25C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:55.7161910Z | | | N/A | 2024-08-20T22:15:55.7162621Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:55.7382054Z | 3 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:55.7383331Z | 0% 26C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:55.7383973Z | | | N/A | 2024-08-20T22:15:55.7384678Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:55.7515960Z 2024-08-20T22:15:55.7518504Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:55.7519828Z | Processes: | 2024-08-20T22:15:55.7521246Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:55.7522620Z | ID ID Usage | 2024-08-20T22:15:55.7523323Z |=========================================================================================| 2024-08-20T22:15:55.7562523Z | No running processes found | 2024-08-20T22:15:55.7563524Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:56.9427410Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:15:56.9632820Z NVIDIA A10G 2024-08-20T22:15:56.9697340Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:15:56.9697910Z + '[' 0 -eq 0 ']' 2024-08-20T22:15:56.9698430Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:15:56.9698930Z + set -e 2024-08-20T22:15:56.9699214Z INFO: Ignoring allowed status 0 2024-08-20T22:15:56.9707984Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:15:56.9711938Z + sudo yum install -y yum-utils 2024-08-20T22:15:57.3942154Z Last metadata expiration check: 0:33:56 ago on Tue Aug 20 21:42:01 2024. 2024-08-20T22:15:57.4162564Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:15:57.4468589Z Dependencies resolved. 2024-08-20T22:15:57.4591130Z Nothing to do. 2024-08-20T22:15:57.4591543Z Complete! 2024-08-20T22:15:57.5406503Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:15:57.5407644Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:57.5408964Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:57.8284678Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:15:57.9022068Z + sudo yum install -y nvidia-docker2 2024-08-20T22:15:58.4158583Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:15:58.4383680Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:15:58.4679569Z Dependencies resolved. 2024-08-20T22:15:58.4805781Z Nothing to do. 2024-08-20T22:15:58.4806521Z Complete! 2024-08-20T22:15:58.6137583Z + sudo systemctl restart docker 2024-08-20T22:17:19.8787322Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:17:19.9274220Z Tue Aug 20 22:17:19 2024 2024-08-20T22:17:19.9274942Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:17:19.9275779Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:17:19.9276593Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:17:19.9277419Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:17:19.9278322Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:17:19.9279018Z | | | MIG M. | 2024-08-20T22:17:19.9281841Z |=========================================+========================+======================| 2024-08-20T22:17:19.9502514Z | 0 NVIDIA A10G On | 00000000:00:1B.0 Off | 0 | 2024-08-20T22:17:19.9503738Z | 0% 25C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:17:19.9504382Z | | | N/A | 2024-08-20T22:17:19.9505237Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:17:19.9684255Z | 1 NVIDIA A10G On | 00000000:00:1C.0 Off | 0 | 2024-08-20T22:17:19.9685724Z | 0% 24C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:17:19.9686983Z | | | N/A | 2024-08-20T22:17:19.9688368Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:17:19.9872174Z | 2 NVIDIA A10G On | 00000000:00:1D.0 Off | 0 | 2024-08-20T22:17:19.9872881Z | 0% 25C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:17:19.9873515Z | | | N/A | 2024-08-20T22:17:19.9874199Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:17:20.0054899Z | 3 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:17:20.0056223Z | 0% 26C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:17:20.0057360Z | | | N/A | 2024-08-20T22:17:20.0058615Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:17:20.0191062Z 2024-08-20T22:17:20.0191846Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:17:20.0192533Z | Processes: | 2024-08-20T22:17:20.0193231Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:17:20.0193919Z | ID ID Usage | 2024-08-20T22:17:20.0194493Z |=========================================================================================| 2024-08-20T22:17:20.0232212Z | No running processes found | 2024-08-20T22:17:20.0233005Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:17:22.1119801Z Command completed after 1 attempt(s). 2024-08-20T22:17:22.1202845Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:17:22.1203602Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:17:22.1204274Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:17:22.1204911Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:17:22.1219699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:17:22.1220201Z env: 2024-08-20T22:17:22.1220491Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:17:22.1221016Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:17:22.1221494Z ##[endgroup] 2024-08-20T22:17:22.3654259Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:17:22.3846656Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T22:17:22.3851812Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T22:17:22.5152124Z Prepare all required actions 2024-08-20T22:17:22.5152672Z Getting action download info 2024-08-20T22:17:22.6259196Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:17:22.8143708Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:17:22.9851079Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:17:22.9851729Z with: 2024-08-20T22:17:22.9852066Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:17:22.9852518Z s3-bucket: gha-artifacts 2024-08-20T22:17:22.9852865Z env: 2024-08-20T22:17:22.9853147Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:17:22.9853595Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:17:22.9854075Z ##[endgroup] 2024-08-20T22:17:22.9887564Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:17:22.9887998Z with: 2024-08-20T22:17:22.9888376Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:17:22.9888837Z s3-bucket: gha-artifacts 2024-08-20T22:17:22.9889183Z region: us-east-1 2024-08-20T22:17:22.9889482Z env: 2024-08-20T22:17:22.9889769Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:17:22.9890213Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:17:22.9890698Z ##[endgroup] 2024-08-20T22:17:23.4798175Z (node:480091) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:17:23.4798881Z 2024-08-20T22:17:23.4799139Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:17:23.4799857Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:17:23.4800719Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:17:23.5677006Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:17:23.5678082Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:17:50.2058565Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:17:50.2065325Z Artifact download has finished successfully 2024-08-20T22:17:50.2431268Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:17:50.2431714Z unzip -o artifacts.zip 2024-08-20T22:17:50.2441951Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:17:50.2442460Z env: 2024-08-20T22:17:50.2442733Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:17:50.2443176Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:17:50.2443647Z ##[endgroup] 2024-08-20T22:17:50.2495812Z Archive: artifacts.zip 2024-08-20T22:17:50.2497418Z creating: dist/ 2024-08-20T22:17:52.4024977Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:17:52.4025674Z creating: build/custom_test_artifacts/ 2024-08-20T22:17:52.4026351Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:17:52.4027128Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:17:52.4027980Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:17:52.4034888Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:17:52.4035836Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:17:52.4036784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:17:52.4037762Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:17:52.4038719Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:17:52.4039929Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:17:52.4041662Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:17:52.4042675Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:17:52.4043670Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:17:52.4044947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:17:52.4046596Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:17:52.4048504Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:17:52.4049790Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:17:52.4051307Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:17:52.4052684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:17:52.4053709Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:17:52.4054707Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:17:52.4094948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:17:52.4135020Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:17:52.4136451Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:17:52.4182262Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:17:52.4183667Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:17:52.4185204Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:17:52.4186771Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:17:52.4188277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:17:52.4189671Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:17:52.4191146Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:17:52.4192528Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:17:52.4193870Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:17:52.4195228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:17:52.4196445Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:17:52.4197639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:17:52.4198864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:17:52.4200093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:17:52.4201290Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:17:52.4259077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:17:52.4318978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:17:52.4320935Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:17:52.4321989Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:17:52.4323164Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:17:52.4324088Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:17:52.4325025Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:17:52.4326072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:17:52.4327245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:17:52.4328363Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:17:52.4329406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:17:52.4330560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:17:52.4331640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:17:52.4332750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:17:52.4333830Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:17:52.4334908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:17:52.4351720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:17:52.4497371Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:17:52.4498370Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:17:52.4499712Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:17:52.4502043Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:17:52.4504190Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:17:52.4506167Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:17:52.4508212Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:17:52.4509971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:17:52.4511108Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:17:52.4512237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:17:52.4513333Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:17:52.4527603Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:17:52.4610735Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:17:52.4611922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:17:52.4612998Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:17:52.4614227Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:17:52.4615149Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:17:52.4616047Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:17:52.4616964Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:17:52.4619551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:17:52.4621422Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:17:52.4622319Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:17:52.4741959Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:17:52.4805210Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:17:52.4805955Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:17:52.4806658Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:17:52.4807483Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:17:52.4814428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:17:52.4815346Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:17:52.4816273Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:17:52.4817267Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:17:52.4818370Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:17:52.4819458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:17:52.4820922Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:17:52.4821928Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:17:52.4822904Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:17:52.4824660Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:17:52.4826353Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:17:52.4828422Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:17:52.4829549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:17:52.4831156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:17:52.4832636Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:17:52.4833665Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:17:52.4834644Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:17:52.4874682Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:17:52.4914410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:17:52.4916063Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:17:52.4961888Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:17:52.4963270Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:17:52.4964842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:17:52.4966316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:17:52.4967713Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:17:52.4969213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:17:52.4970645Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:17:52.4972014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:17:52.4973373Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:17:52.4974642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:17:52.4975860Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:17:52.4977054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:17:52.4978275Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:17:52.4979489Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:17:52.4980708Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:17:52.5038224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:17:52.5098643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:17:52.5099858Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:17:52.5100886Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:17:52.5101753Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:17:52.5102651Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:17:52.5103592Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:17:52.5104648Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:17:52.5105827Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:17:52.5106963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:17:52.5108020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:17:52.5109126Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:17:52.5110260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:17:52.5111376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:17:52.5112473Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:17:52.5113561Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:17:52.5131381Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:17:52.5195891Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:17:52.5197073Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:17:52.5198137Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:17:52.5199109Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:17:52.5200167Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:17:52.5201064Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:17:52.5201952Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:17:52.5204145Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:17:52.5204929Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:17:52.5205866Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:17:52.5256977Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:17:52.5258472Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:17:52.5259573Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:17:52.5261481Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:17:52.5265196Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:17:52.5266197Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:17:52.5267186Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:17:52.5268234Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:17:52.5269281Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:17:52.5270438Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:17:52.5271949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:17:52.5273076Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:17:52.5274173Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:17:52.5275400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:17:52.5276838Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:17:52.5278686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:17:52.5279898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:17:52.5281482Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:17:52.5282873Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:17:52.5283975Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:17:52.5285060Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:17:52.5326355Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:17:52.5365587Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:17:52.5367516Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:17:52.5413012Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:17:52.5414479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:17:52.5416180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:17:52.5417698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:17:52.5419378Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:17:52.5420914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:17:52.5422358Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:17:52.5423808Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:17:52.5425208Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:17:52.5426556Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:17:52.5427836Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:17:52.5429114Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:17:52.5430409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:17:52.5431660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:17:52.5432923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:17:52.5489043Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:17:52.5551343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:17:52.5552586Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:17:52.5553633Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:17:52.5554560Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:17:52.5555517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:17:52.5556516Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:17:52.5557681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:17:52.5558927Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:17:52.5560180Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:17:52.5561308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:17:52.5562470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:17:52.5563812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:17:52.5564995Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:17:52.5566165Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:17:52.5567331Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:17:52.5568685Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:17:52.5694001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:17:52.5695166Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:17:52.5696342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:17:52.5697644Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:17:52.5698889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:17:52.5700125Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:17:52.5701414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:17:52.5702656Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:17:52.5703879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:17:52.5705101Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:17:52.5706319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:17:52.5724726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:17:52.5781235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:17:52.5783570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:17:52.5785630Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:17:52.5787506Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:17:52.5789261Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:17:52.5790385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:17:52.5791345Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:17:52.5792238Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:17:52.5793086Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:17:52.5793940Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:17:52.5894862Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:17:52.5938657Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:17:52.5939284Z creating: build/lib/ 2024-08-20T22:17:52.6028628Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:17:52.6495849Z inflating: build/lib/libprotobuf.a 2024-08-20T22:17:52.6505867Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:17:52.6514245Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:17:52.6522688Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:17:52.6523512Z inflating: build/lib/libclog.a 2024-08-20T22:17:52.6542486Z inflating: build/lib/libnnpack.a 2024-08-20T22:17:52.6545152Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:17:52.6611500Z inflating: build/lib/libgtest.a 2024-08-20T22:17:52.6686572Z inflating: build/lib/libbenchmark.a 2024-08-20T22:17:52.6750717Z inflating: build/lib/libasmjit.a 2024-08-20T22:17:52.6758489Z inflating: build/lib/libittnotify.a 2024-08-20T22:17:52.6786838Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:17:52.6915049Z inflating: build/lib/libgloo.a 2024-08-20T22:17:52.6935635Z inflating: build/lib/libfmt.a 2024-08-20T22:17:52.7031784Z inflating: build/lib/libc10.so 2024-08-20T22:17:52.7033791Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:17:52.7034788Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:17:52.7054732Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:17:52.7561586Z inflating: build/lib/libprotoc.a 2024-08-20T22:17:52.7580704Z inflating: build/lib/libgmock.a 2024-08-20T22:17:52.7581362Z inflating: build/lib/libgtest_main.a 2024-08-20T22:17:52.7582374Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:17:52.7972645Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:17:53.8198680Z inflating: build/lib/libdnnl.a 2024-08-20T22:17:53.8774374Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:17:53.8831955Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:17:53.8832559Z inflating: build/lib/libgmock_main.a 2024-08-20T22:17:54.0101217Z inflating: build/lib/libfbgemm.a 2024-08-20T22:17:54.0357764Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:17:54.0873153Z inflating: build/lib/libkineto.a 2024-08-20T22:17:54.1062840Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:17:54.1106075Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:17:54.1804299Z inflating: build/lib/libonnx.a 2024-08-20T22:17:56.6419342Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:17:56.6424968Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:17:56.6429289Z inflating: build/lib/libshm.so 2024-08-20T22:17:58.7180821Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:17:58.7182646Z inflating: build/lib/libtorch.so 2024-08-20T22:17:58.7186709Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:17:59.5403389Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:17:59.7417025Z inflating: build/lib/libtorch_python.so 2024-08-20T22:17:59.7490227Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:17:59.7510621Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:17:59.7537417Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:17:59.7563619Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:17:59.7598005Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:17:59.7598480Z creating: build/bin/ 2024-08-20T22:17:59.7649387Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:17:59.7701126Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:17:59.7752878Z inflating: build/bin/c10_Device_test 2024-08-20T22:17:59.7812361Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:17:59.7866819Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:17:59.7916027Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:17:59.7968052Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:17:59.8023173Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:17:59.8079430Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:17:59.8136762Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:17:59.8209090Z inflating: build/bin/c10_cow_test 2024-08-20T22:17:59.8262728Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:17:59.8311786Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:17:59.8362544Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:17:59.8413624Z inflating: build/bin/c10_Half_test 2024-08-20T22:17:59.8470597Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:17:59.8526466Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:17:59.8577372Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:17:59.8633377Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:17:59.8685645Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:17:59.8737658Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:17:59.8787043Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:17:59.8839442Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:17:59.8895198Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:17:59.8946491Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:17:59.9003576Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:17:59.9060328Z inflating: build/bin/c10_complex_test 2024-08-20T22:17:59.9112933Z inflating: build/bin/c10_exception_test 2024-08-20T22:17:59.9163967Z inflating: build/bin/c10_flags_test 2024-08-20T22:17:59.9214646Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:17:59.9382015Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:17:59.9434433Z inflating: build/bin/c10_irange_test 2024-08-20T22:17:59.9488329Z inflating: build/bin/c10_lazy_test 2024-08-20T22:17:59.9548417Z inflating: build/bin/c10_logging_test 2024-08-20T22:17:59.9610797Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:17:59.9686808Z inflating: build/bin/c10_optional_test 2024-08-20T22:17:59.9839311Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:17:59.9893675Z inflating: build/bin/c10_registry_test 2024-08-20T22:17:59.9946226Z inflating: build/bin/c10_ssize_test 2024-08-20T22:17:59.9998429Z inflating: build/bin/c10_string_util_test 2024-08-20T22:18:00.0050374Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:18:00.0108984Z inflating: build/bin/c10_string_view_test 2024-08-20T22:18:00.0158300Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:18:00.0215038Z inflating: build/bin/c10_typeid_test 2024-08-20T22:18:00.0664724Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:18:00.1113666Z inflating: build/bin/protoc 2024-08-20T22:18:00.1166893Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:18:00.1220860Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:18:00.1274027Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:18:00.1326693Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:18:00.1379706Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:18:00.1433268Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:18:00.1482659Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:18:00.1536124Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:18:00.1868878Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:18:00.2219255Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:18:00.2580254Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:18:00.2634441Z inflating: build/bin/BackoffTest 2024-08-20T22:18:00.2689940Z inflating: build/bin/FileStoreTest 2024-08-20T22:18:00.2745738Z inflating: build/bin/TCPStoreTest 2024-08-20T22:18:00.2800647Z inflating: build/bin/HashStoreTest 2024-08-20T22:18:00.2814485Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:18:00.2869147Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:18:00.2873651Z inflating: build/bin/torch_shm_manager 2024-08-20T22:18:00.2876649Z inflating: build/bin/example_allreduce 2024-08-20T22:18:00.2932694Z inflating: build/bin/test_dist_autograd 2024-08-20T22:18:00.3002078Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:18:00.3004838Z inflating: build/bin/parallel_benchmark 2024-08-20T22:18:00.3073743Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:18:00.3082632Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:18:00.3431969Z inflating: build/bin/test_lazy 2024-08-20T22:18:00.4628088Z inflating: build/bin/test_api 2024-08-20T22:18:00.4702535Z inflating: build/bin/Dict_test 2024-08-20T22:18:00.4755254Z inflating: build/bin/Dimname_test 2024-08-20T22:18:00.4820459Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:18:00.4878895Z inflating: build/bin/NamedTensor_test 2024-08-20T22:18:00.4938734Z inflating: build/bin/apply_utils_test 2024-08-20T22:18:00.4998647Z inflating: build/bin/atest 2024-08-20T22:18:00.5062198Z inflating: build/bin/basic 2024-08-20T22:18:00.5117555Z inflating: build/bin/broadcast_test 2024-08-20T22:18:00.5168999Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:18:00.5228255Z inflating: build/bin/cpu_generator_test 2024-08-20T22:18:00.5281901Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:18:00.5376353Z inflating: build/bin/cpu_rng_test 2024-08-20T22:18:00.5426611Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:18:00.5477732Z inflating: build/bin/dlconvertor_test 2024-08-20T22:18:00.5538403Z inflating: build/bin/extension_backend_test 2024-08-20T22:18:00.5595226Z inflating: build/bin/half_test 2024-08-20T22:18:00.5691910Z inflating: build/bin/ivalue_test 2024-08-20T22:18:00.5742398Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:18:00.5797202Z inflating: build/bin/math_kernel_test 2024-08-20T22:18:00.5853163Z inflating: build/bin/memory_format_test 2024-08-20T22:18:00.5907058Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:18:00.5961375Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:18:00.6018831Z inflating: build/bin/native_test 2024-08-20T22:18:00.6069545Z inflating: build/bin/operator_name_test 2024-08-20T22:18:00.6122133Z inflating: build/bin/operators_test 2024-08-20T22:18:00.6175614Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:18:00.6242872Z inflating: build/bin/pow_test 2024-08-20T22:18:00.6300603Z inflating: build/bin/quantized_test 2024-08-20T22:18:00.6352183Z inflating: build/bin/reduce_ops_test 2024-08-20T22:18:00.6404342Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:18:00.6461980Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:18:00.6521426Z inflating: build/bin/scalar_test 2024-08-20T22:18:00.6573998Z inflating: build/bin/StorageUtils_test 2024-08-20T22:18:00.6626996Z inflating: build/bin/stride_properties_test 2024-08-20T22:18:00.6705953Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:18:00.6760656Z inflating: build/bin/test_parallel 2024-08-20T22:18:00.6764362Z inflating: build/bin/thread_init_test 2024-08-20T22:18:00.6820470Z inflating: build/bin/type_ptr_test 2024-08-20T22:18:00.6880923Z inflating: build/bin/type_test 2024-08-20T22:18:00.6933691Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:18:00.6935416Z inflating: build/bin/verify_api_visibility 2024-08-20T22:18:00.7005314Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:18:00.7057727Z inflating: build/bin/weakref_test 2024-08-20T22:18:00.7109758Z inflating: build/bin/wrapdim_test 2024-08-20T22:18:00.7162225Z inflating: build/bin/xla_tensor_test 2024-08-20T22:18:00.7223259Z inflating: build/bin/IListRef_test 2024-08-20T22:18:00.7330502Z inflating: build/bin/List_test 2024-08-20T22:18:00.7397036Z inflating: build/bin/KernelFunction_test 2024-08-20T22:18:00.7520206Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:18:00.7618087Z inflating: build/bin/kernel_function_test 2024-08-20T22:18:00.7748347Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:18:00.7854134Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:18:00.7916421Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:18:00.8013886Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:18:00.8066159Z inflating: build/bin/CppSignature_test 2024-08-20T22:18:00.8122713Z inflating: build/bin/backend_fallback_test 2024-08-20T22:18:00.8172397Z inflating: build/bin/op_allowlist_test 2024-08-20T22:18:00.8477599Z inflating: build/bin/op_registration_test 2024-08-20T22:18:00.8542117Z inflating: build/bin/inline_container_test 2024-08-20T22:18:00.8595574Z inflating: build/bin/cuda_apply_test 2024-08-20T22:18:00.8648698Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:18:00.8707568Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:18:00.8763072Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:18:00.8833616Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:18:00.8893641Z inflating: build/bin/cuda_complex_test 2024-08-20T22:18:00.8943862Z inflating: build/bin/cuda_device_test 2024-08-20T22:18:00.9001989Z inflating: build/bin/cuda_cub_test 2024-08-20T22:18:00.9054308Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:18:00.9119793Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:18:00.9176951Z inflating: build/bin/cuda_generator_test 2024-08-20T22:18:00.9227349Z inflating: build/bin/cuda_half_test 2024-08-20T22:18:00.9278797Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:18:00.9329845Z inflating: build/bin/cuda_optional_test 2024-08-20T22:18:00.9381650Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:18:00.9435215Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:18:00.9485773Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:18:00.9547801Z inflating: build/bin/cuda_stream_test 2024-08-20T22:18:00.9597422Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:18:00.9650282Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:18:00.9665441Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:18:00.9732374Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:18:00.9790854Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:18:00.9855942Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:18:00.9921677Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:18:01.0755323Z inflating: build/bin/test_tensorexpr 2024-08-20T22:18:01.1344105Z inflating: build/bin/test_jit 2024-08-20T22:18:01.1344970Z creating: .additional_ci_files/ 2024-08-20T22:18:01.1405397Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:18:01.1648690Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:18:01.1720739Z ##[group]Run rm artifacts.zip 2024-08-20T22:18:01.1721153Z rm artifacts.zip 2024-08-20T22:18:01.1730289Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:01.1730790Z env: 2024-08-20T22:18:01.1731073Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:01.1731568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:01.1732044Z ##[endgroup] 2024-08-20T22:18:01.5005544Z ##[group]Run df -H 2024-08-20T22:18:01.5005871Z df -H 2024-08-20T22:18:01.5015025Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:01.5015518Z env: 2024-08-20T22:18:01.5015808Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:01.5016266Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:01.5016742Z ##[endgroup] 2024-08-20T22:18:01.5068739Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:18:01.5069854Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:18:01.5070877Z tmpfs 101G 50k 101G 1% /dev/shm 2024-08-20T22:18:01.5071974Z tmpfs 41G 697k 41G 1% /run 2024-08-20T22:18:01.5072488Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:18:01.5073144Z tmpfs 101G 46k 101G 1% /tmp 2024-08-20T22:18:01.5073762Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:18:01.5074268Z tmpfs 21G 0 21G 0% /run/user/0 2024-08-20T22:18:01.5112373Z Prepare all required actions 2024-08-20T22:18:01.5112923Z Getting action download info 2024-08-20T22:18:01.6295323Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:18:01.6295777Z with: 2024-08-20T22:18:01.6296232Z env: 2024-08-20T22:18:01.6296512Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:01.6296969Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:01.6297462Z ##[endgroup] 2024-08-20T22:18:01.6331176Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:18:01.6331658Z with: 2024-08-20T22:18:01.6331930Z name: td_results 2024-08-20T22:18:01.6332249Z s3-bucket: gha-artifacts 2024-08-20T22:18:01.6332599Z region: us-east-1 2024-08-20T22:18:01.6332894Z env: 2024-08-20T22:18:01.6333160Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:01.6333612Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:01.6334092Z ##[endgroup] 2024-08-20T22:18:02.1090356Z (node:480118) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:18:02.1091031Z 2024-08-20T22:18:02.1091299Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:18:02.1092071Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:18:02.1092917Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:18:02.1689093Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:18:02.1695729Z Artifact download has finished successfully 2024-08-20T22:18:02.2024200Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:18:02.2024687Z mkdir -p .additional_ci_files 2024-08-20T22:18:02.2025245Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:18:02.2034723Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:02.2035219Z env: 2024-08-20T22:18:02.2035489Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:02.2035949Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:02.2036417Z ##[endgroup] 2024-08-20T22:18:02.2100404Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:18:02.2117097Z ##[error]Process completed with exit code 1. 2024-08-20T22:18:02.2152728Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:18:02.2153210Z .github/scripts/parse_ref.py 2024-08-20T22:18:02.2162005Z shell: /usr/bin/bash -e {0} 2024-08-20T22:18:02.2162350Z env: 2024-08-20T22:18:02.2162622Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:02.2163070Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:02.2163537Z ##[endgroup] 2024-08-20T22:18:02.2456996Z Prepare all required actions 2024-08-20T22:18:02.2496756Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:18:02.2497219Z with: 2024-08-20T22:18:02.2497779Z github-token: *** 2024-08-20T22:18:02.2498082Z env: 2024-08-20T22:18:02.2498354Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:02.2498800Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:02.2499291Z ##[endgroup] 2024-08-20T22:18:02.2516388Z ##[group]Run set -eux 2024-08-20T22:18:02.2516726Z set -eux 2024-08-20T22:18:02.2517316Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:18:02.2526900Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:02.2527396Z env: 2024-08-20T22:18:02.2527672Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:02.2528120Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:02.2528756Z GITHUB_TOKEN: *** 2024-08-20T22:18:02.2529058Z ##[endgroup] 2024-08-20T22:18:02.2557060Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-01a7d20fbad5d17a0 2024-08-20T22:18:03.8715527Z setting job-id=29026527401 2024-08-20T22:18:03.8716486Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:03.8936729Z Prepare all required actions 2024-08-20T22:18:03.8937177Z Getting action download info 2024-08-20T22:18:04.0605887Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:18:04.0606352Z with: 2024-08-20T22:18:04.0606813Z github-token: *** 2024-08-20T22:18:04.0615766Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:18:04.0625401Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:04.0626153Z env: 2024-08-20T22:18:04.0626438Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:04.0626897Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:04.0627371Z ##[endgroup] 2024-08-20T22:18:04.0669389Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:18:04.0669939Z with: 2024-08-20T22:18:04.0670209Z shell: bash 2024-08-20T22:18:04.0670521Z timeout_minutes: 10 2024-08-20T22:18:04.0670842Z max_attempts: 5 2024-08-20T22:18:04.0671163Z retry_wait_seconds: 30 2024-08-20T22:18:04.0672274Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:18:04.0673442Z polling_interval_seconds: 1 2024-08-20T22:18:04.0673823Z warning_on_retry: true 2024-08-20T22:18:04.0674167Z continue_on_error: false 2024-08-20T22:18:04.0674505Z env: 2024-08-20T22:18:04.0674790Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:04.0675235Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:04.0675878Z GITHUB_TOKEN: *** 2024-08-20T22:18:04.0676204Z ##[endgroup] 2024-08-20T22:18:04.1451408Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:18:04.3842817Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:18:04.4033456Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T22:18:04.4037445Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T22:18:04.4159666Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:18:04.4167491Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T22:18:04.4171368Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:18:04.4176541Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:18:05.1222046Z Command completed after 1 attempt(s). 2024-08-20T22:18:05.1282699Z ##[group]Run set -x 2024-08-20T22:18:05.1283065Z set -x 2024-08-20T22:18:05.1283413Z  2024-08-20T22:18:05.1283970Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:18:05.1284653Z # in runner workspace 2024-08-20T22:18:05.1285197Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:18:05.1294443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:05.1294937Z env: 2024-08-20T22:18:05.1295212Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:05.1295669Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:05.1296144Z ##[endgroup] 2024-08-20T22:18:05.1326138Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:18:05.1572302Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:18:05.1572880Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:18:05.1573363Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:18:05.1573780Z  2024-08-20T22:18:05.1574329Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:18:05.1575007Z # in runner workspace 2024-08-20T22:18:05.1575586Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:18:05.1576238Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:18:05.1576703Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:18:05.1585861Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:18:05.1595219Z  --selected-test-configs "" \ 2024-08-20T22:18:05.1595687Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:18:05.1596120Z  --tag "${TAG}" \ 2024-08-20T22:18:05.1596523Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:18:05.1596967Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:18:05.1597412Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:18:05.1606847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:05.1607345Z env: 2024-08-20T22:18:05.1607620Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:05.1608075Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:05.1608769Z GITHUB_TOKEN: *** 2024-08-20T22:18:05.1609454Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:05.1610211Z PR_NUMBER: 2024-08-20T22:18:05.1610523Z TAG: ciflow/inductor/133712 2024-08-20T22:18:05.1610893Z EVENT_NAME: push 2024-08-20T22:18:05.1611207Z SCHEDULE: 2024-08-20T22:18:05.1611491Z HEAD_BRANCH: 2024-08-20T22:18:05.1611817Z ##[endgroup] 2024-08-20T22:18:05.1640683Z Workflow: inductor 2024-08-20T22:18:05.1641687Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:05.4087710Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:18:05.5562321Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:18:05.5562816Z echo "Filtered matrix:" 2024-08-20T22:18:05.5571909Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:18:05.5581354Z  2024-08-20T22:18:05.5581638Z echo 2024-08-20T22:18:05.5582025Z echo "Is the current job unstable? False" 2024-08-20T22:18:05.5582491Z  2024-08-20T22:18:05.5582773Z echo 2024-08-20T22:18:05.5583141Z echo "Is keep-going label set? False" 2024-08-20T22:18:05.5583588Z  2024-08-20T22:18:05.5583862Z echo 2024-08-20T22:18:05.5584184Z echo "Renabled issues? " 2024-08-20T22:18:05.5593110Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:05.5593604Z env: 2024-08-20T22:18:05.5593885Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:05.5594337Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:05.5594814Z ##[endgroup] 2024-08-20T22:18:05.5625218Z Filtered matrix: 2024-08-20T22:18:05.5635882Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:18:05.5644797Z 2024-08-20T22:18:05.5644949Z Is the current job unstable? False 2024-08-20T22:18:05.5645239Z 2024-08-20T22:18:05.5645571Z Is keep-going label set? False 2024-08-20T22:18:05.5645829Z 2024-08-20T22:18:05.5645966Z Renabled issues? 2024-08-20T22:18:05.5689049Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:18:05.5689739Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:18:05.5698010Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:05.5698507Z env: 2024-08-20T22:18:05.5698781Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:05.5699226Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:05.5699706Z JOB_TIMEOUT: 240 2024-08-20T22:18:05.5700003Z ##[endgroup] 2024-08-20T22:18:05.5781725Z ##[group]Run set -x 2024-08-20T22:18:05.5782128Z set -x 2024-08-20T22:18:05.5782423Z  2024-08-20T22:18:05.5782772Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:18:05.5783302Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:18:05.5783853Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:18:05.5784523Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:18:05.5784930Z else 2024-08-20T22:18:05.5785269Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:18:05.5785689Z fi 2024-08-20T22:18:05.5785960Z  2024-08-20T22:18:05.5786424Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:18:05.5787162Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:18:05.5787831Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:18:05.5788397Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:18:05.5788849Z container_name=$(docker run \ 2024-08-20T22:18:05.5789274Z  ${GPU_FLAG:-} \ 2024-08-20T22:18:05.5789640Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:18:05.5790037Z  -e PR_NUMBER \ 2024-08-20T22:18:05.5790390Z  -e GITHUB_ACTIONS \ 2024-08-20T22:18:05.5790778Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:18:05.5791189Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:18:05.5791573Z  -e GITHUB_JOB \ 2024-08-20T22:18:05.5791927Z  -e GITHUB_RUN_ID \ 2024-08-20T22:18:05.5792334Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:18:05.5792762Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:18:05.5793156Z  -e JOB_ID \ 2024-08-20T22:18:05.5793492Z  -e JOB_NAME \ 2024-08-20T22:18:05.5793843Z  -e BASE_SHA \ 2024-08-20T22:18:05.5794197Z  -e BRANCH \ 2024-08-20T22:18:05.5794523Z  -e SHA1 \ 2024-08-20T22:18:05.5794876Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:18:05.5795283Z  -e IN_WHEEL_TEST \ 2024-08-20T22:18:05.5795662Z  -e SHARD_NUMBER \ 2024-08-20T22:18:05.5796021Z  -e TEST_CONFIG \ 2024-08-20T22:18:05.5796395Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:18:05.5796791Z  -e REENABLED_ISSUES \ 2024-08-20T22:18:05.5797203Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:18:05.5797627Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:18:05.5798029Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:18:05.5798425Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:18:05.5798804Z  -e NO_TD \ 2024-08-20T22:18:05.5799142Z  -e TD_DISTRIBUTED \ 2024-08-20T22:18:05.5799527Z  -e PR_LABELS \ 2024-08-20T22:18:05.5799938Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:18:05.5800385Z  -e SCCACHE_BUCKET \ 2024-08-20T22:18:05.5800784Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:18:05.5801190Z  -e XLA_CUDA \ 2024-08-20T22:18:05.5801597Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:18:05.5802099Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:18:05.5802665Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:18:05.5803189Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:18:05.5803660Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:18:05.5804110Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:18:05.5804551Z  -e DASHBOARD_TAG \ 2024-08-20T22:18:05.5805030Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:18:05.5805597Z  --security-opt seccomp=unconfined \ 2024-08-20T22:18:05.5806064Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:18:05.5806471Z  --ipc=host \ 2024-08-20T22:18:05.5806848Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:18:05.5807260Z  --tty \ 2024-08-20T22:18:05.5807579Z  --detach \ 2024-08-20T22:18:05.5808081Z  --name="${container_name}" \ 2024-08-20T22:18:05.5808525Z  --user jenkins \ 2024-08-20T22:18:05.5809031Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:18:05.5809607Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:18:05.5810063Z  "${DOCKER_IMAGE}" 2024-08-20T22:18:05.5810428Z ) 2024-08-20T22:18:05.5810854Z # Propagate download.pytorch.org IP to container 2024-08-20T22:18:05.5811855Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:18:05.5812882Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:18:05.5813822Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:18:05.5822870Z shell: /usr/bin/bash -e {0} 2024-08-20T22:18:05.5823229Z env: 2024-08-20T22:18:05.5823514Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:05.5823989Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:05.5824602Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:18:05.5825088Z PR_NUMBER: 2024-08-20T22:18:05.5825420Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:18:05.5825848Z GITHUB_WORKFLOW: inductor 2024-08-20T22:18:05.5826209Z GITHUB_JOB: test 2024-08-20T22:18:05.5826523Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:18:05.5826898Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:18:05.5827259Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:18:05.5827598Z JOB_ID: 29026527401 2024-08-20T22:18:05.5828293Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:05.5829053Z BRANCH: 2024-08-20T22:18:05.5829405Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:05.5829936Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:05.5830419Z TEST_CONFIG: inductor_distributed 2024-08-20T22:18:05.5830829Z SHARD_NUMBER: 1 2024-08-20T22:18:05.5831140Z NUM_TEST_SHARDS: 1 2024-08-20T22:18:05.5831479Z REENABLED_ISSUES: 2024-08-20T22:18:05.5831819Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:18:05.5832212Z VERBOSE_TEST_LOGS: False 2024-08-20T22:18:05.5832579Z TEST_SHOWLOCALS: False 2024-08-20T22:18:05.5832925Z NO_TEST_TIMEOUT: False 2024-08-20T22:18:05.5833277Z NO_TD: False 2024-08-20T22:18:05.5833591Z TD_DISTRIBUTED: False 2024-08-20T22:18:05.5834028Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:18:05.5834513Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:18:05.5834904Z SHM_SIZE: 2g 2024-08-20T22:18:05.5835980Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:05.5837136Z XLA_CUDA: 2024-08-20T22:18:05.5837616Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:18:05.5838239Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:18:05.5838685Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:18:05.5839097Z DASHBOARD_TAG: 2024-08-20T22:18:05.5839614Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:18:05.5840000Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:18:05.5840381Z ##[endgroup] 2024-08-20T22:18:05.5871071Z + [[ inductor_distributed == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:18:05.5871850Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:18:05.5872388Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:18:05.5881202Z +++ nproc --ignore=2 2024-08-20T22:18:05.5899826Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:14.4321220Z + container_name=2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:18:14.4324044Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:18:14.4325932Z + docker exec -i 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:18:14.5787650Z + echo DOCKER_CONTAINER_ID=2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:18:14.5792803Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:18:14.5796146Z + docker exec -t 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:18:14.9828933Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:18:15.3113492Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:18:15.3117810Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:18:15.3121955Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:18:15.3124938Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:18:15.3126731Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:18:15.3130651Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:18:15.3146840Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:18:15.3159870Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:18:15.3177315Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:18:15.3566779Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:18:15.9316376Z Installing collected packages: torch 2024-08-20T22:18:26.0091892Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:18:26.0093628Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:18:26.0094779Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:18:26.0993766Z + export TERM=vt100 2024-08-20T22:18:26.0994119Z + TERM=vt100 2024-08-20T22:18:26.0997667Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:18:26.1007834Z + source .ci/pytorch/common.sh 2024-08-20T22:18:26.1012760Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:18:26.1024287Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:18:26.1026621Z +++ declare -f -t trap_add 2024-08-20T22:18:26.1031513Z ++ set -ex 2024-08-20T22:18:26.1032063Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:18:26.1032658Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:18:26.1033169Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:18:26.1036605Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:18:26.1057081Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:18:26.1057686Z + trap_add cleanup_workspace EXIT 2024-08-20T22:18:26.1058194Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:18:26.1058584Z + shift 2024-08-20T22:18:26.1058875Z + for trap_add_name in "$@" 2024-08-20T22:18:26.1069801Z +++ trap -p EXIT 2024-08-20T22:18:26.1073684Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:18:26.1074122Z +++ extract_trap_cmd 2024-08-20T22:18:26.1074496Z +++ printf '%s\n' '' 2024-08-20T22:18:26.1074891Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:18:26.1077336Z + trap -- ' 2024-08-20T22:18:26.1077748Z cleanup_workspace' EXIT 2024-08-20T22:18:26.1078225Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:18:26.8194356Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:18:26.8220609Z + echo 'Environment variables:' 2024-08-20T22:18:26.8221543Z Environment variables: 2024-08-20T22:18:26.8222181Z + env 2024-08-20T22:18:26.8230877Z INSTALLED_DB=yes 2024-08-20T22:18:26.8231395Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:18:26.8231870Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:18:26.8232267Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:18:26.8232877Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:18:26.8233457Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:18:26.8233946Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:18:26.8234464Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:18:26.8235046Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:18:26.8235536Z HOSTNAME=2fda783812ab 2024-08-20T22:18:26.8239446Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8240453Z GITHUB_ACTION=__self 2024-08-20T22:18:26.8240848Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:18:26.8245315Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:18:26.8249826Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:18:26.8250486Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:18:26.8250947Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:18:26.8251386Z TEST_CONFIG=inductor_distributed 2024-08-20T22:18:26.8251822Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:18:26.8252346Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:18:26.8252952Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:18:26.8253488Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:18:26.8253906Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:18:26.8254532Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:18:26.8254996Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:18:26.8255481Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:18:26.8255939Z GITHUB_REF_TYPE=tag 2024-08-20T22:18:26.8256260Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:18:26.8256636Z NCCL_VERSION=2.17.1-1 2024-08-20T22:18:26.8257096Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8257837Z XLA_CUDA= 2024-08-20T22:18:26.8258434Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:18:26.8261442Z *** 2024-08-20T22:18:26.8261817Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:18:26.8262234Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:18:26.8262672Z GITHUB_ACTIONS=true 2024-08-20T22:18:26.8263006Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:26.8263528Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:18:26.8264069Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:18:26.8264547Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8265063Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:18:26.8265548Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8266333Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:18:26.8267047Z UCC_HOME=/usr 2024-08-20T22:18:26.8267433Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:18:26.8267842Z VERBOSE_TEST_LOGS=False 2024-08-20T22:18:26.8268188Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:18:26.8268651Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:18:26.8269144Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:18:26.8269614Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:18:26.8269988Z SHARD_NUMBER=1 2024-08-20T22:18:26.8270296Z GITHUB_REF_PROTECTED=false 2024-08-20T22:18:26.8270642Z HOME=/var/lib/jenkins 2024-08-20T22:18:26.8271005Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:18:26.8271450Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:18:26.8271973Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:18:26.8272447Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:18:26.8272818Z CUDA_VERSION=12.1.1 2024-08-20T22:18:26.8273247Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:18:26.8273680Z NUM_TEST_SHARDS=1 2024-08-20T22:18:26.8273971Z UCX_HOME=/usr 2024-08-20T22:18:26.8274488Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:18:26.8275561Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8276811Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:26.8278042Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8279143Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:18:26.8279793Z GITHUB_EVENT_NAME=push 2024-08-20T22:18:26.8280123Z DASHBOARD_TAG= 2024-08-20T22:18:26.8280421Z GITHUB_RUN_ID=10479310725 2024-08-20T22:18:26.8280880Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:18:26.8281413Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:18:26.8282402Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8283292Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:18:26.8283700Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:18:26.8284075Z PR_NUMBER= 2024-08-20T22:18:26.8284360Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:18:26.8284684Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:18:26.8285121Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:18:26.8285582Z TERM=vt100 2024-08-20T22:18:26.8285916Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:18:26.8286321Z INSTALLED_VISION=yes 2024-08-20T22:18:26.8286639Z BRANCH= 2024-08-20T22:18:26.8286925Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:18:26.8287447Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:18:26.8287867Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:18:26.8288618Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:18:26.8289450Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:18:26.8289961Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:18:26.8290551Z REENABLED_ISSUES= 2024-08-20T22:18:26.8290859Z SHLVL=1 2024-08-20T22:18:26.8291133Z MAX_JOBS=46 2024-08-20T22:18:26.8291450Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:18:26.8291819Z NVARCH=x86_64 2024-08-20T22:18:26.8292124Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:18:26.8292598Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8293175Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:18:26.8293703Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:18:26.8294426Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:18:26.8295046Z GITHUB_JOB=test 2024-08-20T22:18:26.8295480Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:18:26.8296075Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:18:26.8296621Z NO_TEST_TIMEOUT=False 2024-08-20T22:18:26.8296956Z TD_DISTRIBUTED=False 2024-08-20T22:18:26.8297363Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:18:26.8297824Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:18:26.8298270Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:18:26.8298660Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:18:26.8299031Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:18:26.8299402Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:18:26.8300547Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:18:26.8301753Z GITHUB_BASE_REF= 2024-08-20T22:18:26.8302078Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:18:26.8302444Z CI=true 2024-08-20T22:18:26.8302759Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:18:26.8303180Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:18:26.8303548Z JOB_ID=29026527401 2024-08-20T22:18:26.8303859Z INSTALLED_PROTOBUF=yes 2024-08-20T22:18:26.8304183Z GITHUB_HEAD_REF= 2024-08-20T22:18:26.8304487Z GITHUB_ACTION_REF= 2024-08-20T22:18:26.8304917Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:18:26.8305371Z TEST_SHOWLOCALS=False 2024-08-20T22:18:26.8305715Z GITHUB_WORKFLOW=inductor 2024-08-20T22:18:26.8306066Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:18:26.8306961Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8307749Z NO_TD=False 2024-08-20T22:18:26.8308056Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:18:26.8308412Z _=/usr/bin/env 2024-08-20T22:18:26.8308897Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:18:26.8462354Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:18:26.8463338Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:18:26.8464310Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:18:26.8465266Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:18:26.8465915Z + BUILD_DIR=build 2024-08-20T22:18:26.8466244Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:18:26.8466637Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:18:26.8466977Z + SHARD_NUMBER=1 2024-08-20T22:18:26.8467305Z + NUM_TEST_SHARDS=1 2024-08-20T22:18:26.8467635Z + export VALGRIND=ON 2024-08-20T22:18:26.8467950Z + VALGRIND=ON 2024-08-20T22:18:26.8468419Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:18:26.8468960Z + [[ 0 == \1 ]] 2024-08-20T22:18:26.8469260Z + [[ False == \1 ]] 2024-08-20T22:18:26.8469799Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:18:26.8470309Z ++ realpath build/custom_test_artifacts 2024-08-20T22:18:26.8480958Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:18:26.8481663Z + [[ -n '' ]] 2024-08-20T22:18:26.8482029Z + echo 'Environment variables' 2024-08-20T22:18:26.8482402Z Environment variables 2024-08-20T22:18:26.8482713Z + env 2024-08-20T22:18:26.8491714Z INSTALLED_DB=yes 2024-08-20T22:18:26.8492166Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:18:26.8492765Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:18:26.8493213Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:18:26.8493951Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:18:26.8494688Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:18:26.8495285Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:18:26.8495799Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:18:26.8496384Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:18:26.8496885Z HOSTNAME=2fda783812ab 2024-08-20T22:18:26.8497878Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8498711Z GITHUB_ACTION=__self 2024-08-20T22:18:26.8499055Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:18:26.8503346Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:18:26.8507129Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:18:26.8507627Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:18:26.8507978Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:18:26.8508322Z TEST_CONFIG=inductor_distributed 2024-08-20T22:18:26.8508720Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:18:26.8509191Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:18:26.8509645Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:18:26.8510092Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:18:26.8510494Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:18:26.8510885Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:18:26.8511328Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:18:26.8511813Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:18:26.8512271Z GITHUB_REF_TYPE=tag 2024-08-20T22:18:26.8512602Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:18:26.8512979Z NCCL_VERSION=2.17.1-1 2024-08-20T22:18:26.8513371Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8513817Z XLA_CUDA= 2024-08-20T22:18:26.8514269Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:18:26.8514655Z *** 2024-08-20T22:18:26.8514940Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:18:26.8515337Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:18:26.8515706Z GITHUB_ACTIONS=true 2024-08-20T22:18:26.8516038Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:18:26.8516533Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:18:26.8517086Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:18:26.8517602Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8518388Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:18:26.8518896Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8519679Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:18:26.8520377Z UCC_HOME=/usr 2024-08-20T22:18:26.8520723Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:18:26.8521118Z VERBOSE_TEST_LOGS=False 2024-08-20T22:18:26.8521708Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:18:26.8522179Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:18:26.8522660Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:18:26.8523123Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:18:26.8523496Z SHARD_NUMBER=1 2024-08-20T22:18:26.8523807Z GITHUB_REF_PROTECTED=false 2024-08-20T22:18:26.8524145Z HOME=/var/lib/jenkins 2024-08-20T22:18:26.8524629Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:18:26.8525062Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:18:26.8525524Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:18:26.8526001Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:18:26.8526371Z CUDA_VERSION=12.1.1 2024-08-20T22:18:26.8526797Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:18:26.8527226Z NUM_TEST_SHARDS=1 2024-08-20T22:18:26.8527532Z UCX_HOME=/usr 2024-08-20T22:18:26.8528044Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:18:26.8529109Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8530367Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, amz2023.linux.g5.12xlarge.nvidia.gpu) 2024-08-20T22:18:26.8531597Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8532690Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:18:26.8533334Z GITHUB_EVENT_NAME=push 2024-08-20T22:18:26.8533651Z DASHBOARD_TAG= 2024-08-20T22:18:26.8533962Z GITHUB_RUN_ID=10479310725 2024-08-20T22:18:26.8534421Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:18:26.8534958Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:18:26.8535930Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8536827Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:18:26.8537239Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:18:26.8537608Z PR_NUMBER= 2024-08-20T22:18:26.8537887Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:18:26.8538196Z VALGRIND=ON 2024-08-20T22:18:26.8538493Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:18:26.8538924Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:18:26.8539377Z TERM=vt100 2024-08-20T22:18:26.8539716Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:18:26.8540123Z INSTALLED_VISION=yes 2024-08-20T22:18:26.8540425Z BRANCH= 2024-08-20T22:18:26.8540704Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:18:26.8541201Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:18:26.8541619Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:18:26.8542362Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:18:26.8543101Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:18:26.8543579Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:18:26.8544053Z REENABLED_ISSUES= 2024-08-20T22:18:26.8544345Z SHLVL=1 2024-08-20T22:18:26.8544604Z MAX_JOBS=46 2024-08-20T22:18:26.8544916Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:18:26.8545261Z NVARCH=x86_64 2024-08-20T22:18:26.8545558Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:18:26.8546012Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:18:26.8546543Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:18:26.8547028Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:18:26.8547732Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:18:26.8548297Z GITHUB_JOB=test 2024-08-20T22:18:26.8548697Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:18:26.8549236Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:18:26.8549738Z NO_TEST_TIMEOUT=False 2024-08-20T22:18:26.8550064Z TD_DISTRIBUTED=False 2024-08-20T22:18:26.8550431Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:18:26.8550966Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:18:26.8551388Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:18:26.8551745Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:18:26.8552080Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:18:26.8552427Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:18:26.8553455Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:18:26.8554581Z GITHUB_BASE_REF= 2024-08-20T22:18:26.8554899Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:18:26.8555268Z CI=true 2024-08-20T22:18:26.8555584Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:18:26.8555996Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:18:26.8556358Z JOB_ID=29026527401 2024-08-20T22:18:26.8556671Z INSTALLED_PROTOBUF=yes 2024-08-20T22:18:26.8556995Z GITHUB_HEAD_REF= 2024-08-20T22:18:26.8557284Z GITHUB_ACTION_REF= 2024-08-20T22:18:26.8557716Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:18:26.8558172Z TEST_SHOWLOCALS=False 2024-08-20T22:18:26.8558508Z GITHUB_WORKFLOW=inductor 2024-08-20T22:18:26.8558855Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:18:26.8559750Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_71155439-d8b5-4dc4-8a2c-0310de63c26a 2024-08-20T22:18:26.8560547Z NO_TD=False 2024-08-20T22:18:26.8560841Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:18:26.8561190Z _=/usr/bin/env 2024-08-20T22:18:26.8561535Z + echo 'Testing pytorch' 2024-08-20T22:18:26.8561878Z Testing pytorch 2024-08-20T22:18:26.8562199Z + export LANG=C.UTF-8 2024-08-20T22:18:26.8562541Z + LANG=C.UTF-8 2024-08-20T22:18:26.8562837Z + PR_NUMBER= 2024-08-20T22:18:26.8563182Z + [[ inductor_distributed == \d\e\f\a\u\l\t ]] 2024-08-20T22:18:26.8563692Z + [[ inductor_distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:18:26.8564188Z + [[ inductor_distributed == \s\l\o\w ]] 2024-08-20T22:18:26.8564819Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:18:26.8565510Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:18:26.8566046Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:18:26.8566512Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:18:26.8566957Z + [[ inductor_distributed == *crossref* ]] 2024-08-20T22:18:26.8567539Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:18:26.8568164Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:18:26.8568813Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:18:26.8569378Z + pip_install --user ninja==1.10.2 2024-08-20T22:18:26.8569911Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:18:29.4475897Z Collecting ninja==1.10.2 2024-08-20T22:18:29.4652300Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:18:29.5701835Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:18:30.2075234Z Installing collected packages: ninja 2024-08-20T22:18:30.2155797Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:18:30.2157106Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:18:30.2193029Z Successfully installed ninja-1.10.2 2024-08-20T22:18:30.2976951Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:18:30.2979052Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:18:30.2980436Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:18:30.2981018Z + install_tlparse 2024-08-20T22:18:30.2981419Z + pip_install --user tlparse==0.3.25 2024-08-20T22:18:30.2982222Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:18:30.6830552Z Collecting tlparse==0.3.25 2024-08-20T22:18:30.6978234Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:18:30.7070890Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:18:31.2763293Z Installing collected packages: tlparse 2024-08-20T22:18:31.3103266Z Successfully installed tlparse-0.3.25 2024-08-20T22:18:31.3939669Z ++ python -m site --user-base 2024-08-20T22:18:31.4185097Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:18:31.4188557Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:18:31.4189254Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:18:31.4189939Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:18:31.4190886Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:18:31.4192024Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:18:31.4192725Z + cd test 2024-08-20T22:18:31.4193263Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:18:33.0631493Z + [[ inductor_distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:18:33.0632127Z + [[ inductor_distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:18:33.0635397Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:18:33.0635983Z + [[ inductor_distributed == *pr_time_benchmarks* ]] 2024-08-20T22:18:33.0636507Z + [[ inductor_distributed == *dynamo_eager* ]] 2024-08-20T22:18:33.0637002Z + [[ inductor_distributed == *aot_eager* ]] 2024-08-20T22:18:33.0637482Z + [[ inductor_distributed == *aot_inductor* ]] 2024-08-20T22:18:33.0637983Z + [[ inductor_distributed == *inductor* ]] 2024-08-20T22:18:33.0638432Z + [[ inductor_distributed != *perf* ]] 2024-08-20T22:18:33.0639241Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:18:33.0639691Z + [[ inductor_distributed == *dynamic* ]] 2024-08-20T22:18:33.0640248Z + [[ inductor_distributed == *cpu* ]] 2024-08-20T22:18:33.0640855Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:18:33.0668086Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:18:33.0668771Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:18:33.0671975Z + cd test 2024-08-20T22:18:33.0672459Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:18:34.8908159Z PyTorch built with: 2024-08-20T22:18:34.8908735Z - GCC 9.4 2024-08-20T22:18:34.8909181Z - C++ Version: 201703 2024-08-20T22:18:34.8910181Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:18:34.8911270Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:18:34.8911941Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:18:34.8912463Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:18:34.8912960Z - NNPACK is enabled 2024-08-20T22:18:34.8913353Z - CPU capability usage: AVX2 2024-08-20T22:18:34.8913768Z - CUDA Runtime 12.1 2024-08-20T22:18:34.8914297Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:18:34.8914908Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:18:34.8915361Z - Magma 2.6.1 2024-08-20T22:18:34.8922637Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:18:34.8929012Z 2024-08-20T22:18:35.3130664Z + cd test 2024-08-20T22:18:35.3131477Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:18:36.6666103Z ATen/Parallel: 2024-08-20T22:18:36.6666536Z at::get_num_threads() : 24 2024-08-20T22:18:36.6667057Z at::get_num_interop_threads() : 48 2024-08-20T22:18:36.6667595Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:18:36.6668131Z omp_get_max_threads() : 24 2024-08-20T22:18:36.6669065Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:18:36.6669879Z mkl_get_max_threads() : 24 2024-08-20T22:18:36.6670514Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:18:36.6671146Z std::thread::hardware_concurrency() : 48 2024-08-20T22:18:36.6671565Z Environment variables: 2024-08-20T22:18:36.6671913Z OMP_NUM_THREADS : [not set] 2024-08-20T22:18:36.6672286Z MKL_NUM_THREADS : [not set] 2024-08-20T22:18:36.6672657Z ATen parallel backend: OpenMP 2024-08-20T22:18:36.6672912Z 2024-08-20T22:18:36.9475357Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:18:36.9476116Z + [[ inductor_distributed == *backward* ]] 2024-08-20T22:18:36.9476702Z + [[ inductor_distributed == *xla* ]] 2024-08-20T22:18:36.9477195Z + [[ inductor_distributed == *executorch* ]] 2024-08-20T22:18:36.9477706Z + [[ inductor_distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:18:36.9478359Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:18:36.9478907Z + [[ inductor_distributed == distributed ]] 2024-08-20T22:18:36.9479427Z + [[ inductor_distributed == *inductor_distributed* ]] 2024-08-20T22:18:36.9479914Z + test_inductor_distributed 2024-08-20T22:18:36.9480416Z + echo 'Testing multi_gpu tests in test_torchinductor' 2024-08-20T22:18:36.9480952Z Testing multi_gpu tests in test_torchinductor 2024-08-20T22:18:36.9481717Z + python test/run_test.py -i inductor/test_torchinductor.py -k test_multi_gpu --verbose 2024-08-20T22:18:37.0459913Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:37.0461093Z import pkg_resources 2024-08-20T22:18:39.8577077Z No TD results found 2024-08-20T22:18:39.8580055Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:39.9159019Z Ignoring disabled issues: [''] 2024-08-20T22:18:39.9301918Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:18:39.9710770Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:18:39.9724656Z Running all tests 2024-08-20T22:18:39.9727687Z Running parallel tests on 3 processes 2024-08-20T22:18:39.9728280Z Name: tests to run (est. time: 0.07min) 2024-08-20T22:18:39.9729151Z Serial tests (1): 2024-08-20T22:18:39.9729613Z inductor/test_torchinductor 1/1 2024-08-20T22:18:39.9730098Z Parallel tests (0): 2024-08-20T22:18:39.9730480Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:39.9730866Z Serial tests (0): 2024-08-20T22:18:39.9731185Z Parallel tests (0): 2024-08-20T22:18:39.9794368Z Running inductor/test_torchinductor 1/1 ... [2024-08-20 22:18:39.979069] 2024-08-20T22:18:39.9795386Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:39.9798677Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_multi_gpu', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:18:39.979412] 2024-08-20T22:18:52.7149817Z 2024-08-20T22:18:52.7151374Z inductor/test_torchinductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_1.1_994d674af7b2e74a_.log 2024-08-20T22:18:52.7153910Z Running 4 items in this shard: test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_device_cpu, test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_recompile_on_index_cpu, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_device_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_recompile_on_index_cuda 2024-08-20T22:18:52.7155635Z 2024-08-20T22:18:53.2533971Z Running test batch 'tests to run' cost 13.28 seconds 2024-08-20T22:18:53.8379339Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_non_default_cuda_device --verbose 2024-08-20T22:18:53.9355693Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:18:56.6535130Z import pkg_resources 2024-08-20T22:18:56.6535493Z No TD results found 2024-08-20T22:18:56.6536649Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:18:56.6650322Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:18:56.7089300Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:18:56.7102859Z Running all tests 2024-08-20T22:18:56.7105885Z Running parallel tests on 3 processes 2024-08-20T22:18:56.7106340Z Name: tests to run (est. time: 0.83min) 2024-08-20T22:18:56.7106753Z Serial tests (1): 2024-08-20T22:18:56.7107090Z inductor/test_aot_inductor 1/1 2024-08-20T22:18:56.7107503Z Parallel tests (0): 2024-08-20T22:18:56.7107862Z Name: excluded (est. time: 0.0min) 2024-08-20T22:18:56.7108261Z Serial tests (0): 2024-08-20T22:18:56.7108592Z Parallel tests (0): 2024-08-20T22:18:56.7169422Z Running inductor/test_aot_inductor 1/1 ... [2024-08-20 22:18:56.716588] 2024-08-20T22:18:56.7170018Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:18:56.7173705Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_cuda_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:18:56.716954] 2024-08-20T22:19:30.0866184Z 2024-08-20T22:19:30.0868766Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_0ea5be7cc849bf03_.log 2024-08-20T22:19:30.0874811Z Running 6 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_non_default_cuda_device_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_non_default_cuda_device_abi_compatible_cpu_with_stack_allocation, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_non_default_cuda_device_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCuda::test_non_default_cuda_device_abi_compatible_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCpu::test_non_default_cuda_device_non_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCuda::test_non_default_cuda_device_non_abi_compatible_cuda 2024-08-20T22:19:30.0879273Z 2024-08-20T22:19:30.6103139Z Running test batch 'tests to run' cost 33.9 seconds 2024-08-20T22:19:31.1918343Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_replicate_on_devices --verbose 2024-08-20T22:19:31.2893571Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:19:31.2894675Z import pkg_resources 2024-08-20T22:19:34.0098381Z No TD results found 2024-08-20T22:19:34.0100519Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:19:34.0211846Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:19:34.0650081Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:19:34.0659080Z Running all tests 2024-08-20T22:19:34.0662102Z Running parallel tests on 3 processes 2024-08-20T22:19:34.0662563Z Name: tests to run (est. time: 0.83min) 2024-08-20T22:19:34.0663004Z Serial tests (1): 2024-08-20T22:19:34.0665478Z inductor/test_aot_inductor 1/1 2024-08-20T22:19:34.0665956Z Parallel tests (0): 2024-08-20T22:19:34.0666327Z Name: excluded (est. time: 0.0min) 2024-08-20T22:19:34.0666718Z Serial tests (0): 2024-08-20T22:19:34.0667044Z Parallel tests (0): 2024-08-20T22:19:34.0727912Z Running inductor/test_aot_inductor 1/1 ... [2024-08-20 22:19:34.072431] 2024-08-20T22:19:34.0728670Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:19:34.0731687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:19:34.072764] 2024-08-20T22:20:05.9392717Z 2024-08-20T22:20:05.9394129Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_c191675c44697cba_.log 2024-08-20T22:20:05.9404098Z Running 6 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_replicate_on_devices_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocation::test_replicate_on_devices_abi_compatible_cpu_with_stack_allocation, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_replicate_on_devices_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCuda::test_replicate_on_devices_abi_compatible_cuda, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCpu::test_replicate_on_devices_non_abi_compatible_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCuda::test_replicate_on_devices_non_abi_compatible_cuda 2024-08-20T22:20:05.9410084Z 2024-08-20T22:20:06.4497019Z Running test batch 'tests to run' cost 32.38 seconds 2024-08-20T22:20:07.0164010Z + python test/run_test.py -i distributed/test_c10d_functional_native.py --verbose 2024-08-20T22:20:07.1140076Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:20:07.1141684Z import pkg_resources 2024-08-20T22:20:09.8550914Z No TD results found 2024-08-20T22:20:09.8552182Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:20:09.8670200Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:20:09.9117624Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:20:09.9131461Z Running all tests 2024-08-20T22:20:09.9134603Z Running parallel tests on 3 processes 2024-08-20T22:20:09.9135062Z Name: tests to run (est. time: 3.0min) 2024-08-20T22:20:09.9135481Z Serial tests (1): 2024-08-20T22:20:09.9138129Z distributed/test_c10d_functional_native 1/1 2024-08-20T22:20:09.9138659Z Parallel tests (0): 2024-08-20T22:20:09.9140398Z Name: excluded (est. time: 0.0min) 2024-08-20T22:20:09.9140811Z Serial tests (0): 2024-08-20T22:20:09.9141217Z Parallel tests (0): 2024-08-20T22:20:09.9196672Z Running distributed/test_c10d_functional_native 1/1 ... [2024-08-20 22:20:09.919300] 2024-08-20T22:20:09.9197338Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:20:09.9200361Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:20:09.919640] 2024-08-20T22:23:19.4886729Z 2024-08-20T22:23:19.4888266Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_f14a5cc44fdeccb8_.log 2024-08-20T22:23:19.4902720Z Running 25 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag 2024-08-20T22:23:19.4913839Z 2024-08-20T22:23:20.0250890Z Running test batch 'tests to run' cost 190.11 seconds 2024-08-20T22:23:20.6324939Z + python test/run_test.py -i distributed/_tensor/test_dtensor_compile.py --verbose 2024-08-20T22:23:20.7297992Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:23:20.7300192Z import pkg_resources 2024-08-20T22:23:23.4595145Z No TD results found 2024-08-20T22:23:23.4596326Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:23:23.4709403Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:23:23.5146837Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:23:23.5154701Z Running all tests 2024-08-20T22:23:23.5157653Z Running parallel tests on 3 processes 2024-08-20T22:23:23.5158113Z Name: tests to run (est. time: 0.74min) 2024-08-20T22:23:23.5158530Z Serial tests (1): 2024-08-20T22:23:23.5158909Z distributed/_tensor/test_dtensor_compile 1/1 2024-08-20T22:23:23.5164921Z Parallel tests (0): 2024-08-20T22:23:23.5165359Z Name: excluded (est. time: 0.0min) 2024-08-20T22:23:23.5165770Z Serial tests (0): 2024-08-20T22:23:23.5166099Z Parallel tests (0): 2024-08-20T22:23:23.5222203Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-08-20 22:23:23.521867] 2024-08-20T22:23:23.5222925Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:23:23.5226449Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:23:23.522215] 2024-08-20T22:24:13.5345737Z 2024-08-20T22:24:13.5347391Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_2296e4d57be8712c_.log 2024-08-20T22:24:13.5362135Z Running 26 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-08-20T22:24:13.5375408Z 2024-08-20T22:24:14.0648901Z Running test batch 'tests to run' cost 50.55 seconds 2024-08-20T22:24:14.6864089Z + python test/run_test.py -i distributed/tensor/parallel/test_micro_pipeline_tp.py --verbose 2024-08-20T22:24:14.7843182Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:24:14.7845353Z import pkg_resources 2024-08-20T22:24:17.5242587Z No TD results found 2024-08-20T22:24:17.5243938Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:24:17.5358736Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:24:17.5789065Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:24:17.5803434Z Running all tests 2024-08-20T22:24:17.5806683Z Running parallel tests on 3 processes 2024-08-20T22:24:17.5807129Z Name: tests to run (est. time: 0.15min) 2024-08-20T22:24:17.5813478Z Serial tests (1): 2024-08-20T22:24:17.5814319Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-08-20T22:24:17.5814862Z Parallel tests (0): 2024-08-20T22:24:17.5815240Z Name: excluded (est. time: 0.0min) 2024-08-20T22:24:17.5815634Z Serial tests (0): 2024-08-20T22:24:17.5815968Z Parallel tests (0): 2024-08-20T22:24:17.5869783Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-08-20 22:24:17.586545] 2024-08-20T22:24:17.5870566Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:24:17.5872994Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:24:17.586905] 2024-08-20T22:24:31.3740247Z 2024-08-20T22:24:31.3742001Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_8c1a24a771813347_.log 2024-08-20T22:24:31.3761530Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-08-20T22:24:31.3779914Z 2024-08-20T22:24:31.9215254Z Running test batch 'tests to run' cost 14.34 seconds 2024-08-20T22:24:32.5257277Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_comm.py --verbose 2024-08-20T22:24:32.6233398Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:24:32.6234500Z import pkg_resources 2024-08-20T22:24:35.2717875Z No TD results found 2024-08-20T22:24:35.2719447Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:24:35.2833256Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:24:35.3272039Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:24:35.3285343Z Running all tests 2024-08-20T22:24:35.3288283Z Running parallel tests on 3 processes 2024-08-20T22:24:35.3288769Z Name: tests to run (est. time: 2.55min) 2024-08-20T22:24:35.3289309Z Serial tests (1): 2024-08-20T22:24:35.3289742Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-08-20T22:24:35.3290247Z Parallel tests (0): 2024-08-20T22:24:35.3290597Z Name: excluded (est. time: 0.0min) 2024-08-20T22:24:35.3290989Z Serial tests (0): 2024-08-20T22:24:35.3293170Z Parallel tests (0): 2024-08-20T22:24:35.3350421Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-08-20 22:24:35.334659] 2024-08-20T22:24:35.3351194Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:24:35.3354449Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:24:35.335002] 2024-08-20T22:27:19.4687322Z 2024-08-20T22:27:19.4688916Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_e59198ae59a61b28_.log 2024-08-20T22:27:19.4699986Z Running 14 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init 2024-08-20T22:27:19.4709014Z 2024-08-20T22:27:20.0122822Z Running test batch 'tests to run' cost 164.68 seconds 2024-08-20T22:27:20.6366868Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_multi_group --verbose 2024-08-20T22:27:20.7345850Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:27:20.7347896Z import pkg_resources 2024-08-20T22:27:23.4633224Z No TD results found 2024-08-20T22:27:23.4634377Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:27:23.4751294Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:27:23.5182225Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:27:23.5192223Z Running all tests 2024-08-20T22:27:23.5195110Z Running parallel tests on 3 processes 2024-08-20T22:27:23.5195679Z Name: tests to run (est. time: 2.95min) 2024-08-20T22:27:23.5196097Z Serial tests (1): 2024-08-20T22:27:23.5199680Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T22:27:23.5200361Z Parallel tests (0): 2024-08-20T22:27:23.5200767Z Name: excluded (est. time: 0.0min) 2024-08-20T22:27:23.5201165Z Serial tests (0): 2024-08-20T22:27:23.5201491Z Parallel tests (0): 2024-08-20T22:27:23.5257397Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:27:23.525379] 2024-08-20T22:27:23.5258223Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:27:23.5261235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_multi_group', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:27:23.525719] 2024-08-20T22:28:23.4425862Z 2024-08-20T22:28:23.4430447Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_f7e6366eb3cf7b0a_.log 2024-08-20T22:28:23.4433280Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager 2024-08-20T22:28:23.4434923Z 2024-08-20T22:28:23.9519298Z Running test batch 'tests to run' cost 60.43 seconds 2024-08-20T22:28:24.5590315Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_with_activation_checkpointing --verbose 2024-08-20T22:28:24.6569451Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:28:24.6571638Z import pkg_resources 2024-08-20T22:28:27.4052469Z No TD results found 2024-08-20T22:28:27.4053798Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:28:27.4168155Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:28:27.4600896Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:28:27.4614538Z Running all tests 2024-08-20T22:28:27.4617670Z Running parallel tests on 3 processes 2024-08-20T22:28:27.4618380Z Name: tests to run (est. time: 2.95min) 2024-08-20T22:28:27.4618891Z Serial tests (1): 2024-08-20T22:28:27.4619453Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T22:28:27.4620084Z Parallel tests (0): 2024-08-20T22:28:27.4623894Z Name: excluded (est. time: 0.0min) 2024-08-20T22:28:27.4624375Z Serial tests (0): 2024-08-20T22:28:27.4624714Z Parallel tests (0): 2024-08-20T22:28:27.4681075Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:28:27.467755] 2024-08-20T22:28:27.4681810Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:28:27.4685077Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_with_activation_checkpointing', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:28:27.468090] 2024-08-20T22:29:40.3015262Z 2024-08-20T22:29:40.3017099Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_8d02d37bb65f1e11_.log 2024-08-20T22:29:40.3019711Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing 2024-08-20T22:29:40.3021103Z 2024-08-20T22:29:40.8328402Z Running test batch 'tests to run' cost 73.37 seconds 2024-08-20T22:29:41.4933476Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_hsdp --verbose 2024-08-20T22:29:41.5913031Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:29:41.5914248Z import pkg_resources 2024-08-20T22:29:44.3258471Z No TD results found 2024-08-20T22:29:44.3259846Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:29:44.3378543Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:29:44.3816736Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:29:44.3830348Z Running all tests 2024-08-20T22:29:44.3833790Z Running parallel tests on 3 processes 2024-08-20T22:29:44.3834293Z Name: tests to run (est. time: 2.95min) 2024-08-20T22:29:44.3834799Z Serial tests (1): 2024-08-20T22:29:44.3835247Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T22:29:44.3835857Z Parallel tests (0): 2024-08-20T22:29:44.3836242Z Name: excluded (est. time: 0.0min) 2024-08-20T22:29:44.3836696Z Serial tests (0): 2024-08-20T22:29:44.3837013Z Parallel tests (0): 2024-08-20T22:29:44.3899496Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:29:44.389504] 2024-08-20T22:29:44.3900499Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:29:44.3903265Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_hsdp', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:29:44.389830] 2024-08-20T22:30:02.7348410Z 2024-08-20T22:30:02.7350179Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_1a85b88dd11178a5_.log 2024-08-20T22:30:02.7352515Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp 2024-08-20T22:30:02.7353318Z 2024-08-20T22:30:03.2463721Z Running test batch 'tests to run' cost 18.86 seconds 2024-08-20T22:30:03.8669220Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_2d_transformer_checkpoint_resume --verbose 2024-08-20T22:30:03.9639371Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:30:03.9641362Z import pkg_resources 2024-08-20T22:30:06.7182668Z No TD results found 2024-08-20T22:30:06.7183998Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:30:06.7296739Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:30:06.7730964Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:30:06.7743970Z Running all tests 2024-08-20T22:30:06.7748335Z Running parallel tests on 3 processes 2024-08-20T22:30:06.7748819Z Name: tests to run (est. time: 2.95min) 2024-08-20T22:30:06.7749374Z Serial tests (1): 2024-08-20T22:30:06.7749839Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T22:30:06.7750359Z Parallel tests (0): 2024-08-20T22:30:06.7750711Z Name: excluded (est. time: 0.0min) 2024-08-20T22:30:06.7751102Z Serial tests (0): 2024-08-20T22:30:06.7755377Z Parallel tests (0): 2024-08-20T22:30:06.7810860Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:30:06.780681] 2024-08-20T22:30:06.7811727Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:30:06.7814481Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_2d_transformer_checkpoint_resume', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:30:06.781026] 2024-08-20T22:30:10.1012685Z 2024-08-20T22:30:10.1014419Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_485e90fe0bf6ce7e_.log 2024-08-20T22:30:10.1015771Z Running 0 items in this shard: 2024-08-20T22:30:10.1016034Z 2024-08-20T22:30:10.6609642Z Running test batch 'tests to run' cost 3.89 seconds 2024-08-20T22:30:11.2783269Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_gradient_accumulation --verbose 2024-08-20T22:30:11.3753401Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:30:11.3754498Z import pkg_resources 2024-08-20T22:30:14.0818041Z No TD results found 2024-08-20T22:30:14.0819968Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:30:14.0934062Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:30:14.1371741Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:30:14.1382380Z Running all tests 2024-08-20T22:30:14.1386367Z Running parallel tests on 3 processes 2024-08-20T22:30:14.1386819Z Name: tests to run (est. time: 2.95min) 2024-08-20T22:30:14.1387283Z Serial tests (1): 2024-08-20T22:30:14.1387875Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-08-20T22:30:14.1388416Z Parallel tests (0): 2024-08-20T22:30:14.1388766Z Name: excluded (est. time: 0.0min) 2024-08-20T22:30:14.1389168Z Serial tests (0): 2024-08-20T22:30:14.1389495Z Parallel tests (0): 2024-08-20T22:30:14.1449536Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-08-20 22:30:14.144578] 2024-08-20T22:30:14.1450436Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:30:14.1453514Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_gradient_accumulation', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:30:14.144915] 2024-08-20T22:30:56.1367856Z 2024-08-20T22:30:56.1369551Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_e75150a9ddf4dcd7_.log 2024-08-20T22:30:56.1374748Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation 2024-08-20T22:30:56.1375790Z 2024-08-20T22:30:56.6592914Z Running test batch 'tests to run' cost 42.52 seconds 2024-08-20T22:30:57.2701101Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_state_dict.py -k test_dp_state_dict_save_load --verbose 2024-08-20T22:30:57.3683338Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:30:57.3685570Z import pkg_resources 2024-08-20T22:31:00.1175445Z No TD results found 2024-08-20T22:31:00.1177750Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:31:00.1292308Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:00.1726579Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:00.1739804Z Running all tests 2024-08-20T22:31:00.1742983Z Running parallel tests on 3 processes 2024-08-20T22:31:00.1743456Z Name: tests to run (est. time: 0.12min) 2024-08-20T22:31:00.1743876Z Serial tests (1): 2024-08-20T22:31:00.1744329Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-08-20T22:31:00.1748685Z Parallel tests (0): 2024-08-20T22:31:00.1749128Z Name: excluded (est. time: 0.0min) 2024-08-20T22:31:00.1749529Z Serial tests (0): 2024-08-20T22:31:00.1749862Z Parallel tests (0): 2024-08-20T22:31:00.1805372Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-08-20 22:31:00.180159] 2024-08-20T22:31:00.1806132Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:31:00.1809709Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_dp_state_dict_save_load', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:31:00.180495] 2024-08-20T22:31:10.7634435Z 2024-08-20T22:31:10.7636875Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_82144112400bc22b_.log 2024-08-20T22:31:10.7638887Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load 2024-08-20T22:31:10.7639788Z 2024-08-20T22:31:11.2741408Z Running test batch 'tests to run' cost 11.1 seconds 2024-08-20T22:31:11.9227236Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_frozen.py --verbose 2024-08-20T22:31:12.0201385Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:31:12.0203576Z import pkg_resources 2024-08-20T22:31:14.7608538Z No TD results found 2024-08-20T22:31:14.7610064Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:31:14.7724049Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:14.8157286Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:14.8170118Z Running all tests 2024-08-20T22:31:14.8173204Z Running parallel tests on 3 processes 2024-08-20T22:31:14.8173831Z Name: tests to run (est. time: 0.46min) 2024-08-20T22:31:14.8174283Z Serial tests (1): 2024-08-20T22:31:14.8174763Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-08-20T22:31:14.8175289Z Parallel tests (0): 2024-08-20T22:31:14.8175714Z Name: excluded (est. time: 0.0min) 2024-08-20T22:31:14.8176250Z Serial tests (0): 2024-08-20T22:31:14.8176637Z Parallel tests (0): 2024-08-20T22:31:14.8235911Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-08-20 22:31:14.823190] 2024-08-20T22:31:14.8236654Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:31:14.8239649Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:31:14.823534] 2024-08-20T22:31:46.8924061Z 2024-08-20T22:31:46.8925890Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_b906e75b54c4280b_.log 2024-08-20T22:31:46.8928960Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-08-20T22:31:46.8930990Z 2024-08-20T22:31:47.4063878Z Running test batch 'tests to run' cost 32.59 seconds 2024-08-20T22:31:48.0399285Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_compute_dtype --verbose 2024-08-20T22:31:48.1370716Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:31:48.1372297Z import pkg_resources 2024-08-20T22:31:50.8928356Z No TD results found 2024-08-20T22:31:50.8929724Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:31:50.9042211Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:50.9480487Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:31:50.9488064Z Running all tests 2024-08-20T22:31:50.9491007Z Running parallel tests on 3 processes 2024-08-20T22:31:50.9491614Z Name: tests to run (est. time: 0.36min) 2024-08-20T22:31:50.9492115Z Serial tests (1): 2024-08-20T22:31:50.9496633Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-08-20T22:31:50.9497268Z Parallel tests (0): 2024-08-20T22:31:50.9497639Z Name: excluded (est. time: 0.0min) 2024-08-20T22:31:50.9498041Z Serial tests (0): 2024-08-20T22:31:50.9498406Z Parallel tests (0): 2024-08-20T22:31:50.9553763Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-08-20 22:31:50.954963] 2024-08-20T22:31:50.9554706Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:31:50.9557237Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_compute_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:31:50.955300] 2024-08-20T22:32:05.1435167Z 2024-08-20T22:32:05.1437049Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_5c37e1a68ab38f59_.log 2024-08-20T22:32:05.1439082Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype 2024-08-20T22:32:05.1439964Z 2024-08-20T22:32:05.6814778Z Running test batch 'tests to run' cost 14.73 seconds 2024-08-20T22:32:06.2997293Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_reduce_dtype --verbose 2024-08-20T22:32:06.3975356Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:32:06.3976867Z import pkg_resources 2024-08-20T22:32:09.1359777Z No TD results found 2024-08-20T22:32:09.1361157Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:32:09.1473593Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:09.1907827Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:09.1918988Z Running all tests 2024-08-20T22:32:09.1922427Z Running parallel tests on 3 processes 2024-08-20T22:32:09.1922988Z Name: tests to run (est. time: 0.36min) 2024-08-20T22:32:09.1923416Z Serial tests (1): 2024-08-20T22:32:09.1923961Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-08-20T22:32:09.1924866Z Parallel tests (0): 2024-08-20T22:32:09.1925270Z Name: excluded (est. time: 0.0min) 2024-08-20T22:32:09.1926259Z Serial tests (0): 2024-08-20T22:32:09.1926626Z Parallel tests (0): 2024-08-20T22:32:09.1983120Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-08-20 22:32:09.197951] 2024-08-20T22:32:09.1983897Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:32:09.1987602Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_reduce_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:32:09.198304] 2024-08-20T22:32:23.3889249Z 2024-08-20T22:32:23.3891368Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_404500108da9da4d_.log 2024-08-20T22:32:23.3893395Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype 2024-08-20T22:32:23.3894268Z 2024-08-20T22:32:23.9122093Z Running test batch 'tests to run' cost 14.72 seconds 2024-08-20T22:32:24.5734949Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py -k test_clip_grad_norm_2d --verbose 2024-08-20T22:32:24.6714923Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:32:24.6717129Z import pkg_resources 2024-08-20T22:32:27.4238073Z No TD results found 2024-08-20T22:32:27.4239430Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:32:27.4352388Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:27.4788971Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:27.4801868Z Running all tests 2024-08-20T22:32:27.4805092Z Running parallel tests on 3 processes 2024-08-20T22:32:27.4805551Z Name: tests to run (est. time: 0.19min) 2024-08-20T22:32:27.4805967Z Serial tests (1): 2024-08-20T22:32:27.4806441Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-08-20T22:32:27.4806987Z Parallel tests (0): 2024-08-20T22:32:27.4807353Z Name: excluded (est. time: 0.0min) 2024-08-20T22:32:27.4807756Z Serial tests (0): 2024-08-20T22:32:27.4808090Z Parallel tests (0): 2024-08-20T22:32:27.4867944Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-08-20 22:32:27.486396] 2024-08-20T22:32:27.4868731Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:32:27.4871393Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_clip_grad_norm_2d', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:32:27.486729] 2024-08-20T22:32:42.8782099Z 2024-08-20T22:32:42.8784054Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f1bcb4ed6018c01f_.log 2024-08-20T22:32:42.8786064Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-08-20T22:32:42.8786907Z 2024-08-20T22:32:43.3967746Z Running test batch 'tests to run' cost 15.92 seconds 2024-08-20T22:32:44.0376444Z + python test/run_test.py -i distributed/fsdp/test_fsdp_tp_integration.py -k test_fsdp_tp_integration --verbose 2024-08-20T22:32:44.1351310Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:32:44.1352775Z import pkg_resources 2024-08-20T22:32:46.8830915Z No TD results found 2024-08-20T22:32:46.8832268Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:32:46.8944508Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:46.9387769Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:32:46.9400703Z Running all tests 2024-08-20T22:32:46.9403927Z Running parallel tests on 3 processes 2024-08-20T22:32:46.9404465Z Name: tests to run (est. time: 0.45min) 2024-08-20T22:32:46.9405035Z Serial tests (1): 2024-08-20T22:32:46.9405533Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-08-20T22:32:46.9406044Z Parallel tests (0): 2024-08-20T22:32:46.9406400Z Name: excluded (est. time: 0.0min) 2024-08-20T22:32:46.9406802Z Serial tests (0): 2024-08-20T22:32:46.9407130Z Parallel tests (0): 2024-08-20T22:32:46.9467408Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-08-20 22:32:46.946310] 2024-08-20T22:32:46.9468170Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:32:46.9470744Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_fsdp_tp_integration', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:32:46.946649] 2024-08-20T22:33:18.7669712Z 2024-08-20T22:33:18.7671178Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_4507c0072375f660_.log 2024-08-20T22:33:18.7673872Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-08-20T22:33:18.7675989Z 2024-08-20T22:33:19.2812441Z Running test batch 'tests to run' cost 32.34 seconds 2024-08-20T22:33:19.9146159Z + python test/run_test.py --include distributed/test_dynamo_distributed distributed/test_inductor_collectives --verbose 2024-08-20T22:33:20.0117108Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-20T22:33:20.0119731Z import pkg_resources 2024-08-20T22:33:22.7650499Z No TD results found 2024-08-20T22:33:22.7652948Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-08-20T22:33:22.7763436Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:33:22.8200562Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.1-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-08-20T22:33:22.8212598Z Running all tests 2024-08-20T22:33:22.8216098Z Running parallel tests on 3 processes 2024-08-20T22:33:22.8216567Z Name: tests to run (est. time: 6.52min) 2024-08-20T22:33:22.8216997Z Serial tests (2): 2024-08-20T22:33:22.8217368Z distributed/test_dynamo_distributed 1/1 2024-08-20T22:33:22.8219952Z distributed/test_inductor_collectives 1/1 2024-08-20T22:33:22.8220472Z Parallel tests (0): 2024-08-20T22:33:22.8220839Z Name: excluded (est. time: 0.0min) 2024-08-20T22:33:22.8221727Z Serial tests (0): 2024-08-20T22:33:22.8222053Z Parallel tests (0): 2024-08-20T22:33:22.8279061Z Running distributed/test_dynamo_distributed 1/1 ... [2024-08-20 22:33:22.827458] 2024-08-20T22:33:22.8279813Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:33:22.8282111Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:33:22.827794] 2024-08-20T22:38:34.2982219Z 2024-08-20T22:38:34.2984190Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_4467ffa44454ddd2_.log 2024-08-20T22:38:34.3007832Z Running 48 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-08-20T22:38:34.3030024Z 2024-08-20T22:38:34.3030626Z Running distributed/test_inductor_collectives 1/1 ... [2024-08-20 22:38:34.298478] 2024-08-20T22:38:34.3031283Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-20T22:38:34.3032945Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-20 22:38:34.298817] 2024-08-20T22:40:19.9913687Z 2024-08-20T22:40:19.9915215Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_956b64bdfb06c5ad_.log 2024-08-20T22:40:19.9947309Z Running 47 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-08-20T22:40:19.9973138Z 2024-08-20T22:40:20.5313098Z Running test batch 'tests to run' cost 417.71 seconds 2024-08-20T22:40:21.1563868Z + assert_git_not_dirty 2024-08-20T22:40:21.1564678Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:40:21.1565331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *xla* ]] 2024-08-20T22:40:21.1571132Z ++ git status --porcelain 2024-08-20T22:40:21.1572937Z ++ grep -v '?? third_party' 2024-08-20T22:40:23.5808609Z ++ true 2024-08-20T22:40:23.5811584Z + git_status= 2024-08-20T22:40:23.5812117Z + [[ -n '' ]] 2024-08-20T22:40:23.5812920Z + cleanup_workspace 2024-08-20T22:40:23.5813736Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T22:40:23.5814814Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T22:40:23.5815729Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T22:40:23.5816359Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T22:40:23.5817184Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T22:40:23.5818529Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T22:40:23.5819319Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T22:40:24.3317298Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T22:40:24.3317818Z cat test/**/*_toprint.log || true 2024-08-20T22:40:24.3331876Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.3332389Z env: 2024-08-20T22:40:24.3332666Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.3333129Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.3333874Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.3334524Z ##[endgroup] 2024-08-20T22:40:24.3429105Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T22:40:24.3464948Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:40:24.3465427Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T22:40:24.3474443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.3474928Z env: 2024-08-20T22:40:24.3475211Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.3475660Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.3476399Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.3477046Z MONITOR_SCRIPT_PID: 480059 2024-08-20T22:40:24.3477423Z ##[endgroup] 2024-08-20T22:40:24.3677935Z Prepare all required actions 2024-08-20T22:40:24.3678391Z Getting action download info 2024-08-20T22:40:24.5542328Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T22:40:24.7090598Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T22:40:24.7091101Z with: 2024-08-20T22:40:24.7091748Z file-suffix: test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401 2024-08-20T22:40:24.7092543Z s3-bucket: gha-artifacts 2024-08-20T22:40:24.7092895Z env: 2024-08-20T22:40:24.7093168Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.7093621Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.7094365Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.7095014Z ##[endgroup] 2024-08-20T22:40:24.7122755Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T22:40:24.7123418Z # Remove any previous test jsons if they exist 2024-08-20T22:40:24.7123937Z rm -f test-jsons-*.zip 2024-08-20T22:40:24.7124461Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T22:40:24.7133664Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.7134165Z env: 2024-08-20T22:40:24.7134437Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.7134895Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.7135639Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.7136596Z FILE_SUFFIX: test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401 2024-08-20T22:40:24.7137278Z ##[endgroup] 2024-08-20T22:40:24.7423770Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T22:40:24.7453426Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T22:40:24.7454183Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T22:40:24.7460517Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T22:40:24.7467009Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T22:40:24.7467836Z adding: test/test-reports/td_exclusions-75ec53a9c5c48eef137e.json (deflated 25%) 2024-08-20T22:40:24.7468774Z adding: test/test-reports/td_exclusions-8e874f9d138e54547229.json (deflated 24%) 2024-08-20T22:40:24.7469704Z adding: test/test-reports/td_exclusions-7f519d28c42ec9e4324c.json (deflated 24%) 2024-08-20T22:40:24.7470650Z adding: test/test-reports/td_exclusions-927f13f51b7a7bafad1b.json (deflated 13%) 2024-08-20T22:40:24.7471589Z adding: test/test-reports/td_exclusions-7c8ba27aef9aa3c15a4a.json (deflated 19%) 2024-08-20T22:40:24.7472510Z adding: test/test-reports/td_exclusions-efffda38267ec4e18d9a.json (deflated 14%) 2024-08-20T22:40:24.7473443Z adding: test/test-reports/td_exclusions-fff170e7c062c7314b29.json (deflated 14%) 2024-08-20T22:40:24.7474385Z adding: test/test-reports/td_exclusions-a425d53c88b588d0b806.json (deflated 15%) 2024-08-20T22:40:24.7475327Z adding: test/test-reports/td_exclusions-f3123b56ef0e2bff7c77.json (deflated 15%) 2024-08-20T22:40:24.7476247Z adding: test/test-reports/td_exclusions-a1f837b8c9ea899cf483.json (deflated 15%) 2024-08-20T22:40:24.7477178Z adding: test/test-reports/td_exclusions-75309ae19ca13ef9a881.json (deflated 15%) 2024-08-20T22:40:24.7478090Z adding: test/test-reports/td_exclusions-7d54bb6670d631444c2d.json (deflated 15%) 2024-08-20T22:40:24.7479021Z adding: test/test-reports/td_exclusions-7467e7d168ab04f0a0ec.json (deflated 16%) 2024-08-20T22:40:24.7479940Z adding: test/test-reports/td_exclusions-962ad4bf63496d24bfb1.json (deflated 13%) 2024-08-20T22:40:24.7480865Z adding: test/test-reports/td_exclusions-d546fc1c0336fa193ec9.json (deflated 16%) 2024-08-20T22:40:24.7481784Z adding: test/test-reports/td_exclusions-309f8d4e5114ee77758e.json (deflated 16%) 2024-08-20T22:40:24.7482940Z adding: test/test-reports/td_exclusions-510dbf269d9ee1f1847d.json (deflated 15%) 2024-08-20T22:40:24.7483879Z adding: test/test-reports/td_exclusions-136926cf631fe78fd0c7.json (deflated 14%) 2024-08-20T22:40:24.7484786Z adding: test/test-reports/td_exclusions-b212e56fdc682ad71ce5.json (deflated 36%) 2024-08-20T22:40:24.7485587Z adding: test/.pytorch-disabled-tests.json (deflated 88%) 2024-08-20T22:40:24.7517565Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T22:40:24.7518577Z # Remove any previous test reports if they exist 2024-08-20T22:40:24.7519160Z rm -f test-reports-*.zip 2024-08-20T22:40:24.7519756Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T22:40:24.7528942Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.7529455Z env: 2024-08-20T22:40:24.7529730Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.7530232Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.7530989Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.7531954Z FILE_SUFFIX: test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401 2024-08-20T22:40:24.7532645Z ##[endgroup] 2024-08-20T22:40:24.7811322Z adding: test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-f41d60ff6c037ddf.xml (deflated 74%) 2024-08-20T22:40:24.7812847Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-de3f5d5a89a9d881.xml (deflated 73%) 2024-08-20T22:40:24.7814317Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-24bea173db269f4f.xml (deflated 76%) 2024-08-20T22:40:24.7815931Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-68adb181675da5b4.xml (deflated 81%) 2024-08-20T22:40:24.7817676Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-1966d5f1e8512546.xml (deflated 82%) 2024-08-20T22:40:24.7819941Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-c90903e8aef28df8.xml (deflated 87%) 2024-08-20T22:40:24.7821990Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-d53273e1197cb329.xml (deflated 80%) 2024-08-20T22:40:24.7824004Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-7e9e71055a3c353e.xml (deflated 55%) 2024-08-20T22:40:24.7826028Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-1d6c1430f8bc1c6d.xml (deflated 35%) 2024-08-20T22:40:24.7828062Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-5850b3fe7d49cf9c.xml (deflated 35%) 2024-08-20T22:40:24.7830093Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-9c9789d6e0788bb5.xml (deflated 28%) 2024-08-20T22:40:24.7832117Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-41d3b862605af0ad.xml (deflated 36%) 2024-08-20T22:40:24.7834484Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-4a0861ac0af73455.xml (deflated 36%) 2024-08-20T22:40:24.7836512Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-49ac0607d436aadf.xml (deflated 57%) 2024-08-20T22:40:24.7838779Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-109e8f5b0192223b.xml (deflated 37%) 2024-08-20T22:40:24.7840958Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-0fe97f9852964a50.xml (deflated 37%) 2024-08-20T22:40:24.7843099Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-92c6a71bc4bb554b.xml (deflated 37%) 2024-08-20T22:40:24.7845052Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-5d295dea28ef3537.xml (deflated 57%) 2024-08-20T22:40:24.7846778Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-ca16666e6ff888e3.xml (deflated 90%) 2024-08-20T22:40:24.7848460Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-fb0ec88f6691d690.xml (deflated 87%) 2024-08-20T22:40:24.7873136Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T22:40:24.7873758Z # Remove any previous usage logs if they exist 2024-08-20T22:40:24.7874252Z rm -f logs-*.zip 2024-08-20T22:40:24.7874920Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T22:40:24.7875712Z # so check to see if the file exists first 2024-08-20T22:40:24.7876227Z if [ -f 'usage_log.txt' ]; then 2024-08-20T22:40:24.7876754Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T22:40:24.7877250Z fi 2024-08-20T22:40:24.7877623Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T22:40:24.7878214Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T22:40:24.7878859Z fi 2024-08-20T22:40:24.7886890Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.7887399Z env: 2024-08-20T22:40:24.7887682Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.7888131Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.7888884Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.7889847Z FILE_SUFFIX: test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401 2024-08-20T22:40:24.7890555Z ##[endgroup] 2024-08-20T22:40:24.7957544Z adding: usage_log.txt (deflated 92%) 2024-08-20T22:40:24.8258739Z adding: test/test-reports/inductor.test_torchinductor_1.1_994d674af7b2e74a_.log (deflated 63%) 2024-08-20T22:40:24.8260621Z adding: test/test-reports/inductor.test_aot_inductor_1.1_0ea5be7cc849bf03_.log (deflated 74%) 2024-08-20T22:40:24.8261879Z adding: test/test-reports/inductor.test_aot_inductor_1.1_c191675c44697cba_.log (deflated 74%) 2024-08-20T22:40:24.8263135Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_f14a5cc44fdeccb8_.log (deflated 88%) 2024-08-20T22:40:24.8264427Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_2296e4d57be8712c_.log (deflated 84%) 2024-08-20T22:40:24.8265705Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_8c1a24a771813347_.log (deflated 88%) 2024-08-20T22:40:24.8286485Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_e59198ae59a61b28_.log (deflated 95%) 2024-08-20T22:40:24.8287872Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_f7e6366eb3cf7b0a_.log (deflated 60%) 2024-08-20T22:40:24.8289221Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_8d02d37bb65f1e11_.log (deflated 60%) 2024-08-20T22:40:24.8290609Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_1a85b88dd11178a5_.log (deflated 57%) 2024-08-20T22:40:24.8292208Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_485e90fe0bf6ce7e_.log (deflated 53%) 2024-08-20T22:40:24.8293558Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_e75150a9ddf4dcd7_.log (deflated 57%) 2024-08-20T22:40:24.8294889Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_82144112400bc22b_.log (deflated 58%) 2024-08-20T22:40:24.8296211Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_b906e75b54c4280b_.log (deflated 83%) 2024-08-20T22:40:24.8297562Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_5c37e1a68ab38f59_.log (deflated 58%) 2024-08-20T22:40:24.8298959Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_404500108da9da4d_.log (deflated 57%) 2024-08-20T22:40:24.8300365Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_f1bcb4ed6018c01f_.log (deflated 58%) 2024-08-20T22:40:24.8301741Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_4507c0072375f660_.log (deflated 66%) 2024-08-20T22:40:24.8302902Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_4467ffa44454ddd2_.log (deflated 90%) 2024-08-20T22:40:24.8304048Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_956b64bdfb06c5ad_.log (deflated 89%) 2024-08-20T22:40:24.8331552Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T22:40:24.8332261Z # Remove any previous debugging artifacts if they exist 2024-08-20T22:40:24.8332813Z rm -f debug-*.zip 2024-08-20T22:40:24.8333205Z if [ -d 'test/debug' ]; then 2024-08-20T22:40:24.8333721Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T22:40:24.8334192Z fi 2024-08-20T22:40:24.8342338Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:24.8343020Z env: 2024-08-20T22:40:24.8343303Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.8343745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.8344476Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.8345418Z FILE_SUFFIX: test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401 2024-08-20T22:40:24.8346104Z ##[endgroup] 2024-08-20T22:40:24.8442411Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:40:24.8442850Z with: 2024-08-20T22:40:24.8443134Z s3-bucket: gha-artifacts 2024-08-20T22:40:24.8443579Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:24.8444054Z retention-days: 14 2024-08-20T22:40:24.8444386Z if-no-files-found: warn 2024-08-20T22:40:24.8444751Z path: test-jsons-*.zip 2024-08-20T22:40:24.8445095Z name: artifact 2024-08-20T22:40:24.8445391Z region: us-east-1 2024-08-20T22:40:24.8445697Z env: 2024-08-20T22:40:24.8445976Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:24.8446461Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:24.8447193Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:24.8447835Z ##[endgroup] 2024-08-20T22:40:25.1988274Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:40:25.1989455Z With the provided path, there will be 1 file uploaded 2024-08-20T22:40:25.1990555Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:25.2045404Z Starting upload of test-jsons-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:25.3150216Z Finished upload of test-jsons-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:25.3428450Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:40:25.3428909Z with: 2024-08-20T22:40:25.3429223Z s3-bucket: gha-artifacts 2024-08-20T22:40:25.3429873Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:25.3430472Z retention-days: 14 2024-08-20T22:40:25.3430812Z if-no-files-found: error 2024-08-20T22:40:25.3431185Z path: test-reports-*.zip 2024-08-20T22:40:25.3431526Z name: artifact 2024-08-20T22:40:25.3431834Z region: us-east-1 2024-08-20T22:40:25.3432144Z env: 2024-08-20T22:40:25.3432446Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:25.3432922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:25.3433698Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:25.3434460Z ##[endgroup] 2024-08-20T22:40:25.6966869Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:40:25.6968036Z With the provided path, there will be 1 file uploaded 2024-08-20T22:40:25.6969259Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:25.7024274Z Starting upload of test-reports-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:25.8536004Z Finished upload of test-reports-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:25.8815202Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:40:25.8815651Z with: 2024-08-20T22:40:25.8815930Z s3-bucket: gha-artifacts 2024-08-20T22:40:25.8816363Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:25.8816837Z retention-days: 14 2024-08-20T22:40:25.8817174Z if-no-files-found: ignore 2024-08-20T22:40:25.8817527Z path: logs-*.zip 2024-08-20T22:40:25.8817847Z name: artifact 2024-08-20T22:40:25.8818452Z region: us-east-1 2024-08-20T22:40:25.8818767Z env: 2024-08-20T22:40:25.8819039Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:25.8819495Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:25.8820234Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:25.8820937Z ##[endgroup] 2024-08-20T22:40:26.2340523Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T22:40:26.2341992Z With the provided path, there will be 1 file uploaded 2024-08-20T22:40:26.2343107Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:26.2397172Z Starting upload of logs-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:26.3491358Z Finished upload of logs-test-inductor_distributed-1-1-amz2023.linux.g5.12xlarge.nvidia.gpu_29026527401.zip 2024-08-20T22:40:26.3771246Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T22:40:26.3771693Z with: 2024-08-20T22:40:26.3771969Z s3-bucket: gha-artifacts 2024-08-20T22:40:26.3772408Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T22:40:26.3772885Z retention-days: 14 2024-08-20T22:40:26.3773227Z if-no-files-found: ignore 2024-08-20T22:40:26.3773584Z path: debug-*.zip 2024-08-20T22:40:26.3773904Z name: artifact 2024-08-20T22:40:26.3774211Z region: us-east-1 2024-08-20T22:40:26.3774501Z env: 2024-08-20T22:40:26.3774809Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:26.3775276Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:26.3776019Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:26.3776657Z ##[endgroup] 2024-08-20T22:40:26.7238383Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T22:40:26.7517829Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T22:40:26.7518614Z # shellcheck disable=SC2156 2024-08-20T22:40:26.7519398Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T22:40:26.7528813Z shell: /usr/bin/bash -e {0} 2024-08-20T22:40:26.7529168Z env: 2024-08-20T22:40:26.7529453Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:26.7529910Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:26.7530698Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:26.7531365Z ##[endgroup] 2024-08-20T22:40:27.0112328Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T22:40:27.0112889Z with: 2024-08-20T22:40:27.0113150Z env: 2024-08-20T22:40:27.0113429Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:27.0113885Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:27.0114627Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:27.0115281Z ##[endgroup] 2024-08-20T22:40:27.0154228Z ##[group]Run set -eou pipefail 2024-08-20T22:40:27.0154628Z set -eou pipefail 2024-08-20T22:40:27.0154975Z  2024-08-20T22:40:27.0155621Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T22:40:27.0156243Z for _ in $(seq 1440); do 2024-08-20T22:40:27.0156718Z  # Break if no ssh session exists anymore 2024-08-20T22:40:27.0157216Z  if [ "$(who)" = "" ]; then 2024-08-20T22:40:27.0157644Z  break 2024-08-20T22:40:27.0157988Z  fi 2024-08-20T22:40:27.0158280Z  echo "." 2024-08-20T22:40:27.0158608Z  sleep 5 2024-08-20T22:40:27.0158921Z done 2024-08-20T22:40:27.0168180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:27.0168679Z env: 2024-08-20T22:40:27.0168952Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:27.0169405Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:27.0170120Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:27.0170757Z ##[endgroup] 2024-08-20T22:40:27.0203197Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T22:40:27.0267262Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:40:27.0268038Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:40:27.0268629Z # shellcheck disable=SC2046 2024-08-20T22:40:27.0269297Z docker stop $(docker ps -q) || true 2024-08-20T22:40:27.0269770Z # Prune all of the docker images 2024-08-20T22:40:27.0270223Z docker system prune -af 2024-08-20T22:40:27.0279149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:27.0279643Z env: 2024-08-20T22:40:27.0279915Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:40:27.0280362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:40:27.0281150Z DOCKER_CONTAINER_ID: 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:27.0281788Z ##[endgroup] 2024-08-20T22:40:27.6603883Z 2fda783812ab 2024-08-20T22:40:28.5912487Z Deleted Containers: 2024-08-20T22:40:28.5913011Z 2fda783812aba350b370a0afed83cbd6f46e9620d046d63c7395931fdf3d2580 2024-08-20T22:40:28.5913448Z 2024-08-20T22:40:39.6576239Z Deleted Images: 2024-08-20T22:40:39.6577892Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:40:39.6581046Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:40:39.6582678Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T22:40:39.6583556Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T22:40:39.6584430Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T22:40:39.6585255Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T22:40:39.6586068Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T22:40:39.6586866Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T22:40:39.6587679Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T22:40:39.6588813Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T22:40:39.6589643Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T22:40:39.6590482Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T22:40:39.6591301Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T22:40:39.6592109Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T22:40:39.6592924Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T22:40:39.6593739Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T22:40:39.6594545Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T22:40:39.6595374Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T22:40:39.6596213Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T22:40:39.6597037Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T22:40:39.6597850Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T22:40:39.6598661Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T22:40:39.6599481Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T22:40:39.6600313Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T22:40:39.6601132Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T22:40:39.6601947Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T22:40:39.6602773Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T22:40:39.6603806Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T22:40:39.6604642Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T22:40:39.6605476Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T22:40:39.6606304Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T22:40:39.6607126Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T22:40:39.6607949Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T22:40:39.6608759Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T22:40:39.6609573Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T22:40:39.6610393Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T22:40:39.6611218Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T22:40:39.6612038Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T22:40:39.6612858Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T22:40:39.6613726Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T22:40:39.6614550Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T22:40:39.6615364Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T22:40:39.6616194Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T22:40:39.6617039Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T22:40:39.6617863Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T22:40:39.6618982Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T22:40:39.6621068Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T22:40:39.6621927Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T22:40:39.6622758Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T22:40:39.6623898Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T22:40:39.6624716Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T22:40:39.6625549Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T22:40:39.6626379Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T22:40:39.6627187Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T22:40:39.6627989Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T22:40:39.6628827Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T22:40:39.6629662Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T22:40:39.6630481Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T22:40:39.6631290Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T22:40:39.6632116Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T22:40:39.6632918Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T22:40:39.6633736Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T22:40:39.6634564Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T22:40:39.6635387Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T22:40:39.6636222Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T22:40:39.6637170Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T22:40:39.6637983Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T22:40:39.6638804Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T22:40:39.6639641Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T22:40:39.6640483Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T22:40:39.6641323Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T22:40:39.6642148Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T22:40:39.6642952Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T22:40:39.6643823Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T22:40:39.6644665Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T22:40:39.6645482Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T22:40:39.6646312Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T22:40:39.6647156Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T22:40:39.6647997Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T22:40:39.6648842Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T22:40:39.6649651Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T22:40:39.6650147Z 2024-08-20T22:40:39.6650294Z Total reclaimed space: 39.94GB 2024-08-20T22:40:39.6716841Z Post job cleanup. 2024-08-20T22:40:39.6767198Z Post job cleanup. 2024-08-20T22:40:39.7662923Z [command]/usr/bin/git version 2024-08-20T22:40:39.7716250Z git version 2.40.1 2024-08-20T22:40:39.7755339Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b377b10e-943e-4e4d-874c-c67801496482' before making global git config changes 2024-08-20T22:40:39.7756565Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:40:39.7760446Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:40:39.7812103Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:40:39.7859494Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:40:39.8252486Z Entering 'android/libs/fbjni' 2024-08-20T22:40:39.8324126Z Entering 'third_party/FP16' 2024-08-20T22:40:39.8400180Z Entering 'third_party/FXdiv' 2024-08-20T22:40:39.8470533Z Entering 'third_party/NNPACK' 2024-08-20T22:40:39.8546740Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:40:39.8618403Z Entering 'third_party/XNNPACK' 2024-08-20T22:40:39.8703236Z Entering 'third_party/benchmark' 2024-08-20T22:40:39.8779092Z Entering 'third_party/cpp-httplib' 2024-08-20T22:40:39.8849548Z Entering 'third_party/cpuinfo' 2024-08-20T22:40:39.8920275Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:40:39.8995330Z Entering 'third_party/cutlass' 2024-08-20T22:40:39.9071674Z Entering 'third_party/eigen' 2024-08-20T22:40:39.9150167Z Entering 'third_party/fbgemm' 2024-08-20T22:40:39.9221533Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:40:39.9289984Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:40:39.9358216Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:40:39.9436584Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:40:39.9514309Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:40:39.9585599Z Entering 'third_party/flatbuffers' 2024-08-20T22:40:39.9659055Z Entering 'third_party/fmt' 2024-08-20T22:40:39.9732235Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:40:39.9803324Z Entering 'third_party/gloo' 2024-08-20T22:40:39.9873475Z Entering 'third_party/googletest' 2024-08-20T22:40:39.9944673Z Entering 'third_party/ideep' 2024-08-20T22:40:40.0013094Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:40:40.0095465Z Entering 'third_party/ittapi' 2024-08-20T22:40:40.0169998Z Entering 'third_party/kineto' 2024-08-20T22:40:40.0245379Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:40:40.0312072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:40:40.0388521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:40:40.0460193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:40:40.0529022Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:40:40.0598286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:40:40.0674261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:40:40.0747600Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:40:40.0821461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:40:40.0891287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:40:40.0963015Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:40:40.1031307Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:40:40.1103930Z Entering 'third_party/mimalloc' 2024-08-20T22:40:40.1175377Z Entering 'third_party/nccl/nccl' 2024-08-20T22:40:40.1249892Z Entering 'third_party/nlohmann' 2024-08-20T22:40:40.1321055Z Entering 'third_party/onnx' 2024-08-20T22:40:40.1410697Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:40:40.1480062Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:40:40.1557180Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:40:40.1627269Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:40:40.1695665Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:40:40.1769887Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:40:40.1838498Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:40:40.1908471Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:40:40.1976214Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:40:40.2046934Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:40:40.2112901Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:40:40.2189900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:40:40.2263104Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:40:40.2356643Z Entering 'third_party/pocketfft' 2024-08-20T22:40:40.2430374Z Entering 'third_party/protobuf' 2024-08-20T22:40:40.2510873Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:40:40.2578685Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:40:40.2649923Z Entering 'third_party/psimd' 2024-08-20T22:40:40.2721548Z Entering 'third_party/pthreadpool' 2024-08-20T22:40:40.2792267Z Entering 'third_party/pybind11' 2024-08-20T22:40:40.2867983Z Entering 'third_party/python-peachpy' 2024-08-20T22:40:40.2941024Z Entering 'third_party/sleef' 2024-08-20T22:40:40.3011748Z Entering 'third_party/tensorpipe' 2024-08-20T22:40:40.3088859Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:40:40.3158372Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:40:40.3231311Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:40:40.3301660Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:40:40.3371925Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:40:40.3468510Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:40:40.3506848Z http.https://github.com/.extraheader 2024-08-20T22:40:40.3517863Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T22:40:40.3567674Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:40:40.3965520Z Entering 'android/libs/fbjni' 2024-08-20T22:40:40.4010710Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4059941Z Entering 'third_party/FP16' 2024-08-20T22:40:40.4104507Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4150637Z Entering 'third_party/FXdiv' 2024-08-20T22:40:40.4194954Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4241880Z Entering 'third_party/NNPACK' 2024-08-20T22:40:40.4287958Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4334087Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:40:40.4383139Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4430285Z Entering 'third_party/XNNPACK' 2024-08-20T22:40:40.4474823Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4538527Z Entering 'third_party/benchmark' 2024-08-20T22:40:40.4586282Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4631697Z Entering 'third_party/cpp-httplib' 2024-08-20T22:40:40.4680616Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4725056Z Entering 'third_party/cpuinfo' 2024-08-20T22:40:40.4769665Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4815380Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:40:40.4859693Z http.https://github.com/.extraheader 2024-08-20T22:40:40.4904713Z Entering 'third_party/cutlass' 2024-08-20T22:40:40.4954828Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5012043Z Entering 'third_party/eigen' 2024-08-20T22:40:40.5056346Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5104923Z Entering 'third_party/fbgemm' 2024-08-20T22:40:40.5153770Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5201214Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:40:40.5245826Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5291342Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:40:40.5335170Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5380683Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:40:40.5426554Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5478616Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:40:40.5524981Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5571768Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:40:40.5615082Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5662946Z Entering 'third_party/flatbuffers' 2024-08-20T22:40:40.5709019Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5757082Z Entering 'third_party/fmt' 2024-08-20T22:40:40.5801781Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5847214Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:40:40.5894402Z http.https://github.com/.extraheader 2024-08-20T22:40:40.5939618Z Entering 'third_party/gloo' 2024-08-20T22:40:40.5983928Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6032088Z Entering 'third_party/googletest' 2024-08-20T22:40:40.6075904Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6123364Z Entering 'third_party/ideep' 2024-08-20T22:40:40.6168652Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6212319Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:40:40.6261031Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6314552Z Entering 'third_party/ittapi' 2024-08-20T22:40:40.6363286Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6410718Z Entering 'third_party/kineto' 2024-08-20T22:40:40.6461559Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6510420Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:40:40.6556890Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6602100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:40:40.6647011Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6696339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:40:40.6741626Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6787643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:40:40.6834383Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6880540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:40:40.6929639Z http.https://github.com/.extraheader 2024-08-20T22:40:40.6973870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:40:40.7023447Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7076023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:40:40.7121736Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7169791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:40:40.7214508Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7264429Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:40:40.7314891Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7363125Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:40:40.7407864Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7462146Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:40:40.7507863Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7551785Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:40:40.7595424Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7644112Z Entering 'third_party/mimalloc' 2024-08-20T22:40:40.7689003Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7735278Z Entering 'third_party/nccl/nccl' 2024-08-20T22:40:40.7780697Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7830249Z Entering 'third_party/nlohmann' 2024-08-20T22:40:40.7881026Z http.https://github.com/.extraheader 2024-08-20T22:40:40.7930522Z Entering 'third_party/onnx' 2024-08-20T22:40:40.7976336Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8038066Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:40:40.8083034Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8129027Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:40:40.8175357Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8226673Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:40:40.8271446Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8317238Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:40:40.8360639Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8405522Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:40:40.8449922Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8494366Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:40:40.8538343Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8582890Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:40:40.8626213Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8673735Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:40:40.8716784Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8763432Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:40:40.8810926Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8857109Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:40:40.8902277Z http.https://github.com/.extraheader 2024-08-20T22:40:40.8946259Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:40:40.8993309Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9044548Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:40:40.9088765Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9137705Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:40:40.9180612Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9253909Z Entering 'third_party/pocketfft' 2024-08-20T22:40:40.9302952Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9347712Z Entering 'third_party/protobuf' 2024-08-20T22:40:40.9393329Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9441614Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:40:40.9485020Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9532734Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:40:40.9580622Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9630045Z Entering 'third_party/psimd' 2024-08-20T22:40:40.9674613Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9722515Z Entering 'third_party/pthreadpool' 2024-08-20T22:40:40.9767049Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9813735Z Entering 'third_party/pybind11' 2024-08-20T22:40:40.9864114Z http.https://github.com/.extraheader 2024-08-20T22:40:40.9911608Z Entering 'third_party/python-peachpy' 2024-08-20T22:40:40.9956366Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0002265Z Entering 'third_party/sleef' 2024-08-20T22:40:41.0047208Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0094914Z Entering 'third_party/tensorpipe' 2024-08-20T22:40:41.0139202Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0183571Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:40:41.0227405Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0276666Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:40:41.0319989Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0363898Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:40:41.0407362Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0454773Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:40:41.0497884Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0540794Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:40:41.0584522Z http.https://github.com/.extraheader 2024-08-20T22:40:41.0746451Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T22:40:41.0768889Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T22:40:41.0776778Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:40:41.0777309Z ##[endgroup] 2024-08-20T22:40:48.1487360Z Cleaning up orphan processes