2024-08-20T22:04:47.6278614Z Current runner version: '2.319.1' 2024-08-20T22:04:47.6285011Z Runner name: 'i-08b1654f015502267' 2024-08-20T22:04:47.6285974Z Runner group name: 'Default' 2024-08-20T22:04:47.6286860Z Machine name: 'ip-10-0-55-192' 2024-08-20T22:04:47.6304066Z Testing runner upgrade compatibility 2024-08-20T22:04:47.7832915Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:04:47.7835392Z Actions: read 2024-08-20T22:04:47.7835956Z Attestations: read 2024-08-20T22:04:47.7836649Z Checks: read 2024-08-20T22:04:47.7837107Z Contents: read 2024-08-20T22:04:47.7837639Z Deployments: read 2024-08-20T22:04:47.7838218Z Discussions: read 2024-08-20T22:04:47.7838690Z Issues: read 2024-08-20T22:04:47.7839196Z Metadata: read 2024-08-20T22:04:47.7839748Z Packages: read 2024-08-20T22:04:47.7840248Z Pages: read 2024-08-20T22:04:47.7840717Z PullRequests: read 2024-08-20T22:04:47.7841317Z RepositoryProjects: read 2024-08-20T22:04:47.7841938Z SecurityEvents: read 2024-08-20T22:04:47.7842439Z Statuses: read 2024-08-20T22:04:47.7843352Z ##[endgroup] 2024-08-20T22:04:47.7846714Z Secret source: Actions 2024-08-20T22:04:47.7847564Z Prepare workflow directory 2024-08-20T22:04:47.9283073Z Prepare all required actions 2024-08-20T22:04:47.9455897Z Getting action download info 2024-08-20T22:04:48.1315528Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:04:48.6510304Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:04:52.6795966Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:04:52.8497758Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:04:53.1613705Z Getting action download info 2024-08-20T22:04:53.2408972Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:04:53.4270824Z Getting action download info 2024-08-20T22:04:53.5136935Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:04:53.6589944Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:04:53.6592211Z ##[group] Inputs 2024-08-20T22:04:53.6592685Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:04:53.6599911Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:04:53.6607904Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:04:53.6609093Z sync-tag: 2024-08-20T22:04:53.6609930Z timeout-minutes: 240 2024-08-20T22:04:53.6610256Z use-gha: 2024-08-20T22:04:53.6610540Z dashboard-tag: 2024-08-20T22:04:53.6610875Z s3-bucket: gha-artifacts 2024-08-20T22:04:53.6611222Z aws-role-to-assume: 2024-08-20T22:04:53.6611557Z ##[endgroup] 2024-08-20T22:04:53.6612592Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:04:53.7309771Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:04:53.7451656Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:04:53.7463251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:04:53.7463790Z ##[endgroup] 2024-08-20T22:04:55.6775735Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:04:55.6776303Z Instance Type: g5.4xlarge 2024-08-20T22:04:55.6776999Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:04:55.6777560Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:05:01.5033225Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:05:01.5033772Z with: 2024-08-20T22:05:01.5034455Z github-secret: *** 2024-08-20T22:05:01.5035396Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:01.5036380Z activate-with-label: false 2024-08-20T22:05:01.5036764Z label: with-ssh 2024-08-20T22:05:01.5037113Z remove-existing-keys: true 2024-08-20T22:05:01.5037473Z fail-silently: true 2024-08-20T22:05:01.5037787Z env: 2024-08-20T22:05:01.5038058Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:01.5038419Z ##[endgroup] 2024-08-20T22:05:01.5934975Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:05:01.5937881Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:05:01.8667747Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:05:01.9236147Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:05:01.9236771Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:05:01.9991451Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:05:02.0004866Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:05:02.0029296Z Login using: ssh ec2-user@ec2-34-224-85-126.compute-1.amazonaws.com 2024-08-20T22:05:02.0030366Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:05:02.0031207Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:02.0161913Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:05:02.0162464Z with: 2024-08-20T22:05:02.0162742Z submodules: recursive 2024-08-20T22:05:02.0163069Z fetch-depth: 0 2024-08-20T22:05:02.0163360Z env: 2024-08-20T22:05:02.0163633Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:02.0163966Z ##[endgroup] 2024-08-20T22:05:02.0355970Z ##[group]Run retry () { 2024-08-20T22:05:02.0356359Z retry () { 2024-08-20T22:05:02.0356885Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:05:02.0357446Z } 2024-08-20T22:05:02.0357753Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:05:02.0358180Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:05:02.0358665Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:02.0359127Z else 2024-08-20T22:05:02.0359472Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:02.0359901Z fi 2024-08-20T22:05:02.0360482Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:05:02.0370468Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:02.0370981Z env: 2024-08-20T22:05:02.0371267Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:02.0371625Z NO_SUDO: 2024-08-20T22:05:02.0371905Z ##[endgroup] 2024-08-20T22:05:02.0403963Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:07.8431225Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:05:07.8431666Z with: 2024-08-20T22:05:07.8432011Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:07.8432467Z fetch-depth: 0 2024-08-20T22:05:07.8432790Z submodules: recursive 2024-08-20T22:05:07.8433136Z quiet-checkout: true 2024-08-20T22:05:07.8433502Z repository: pytorch/pytorch 2024-08-20T22:05:07.8433990Z token: *** 2024-08-20T22:05:07.8434294Z ssh-strict: true 2024-08-20T22:05:07.8434623Z persist-credentials: true 2024-08-20T22:05:07.8434989Z clean: true 2024-08-20T22:05:07.8435349Z sparse-checkout-cone-mode: true 2024-08-20T22:05:07.8435749Z lfs: false 2024-08-20T22:05:07.8436055Z set-safe-directory: true 2024-08-20T22:05:07.8436406Z env: 2024-08-20T22:05:07.8436689Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:07.8437031Z ##[endgroup] 2024-08-20T22:05:07.9416995Z Syncing repository: pytorch/pytorch 2024-08-20T22:05:07.9418680Z ##[group]Getting Git version info 2024-08-20T22:05:07.9419438Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:07.9420296Z [command]/usr/bin/git version 2024-08-20T22:05:07.9420664Z git version 2.40.1 2024-08-20T22:05:07.9424251Z ##[endgroup] 2024-08-20T22:05:07.9441489Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/70237829-45a8-4dc2-99c9-e116abca6c57/.gitconfig' 2024-08-20T22:05:07.9445605Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/70237829-45a8-4dc2-99c9-e116abca6c57' before making global git config changes 2024-08-20T22:05:07.9446850Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:05:07.9450544Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:07.9508128Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:07.9512015Z ##[group]Initializing the repository 2024-08-20T22:05:07.9515029Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:07.9560708Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:05:07.9561799Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:05:07.9562613Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:05:07.9563361Z hint: 2024-08-20T22:05:07.9563832Z hint: git config --global init.defaultBranch 2024-08-20T22:05:07.9564312Z hint: 2024-08-20T22:05:07.9565176Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:05:07.9566142Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:05:07.9566739Z hint: 2024-08-20T22:05:07.9567075Z hint: git branch -m 2024-08-20T22:05:07.9567814Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:05:07.9573998Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:05:07.9618434Z ##[endgroup] 2024-08-20T22:05:07.9619128Z ##[group]Disabling automatic garbage collection 2024-08-20T22:05:07.9621671Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:05:07.9665167Z ##[endgroup] 2024-08-20T22:05:07.9665741Z ##[group]Setting up auth 2024-08-20T22:05:07.9670875Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:05:07.9715187Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:05:08.0074597Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:05:08.0118342Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:05:08.0477336Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:08.0540781Z ##[endgroup] 2024-08-20T22:05:08.0541494Z ##[group]Fetching the repository 2024-08-20T22:05:08.0547382Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:05:11.0140422Z remote: Enumerating objects: 1008509 2024-08-20T22:05:11.0141097Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:05:11.0142248Z remote: Counting objects: 0% (1/1699) 2024-08-20T22:05:11.0142968Z remote: Counting objects: 1% (17/1699) 2024-08-20T22:05:11.0143633Z remote: Counting objects: 2% (34/1699) 2024-08-20T22:05:11.0144454Z remote: Counting objects: 3% (51/1699) 2024-08-20T22:05:11.0145087Z remote: Counting objects: 4% (68/1699) 2024-08-20T22:05:11.0145586Z remote: Counting objects: 5% (85/1699) 2024-08-20T22:05:11.0146093Z remote: Counting objects: 6% (102/1699) 2024-08-20T22:05:11.0146596Z remote: Counting objects: 7% (119/1699) 2024-08-20T22:05:11.0147089Z remote: Counting objects: 8% (136/1699) 2024-08-20T22:05:11.0147719Z remote: Counting objects: 9% (153/1699) 2024-08-20T22:05:11.0148387Z remote: Counting objects: 10% (170/1699) 2024-08-20T22:05:11.0149046Z remote: Counting objects: 11% (187/1699) 2024-08-20T22:05:11.0149697Z remote: Counting objects: 12% (204/1699) 2024-08-20T22:05:11.0150339Z remote: Counting objects: 13% (221/1699) 2024-08-20T22:05:11.0150873Z remote: Counting objects: 14% (238/1699) 2024-08-20T22:05:11.0151545Z remote: Counting objects: 15% (255/1699) 2024-08-20T22:05:11.0152160Z remote: Counting objects: 16% (272/1699) 2024-08-20T22:05:11.0152658Z remote: Counting objects: 17% (289/1699) 2024-08-20T22:05:11.0153148Z remote: Counting objects: 18% (306/1699) 2024-08-20T22:05:11.0153789Z remote: Counting objects: 19% (323/1699) 2024-08-20T22:05:11.0154450Z remote: Counting objects: 20% (340/1699) 2024-08-20T22:05:11.0155095Z remote: Counting objects: 21% (357/1699) 2024-08-20T22:05:11.0155762Z remote: Counting objects: 22% (374/1699) 2024-08-20T22:05:11.0156371Z remote: Counting objects: 23% (391/1699) 2024-08-20T22:05:11.0157045Z remote: Counting objects: 24% (408/1699) 2024-08-20T22:05:11.0157694Z remote: Counting objects: 25% (425/1699) 2024-08-20T22:05:11.0158583Z remote: Counting objects: 26% (442/1699) 2024-08-20T22:05:11.0159172Z remote: Counting objects: 27% (459/1699) 2024-08-20T22:05:11.0159901Z remote: Counting objects: 28% (476/1699) 2024-08-20T22:05:11.0160533Z remote: Counting objects: 29% (493/1699) 2024-08-20T22:05:11.0161028Z remote: Counting objects: 30% (510/1699) 2024-08-20T22:05:11.0161620Z remote: Counting objects: 31% (527/1699) 2024-08-20T22:05:11.0162281Z remote: Counting objects: 32% (544/1699) 2024-08-20T22:05:11.0162849Z remote: Counting objects: 33% (561/1699) 2024-08-20T22:05:11.0163351Z remote: Counting objects: 34% (578/1699) 2024-08-20T22:05:11.0163850Z remote: Counting objects: 35% (595/1699) 2024-08-20T22:05:11.0164350Z remote: Counting objects: 36% (612/1699) 2024-08-20T22:05:11.0164840Z remote: Counting objects: 37% (629/1699) 2024-08-20T22:05:11.0165343Z remote: Counting objects: 38% (646/1699) 2024-08-20T22:05:11.0166043Z remote: Counting objects: 39% (663/1699) 2024-08-20T22:05:11.0166540Z remote: Counting objects: 40% (680/1699) 2024-08-20T22:05:11.0167033Z remote: Counting objects: 41% (697/1699) 2024-08-20T22:05:11.0167535Z remote: Counting objects: 42% (714/1699) 2024-08-20T22:05:11.0168062Z remote: Counting objects: 43% (731/1699) 2024-08-20T22:05:11.0168559Z remote: Counting objects: 44% (748/1699) 2024-08-20T22:05:11.0169047Z remote: Counting objects: 45% (765/1699) 2024-08-20T22:05:11.0169545Z remote: Counting objects: 46% (782/1699) 2024-08-20T22:05:11.0170095Z remote: Counting objects: 47% (799/1699) 2024-08-20T22:05:11.0170590Z remote: Counting objects: 48% (816/1699) 2024-08-20T22:05:11.0171078Z remote: Counting objects: 49% (833/1699) 2024-08-20T22:05:11.0171574Z remote: Counting objects: 50% (850/1699) 2024-08-20T22:05:11.0172073Z remote: Counting objects: 51% (867/1699) 2024-08-20T22:05:11.0172576Z remote: Counting objects: 52% (884/1699) 2024-08-20T22:05:11.0173064Z remote: Counting objects: 53% (901/1699) 2024-08-20T22:05:11.0173557Z remote: Counting objects: 54% (918/1699) 2024-08-20T22:05:11.0174049Z remote: Counting objects: 55% (935/1699) 2024-08-20T22:05:11.0174538Z remote: Counting objects: 56% (952/1699) 2024-08-20T22:05:11.0175025Z remote: Counting objects: 57% (969/1699) 2024-08-20T22:05:11.0175521Z remote: Counting objects: 58% (986/1699) 2024-08-20T22:05:11.0176021Z remote: Counting objects: 59% (1003/1699) 2024-08-20T22:05:11.0176539Z remote: Counting objects: 60% (1020/1699) 2024-08-20T22:05:11.0177049Z remote: Counting objects: 61% (1037/1699) 2024-08-20T22:05:11.0177566Z remote: Counting objects: 62% (1054/1699) 2024-08-20T22:05:11.0178086Z remote: Counting objects: 63% (1071/1699) 2024-08-20T22:05:11.0178613Z remote: Counting objects: 64% (1088/1699) 2024-08-20T22:05:11.0179135Z remote: Counting objects: 65% (1105/1699) 2024-08-20T22:05:11.0179638Z remote: Counting objects: 66% (1122/1699) 2024-08-20T22:05:11.0180149Z remote: Counting objects: 67% (1139/1699) 2024-08-20T22:05:11.0180658Z remote: Counting objects: 68% (1156/1699) 2024-08-20T22:05:11.0181166Z remote: Counting objects: 69% (1173/1699) 2024-08-20T22:05:11.0181667Z remote: Counting objects: 70% (1190/1699) 2024-08-20T22:05:11.0182176Z remote: Counting objects: 71% (1207/1699) 2024-08-20T22:05:11.0182682Z remote: Counting objects: 72% (1224/1699) 2024-08-20T22:05:11.0183187Z remote: Counting objects: 73% (1241/1699) 2024-08-20T22:05:11.0183692Z remote: Counting objects: 74% (1258/1699) 2024-08-20T22:05:11.0184364Z remote: Counting objects: 75% (1275/1699) 2024-08-20T22:05:11.0184867Z remote: Counting objects: 76% (1292/1699) 2024-08-20T22:05:11.0185471Z remote: Counting objects: 77% (1309/1699) 2024-08-20T22:05:11.0185980Z remote: Counting objects: 78% (1326/1699) 2024-08-20T22:05:11.0186484Z remote: Counting objects: 79% (1343/1699) 2024-08-20T22:05:11.0186987Z remote: Counting objects: 80% (1360/1699) 2024-08-20T22:05:11.0187489Z remote: Counting objects: 81% (1377/1699) 2024-08-20T22:05:11.0187988Z remote: Counting objects: 82% (1394/1699) 2024-08-20T22:05:11.0188494Z remote: Counting objects: 83% (1411/1699) 2024-08-20T22:05:11.0189518Z remote: Counting objects: 84% (1428/1699) 2024-08-20T22:05:11.0190151Z remote: Counting objects: 85% (1445/1699) 2024-08-20T22:05:11.0190652Z remote: Counting objects: 86% (1462/1699) 2024-08-20T22:05:11.0191154Z remote: Counting objects: 87% (1479/1699) 2024-08-20T22:05:11.0191656Z remote: Counting objects: 88% (1496/1699) 2024-08-20T22:05:11.0192160Z remote: Counting objects: 89% (1513/1699) 2024-08-20T22:05:11.0192825Z remote: Counting objects: 90% (1530/1699) 2024-08-20T22:05:11.0193330Z remote: Counting objects: 91% (1547/1699) 2024-08-20T22:05:11.0193835Z remote: Counting objects: 92% (1564/1699) 2024-08-20T22:05:11.0194340Z remote: Counting objects: 93% (1581/1699) 2024-08-20T22:05:11.0194836Z remote: Counting objects: 94% (1598/1699) 2024-08-20T22:05:11.0195336Z remote: Counting objects: 95% (1615/1699) 2024-08-20T22:05:11.0195838Z remote: Counting objects: 96% (1632/1699) 2024-08-20T22:05:11.0196338Z remote: Counting objects: 97% (1649/1699) 2024-08-20T22:05:11.0196833Z remote: Counting objects: 98% (1666/1699) 2024-08-20T22:05:11.0197333Z remote: Counting objects: 99% (1683/1699) 2024-08-20T22:05:11.0197835Z remote: Counting objects: 100% (1699/1699) 2024-08-20T22:05:11.0198376Z remote: Counting objects: 100% (1699/1699), done. 2024-08-20T22:05:11.0316561Z remote: Compressing objects: 0% (1/826) 2024-08-20T22:05:11.0574384Z remote: Compressing objects: 1% (9/826) 2024-08-20T22:05:11.0860917Z remote: Compressing objects: 2% (17/826) 2024-08-20T22:05:11.1341810Z remote: Compressing objects: 3% (25/826) 2024-08-20T22:05:11.2152648Z remote: Compressing objects: 4% (34/826) 2024-08-20T22:05:11.2580065Z remote: Compressing objects: 5% (42/826) 2024-08-20T22:05:11.3223045Z remote: Compressing objects: 6% (50/826) 2024-08-20T22:05:11.3643768Z remote: Compressing objects: 7% (58/826) 2024-08-20T22:05:11.3897468Z remote: Compressing objects: 8% (67/826) 2024-08-20T22:05:11.4131088Z remote: Compressing objects: 9% (75/826) 2024-08-20T22:05:11.4372297Z remote: Compressing objects: 10% (83/826) 2024-08-20T22:05:11.4582369Z remote: Compressing objects: 11% (91/826) 2024-08-20T22:05:11.4778530Z remote: Compressing objects: 12% (100/826) 2024-08-20T22:05:11.4926148Z remote: Compressing objects: 13% (108/826) 2024-08-20T22:05:11.5048221Z remote: Compressing objects: 14% (116/826) 2024-08-20T22:05:11.5136138Z remote: Compressing objects: 15% (124/826) 2024-08-20T22:05:11.5221071Z remote: Compressing objects: 16% (133/826) 2024-08-20T22:05:11.5278439Z remote: Compressing objects: 17% (141/826) 2024-08-20T22:05:11.5325228Z remote: Compressing objects: 18% (149/826) 2024-08-20T22:05:11.5380312Z remote: Compressing objects: 19% (157/826) 2024-08-20T22:05:11.5411062Z remote: Compressing objects: 20% (166/826) 2024-08-20T22:05:11.5430628Z remote: Compressing objects: 21% (174/826) 2024-08-20T22:05:11.5435108Z remote: Compressing objects: 22% (182/826) 2024-08-20T22:05:11.5440215Z remote: Compressing objects: 23% (190/826) 2024-08-20T22:05:11.5448122Z remote: Compressing objects: 24% (199/826) 2024-08-20T22:05:11.5452784Z remote: Compressing objects: 25% (207/826) 2024-08-20T22:05:11.5462759Z remote: Compressing objects: 26% (215/826) 2024-08-20T22:05:11.5471126Z remote: Compressing objects: 27% (224/826) 2024-08-20T22:05:11.5498189Z remote: Compressing objects: 28% (232/826) 2024-08-20T22:05:11.5503394Z remote: Compressing objects: 29% (240/826) 2024-08-20T22:05:11.5516257Z remote: Compressing objects: 30% (248/826) 2024-08-20T22:05:11.5525610Z remote: Compressing objects: 31% (257/826) 2024-08-20T22:05:11.5531221Z remote: Compressing objects: 32% (265/826) 2024-08-20T22:05:11.5540407Z remote: Compressing objects: 33% (273/826) 2024-08-20T22:05:11.5547832Z remote: Compressing objects: 34% (281/826) 2024-08-20T22:05:11.5554227Z remote: Compressing objects: 35% (290/826) 2024-08-20T22:05:11.5558165Z remote: Compressing objects: 36% (298/826) 2024-08-20T22:05:11.5568343Z remote: Compressing objects: 37% (306/826) 2024-08-20T22:05:11.5574808Z remote: Compressing objects: 38% (314/826) 2024-08-20T22:05:11.5580920Z remote: Compressing objects: 39% (323/826) 2024-08-20T22:05:11.5586939Z remote: Compressing objects: 40% (331/826) 2024-08-20T22:05:11.5591969Z remote: Compressing objects: 41% (339/826) 2024-08-20T22:05:11.5602093Z remote: Compressing objects: 42% (347/826) 2024-08-20T22:05:11.5605552Z remote: Compressing objects: 43% (356/826) 2024-08-20T22:05:11.5612549Z remote: Compressing objects: 44% (364/826) 2024-08-20T22:05:11.5616857Z remote: Compressing objects: 45% (372/826) 2024-08-20T22:05:11.5622506Z remote: Compressing objects: 46% (380/826) 2024-08-20T22:05:11.5625723Z remote: Compressing objects: 47% (389/826) 2024-08-20T22:05:11.5629226Z remote: Compressing objects: 48% (397/826) 2024-08-20T22:05:11.5632121Z remote: Compressing objects: 49% (405/826) 2024-08-20T22:05:11.5634617Z remote: Compressing objects: 50% (413/826) 2024-08-20T22:05:11.5638030Z remote: Compressing objects: 51% (422/826) 2024-08-20T22:05:11.5640625Z remote: Compressing objects: 52% (430/826) 2024-08-20T22:05:11.5643233Z remote: Compressing objects: 53% (438/826) 2024-08-20T22:05:11.5646056Z remote: Compressing objects: 54% (447/826) 2024-08-20T22:05:11.5646778Z remote: Compressing objects: 55% (455/826) 2024-08-20T22:05:11.5647370Z remote: Compressing objects: 56% (463/826) 2024-08-20T22:05:11.5648465Z remote: Compressing objects: 57% (471/826) 2024-08-20T22:05:11.5649167Z remote: Compressing objects: 58% (480/826) 2024-08-20T22:05:11.5649900Z remote: Compressing objects: 59% (488/826) 2024-08-20T22:05:11.5650619Z remote: Compressing objects: 60% (496/826) 2024-08-20T22:05:11.5651337Z remote: Compressing objects: 61% (504/826) 2024-08-20T22:05:11.5651992Z remote: Compressing objects: 62% (513/826) 2024-08-20T22:05:11.5652536Z remote: Compressing objects: 63% (521/826) 2024-08-20T22:05:11.5653098Z remote: Compressing objects: 64% (529/826) 2024-08-20T22:05:11.5659975Z remote: Compressing objects: 65% (537/826) 2024-08-20T22:05:11.5665303Z remote: Compressing objects: 66% (546/826) 2024-08-20T22:05:11.5669519Z remote: Compressing objects: 67% (554/826) 2024-08-20T22:05:11.5673443Z remote: Compressing objects: 68% (562/826) 2024-08-20T22:05:11.5678369Z remote: Compressing objects: 69% (570/826) 2024-08-20T22:05:11.5682660Z remote: Compressing objects: 70% (579/826) 2024-08-20T22:05:11.5685671Z remote: Compressing objects: 71% (587/826) 2024-08-20T22:05:11.5688038Z remote: Compressing objects: 72% (595/826) 2024-08-20T22:05:11.5692185Z remote: Compressing objects: 73% (603/826) 2024-08-20T22:05:11.5694904Z remote: Compressing objects: 74% (612/826) 2024-08-20T22:05:11.5697519Z remote: Compressing objects: 75% (620/826) 2024-08-20T22:05:11.5700358Z remote: Compressing objects: 76% (628/826) 2024-08-20T22:05:11.5703572Z remote: Compressing objects: 77% (637/826) 2024-08-20T22:05:11.5704858Z remote: Compressing objects: 78% (645/826) 2024-08-20T22:05:11.5706488Z remote: Compressing objects: 79% (653/826) 2024-08-20T22:05:11.5709085Z remote: Compressing objects: 80% (661/826) 2024-08-20T22:05:11.5710689Z remote: Compressing objects: 81% (670/826) 2024-08-20T22:05:11.5712717Z remote: Compressing objects: 82% (678/826) 2024-08-20T22:05:11.5714541Z remote: Compressing objects: 83% (686/826) 2024-08-20T22:05:11.5716027Z remote: Compressing objects: 84% (694/826) 2024-08-20T22:05:11.5717845Z remote: Compressing objects: 85% (703/826) 2024-08-20T22:05:11.5719882Z remote: Compressing objects: 86% (711/826) 2024-08-20T22:05:11.5721015Z remote: Compressing objects: 87% (719/826) 2024-08-20T22:05:11.5724011Z remote: Compressing objects: 88% (727/826) 2024-08-20T22:05:11.5725496Z remote: Compressing objects: 89% (736/826) 2024-08-20T22:05:11.5727425Z remote: Compressing objects: 90% (744/826) 2024-08-20T22:05:11.5728472Z remote: Compressing objects: 91% (752/826) 2024-08-20T22:05:11.5729909Z remote: Compressing objects: 92% (760/826) 2024-08-20T22:05:11.5730607Z remote: Compressing objects: 93% (769/826) 2024-08-20T22:05:11.5731568Z remote: Compressing objects: 94% (777/826) 2024-08-20T22:05:11.5732698Z remote: Compressing objects: 95% (785/826) 2024-08-20T22:05:11.5733920Z remote: Compressing objects: 96% (793/826) 2024-08-20T22:05:11.5735108Z remote: Compressing objects: 97% (802/826) 2024-08-20T22:05:11.5736544Z remote: Compressing objects: 98% (810/826) 2024-08-20T22:05:11.5737777Z remote: Compressing objects: 99% (818/826) 2024-08-20T22:05:11.5739006Z remote: Compressing objects: 100% (826/826) 2024-08-20T22:05:11.5739752Z remote: Compressing objects: 100% (826/826), done. 2024-08-20T22:05:35.0771402Z remote: Total 1010208 (delta 1106), reused 1335 (delta 870), pack-reused 1008509 (from 1) 2024-08-20T22:05:57.1175860Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:05:57.1218653Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:57.1224912Z ##[endgroup] 2024-08-20T22:05:57.1225498Z ##[group]Determining the checkout info 2024-08-20T22:05:57.1226494Z ##[endgroup] 2024-08-20T22:05:57.1227307Z ##[group]Checking out the ref 2024-08-20T22:05:57.1230552Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:58.8356304Z ##[endgroup] 2024-08-20T22:05:58.8356894Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:05:58.8361212Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:58.8423539Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:05:58.8467902Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:05:58.8509514Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:05:58.8548498Z ##[endgroup] 2024-08-20T22:05:58.8549080Z ##[group]Fetching submodules 2024-08-20T22:05:58.8552173Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:05:58.8935679Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:05:58.9306243Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:05:58.9308117Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:05:58.9311531Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:05:58.9316426Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:05:58.9319796Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:05:58.9322797Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:05:58.9326336Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:05:58.9329933Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:05:58.9333586Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:05:58.9337415Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:05:58.9341212Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:05:58.9345190Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:05:58.9349112Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:05:58.9353303Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:05:58.9357301Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:05:58.9361716Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:05:58.9369550Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:05:58.9373879Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:05:58.9377991Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:05:58.9382591Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:05:58.9387202Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:05:58.9392502Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:05:58.9396916Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:05:58.9401565Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:05:58.9406192Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:05:58.9411366Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:05:58.9416075Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:05:58.9421030Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:05:58.9426203Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:05:58.9431333Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:05:58.9436585Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:05:58.9441793Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:05:58.9446876Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:05:58.9455970Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:05:58.9489518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:05:59.2579185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:05:59.4534702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:05:59.6225095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:05:59.8614656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:06:01.8571962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:06:13.2462148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:06:13.6676485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:06:14.0845893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:06:14.6788446Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:06:15.9339097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:06:18.0416033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:06:23.9877179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:06:25.4112842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:06:27.0968974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:06:28.2980645Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:06:28.7236514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:06:29.0829283Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:06:30.3485187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:06:30.7060791Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:06:30.9787628Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:06:32.6947764Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:06:33.5189861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:06:34.2247927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:06:40.3476412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:06:42.3475533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:06:46.6025160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:06:46.8579845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:06:56.2125769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:06:56.3712537Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:06:56.5546560Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:06:57.5759371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:06:57.8624075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:06:58.4781330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:06:58.9173585Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:06:58.9326184Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:06:58.9444412Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:06:58.9764945Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:06:59.0228067Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:07:00.2179074Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:07:00.2467495Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:07:00.3001986Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:07:00.4154914Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:07:00.4563915Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:07:01.0875708Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:07:01.3781164Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:07:01.4735212Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:07:01.4757995Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:07:01.4761457Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:07:01.4765002Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:07:01.4768624Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:07:01.4772368Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:07:01.4804953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:07:02.4794605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:07:03.0873370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:07:05.2242954Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:07:06.2755212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:07:06.6226369Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:07:06.7347165Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:07:07.2498827Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:07:07.3215872Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:07:07.3370272Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:07:07.4905678Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:07:07.5382733Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:07:07.5850179Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:07:07.6178595Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:07:07.6719138Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:07.6879527Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:07:07.6900626Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:07:07.6931978Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:07:21.1438871Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:07:21.1661991Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:07:21.2667943Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:07:21.2691212Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:07:21.2694616Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:07:21.2698171Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:07:21.2729849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:07:21.8065288Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:07:23.0257066Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:07:24.2038378Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:07:24.2059678Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:07:24.2063047Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:07:24.2067270Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:07:24.2070936Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:07:24.2074625Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:07:24.2078556Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:07:24.2082384Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:07:24.2086746Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:07:24.2118891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:07:25.1237914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:07:25.5147633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:07:26.7291898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:07:27.0274118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:07:27.5744023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:07:28.6762034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:07:35.7449599Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:07:36.1616638Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:07:36.1852276Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:07:36.2290111Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:07:36.2457524Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:07:36.2479241Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:07:36.2511069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:07:36.5785446Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:07:36.6013149Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:07:36.6501281Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:07:36.7698984Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:07:36.7903388Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:07:36.8368357Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:07:36.9040129Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:07:36.9504129Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:07:36.9804968Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:07:37.1075252Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:07:37.6055710Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:07:37.6097451Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:07:37.6099029Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:07:37.6133431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:07:38.0820052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:07:38.9873821Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:07:39.0287824Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:07:39.1198221Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:07:39.1220909Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:07:39.1223944Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:07:39.1227562Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:07:39.1231203Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:07:39.1235003Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:07:39.1238615Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:07:39.1242486Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:07:39.1246106Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:07:39.1278594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:07:39.5750703Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:07:40.6227924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:07:40.9347804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:07:46.9939006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:07:47.2474159Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:07:47.4623335Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:07:47.7789666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:07:54.5917698Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:07:54.6397669Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:07:54.6587968Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:07:54.7837271Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:07:54.7999893Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:07:54.8188018Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:07:54.8397727Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:07:54.8418067Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:07:54.8421435Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:07:54.8452818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:07:56.7615514Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:07:58.0685675Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:07:58.1228784Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:58.8038779Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:07:58.8186447Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:07:59.1399885Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:07:59.1425672Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:07:59.1429028Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:07:59.1462346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:07:59.6678737Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:08:00.7548956Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:08:00.8372237Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:08:00.8492800Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:08:00.8649097Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:08:00.9103513Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:08:00.9454345Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:08:00.9949333Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:08:01.0292520Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:08:01.0315845Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:01.0317792Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:01.0321461Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:01.0324885Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:01.0355664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:08:02.1392341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:08:02.3845918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:08:03.6903296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:08:04.6771029Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:08:04.6965451Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:08:04.7759382Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:08:04.8108576Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:08:04.8129268Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:04.8160504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:08:05.0216998Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:08:05.0262573Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:08:05.0633777Z Entering 'android/libs/fbjni' 2024-08-20T22:08:05.0686690Z Entering 'third_party/FP16' 2024-08-20T22:08:05.0742239Z Entering 'third_party/FXdiv' 2024-08-20T22:08:05.0796029Z Entering 'third_party/NNPACK' 2024-08-20T22:08:05.0851416Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:05.0906680Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:05.0975218Z Entering 'third_party/benchmark' 2024-08-20T22:08:05.1029305Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:05.1082005Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:05.1135812Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:05.1189353Z Entering 'third_party/cutlass' 2024-08-20T22:08:05.1251540Z Entering 'third_party/eigen' 2024-08-20T22:08:05.1307285Z Entering 'third_party/fbgemm' 2024-08-20T22:08:05.1359910Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:05.1410907Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:05.1462061Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:05.1520891Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:05.1571340Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:05.1626226Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:05.1681712Z Entering 'third_party/fmt' 2024-08-20T22:08:05.1735315Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:05.1788366Z Entering 'third_party/gloo' 2024-08-20T22:08:05.1841852Z Entering 'third_party/googletest' 2024-08-20T22:08:05.1894738Z Entering 'third_party/ideep' 2024-08-20T22:08:05.1945852Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:05.2005386Z Entering 'third_party/ittapi' 2024-08-20T22:08:05.2057683Z Entering 'third_party/kineto' 2024-08-20T22:08:05.2110153Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:05.2159831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:05.2213910Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:05.2264811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:05.2316732Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:05.2365672Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:05.2422111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:05.2471717Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:05.2525248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:05.2576886Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:05.2633918Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:05.2684872Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:05.2746904Z Entering 'third_party/mimalloc' 2024-08-20T22:08:05.2800404Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:05.2852652Z Entering 'third_party/nlohmann' 2024-08-20T22:08:05.2905912Z Entering 'third_party/onnx' 2024-08-20T22:08:05.2970984Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:05.3023312Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:05.3080403Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:05.3135201Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:05.3185403Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:05.3237240Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:05.3288066Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:05.3345319Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:05.3395740Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:05.3446056Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:05.3495298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:05.3551064Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:05.3605942Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:05.3679906Z Entering 'third_party/pocketfft' 2024-08-20T22:08:05.3734558Z Entering 'third_party/protobuf' 2024-08-20T22:08:05.3790006Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:05.3841674Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:05.3897246Z Entering 'third_party/psimd' 2024-08-20T22:08:05.3950109Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:05.4003830Z Entering 'third_party/pybind11' 2024-08-20T22:08:05.4057500Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:05.4110868Z Entering 'third_party/sleef' 2024-08-20T22:08:05.4162324Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:05.4213503Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:05.4264845Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:05.4315605Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:05.4370339Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:05.4419487Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:05.4496994Z ##[endgroup] 2024-08-20T22:08:05.4499931Z ##[group]Persisting credentials for submodules 2024-08-20T22:08:05.4503140Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:08:05.4866365Z Entering 'android/libs/fbjni' 2024-08-20T22:08:05.4934313Z Entering 'third_party/FP16' 2024-08-20T22:08:05.5010083Z Entering 'third_party/FXdiv' 2024-08-20T22:08:05.5076684Z Entering 'third_party/NNPACK' 2024-08-20T22:08:05.5144322Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:05.5211686Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:05.5294471Z Entering 'third_party/benchmark' 2024-08-20T22:08:05.5361396Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:05.5429066Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:05.5497122Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:05.5564208Z Entering 'third_party/cutlass' 2024-08-20T22:08:05.5638986Z Entering 'third_party/eigen' 2024-08-20T22:08:05.5709336Z Entering 'third_party/fbgemm' 2024-08-20T22:08:05.5776399Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:05.5844885Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:05.5913336Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:05.5986542Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:05.6054587Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:05.6124673Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:05.6194391Z Entering 'third_party/fmt' 2024-08-20T22:08:05.6261153Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:05.6329020Z Entering 'third_party/gloo' 2024-08-20T22:08:05.6396226Z Entering 'third_party/googletest' 2024-08-20T22:08:05.6464998Z Entering 'third_party/ideep' 2024-08-20T22:08:05.6530267Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:05.6607158Z Entering 'third_party/ittapi' 2024-08-20T22:08:05.6673644Z Entering 'third_party/kineto' 2024-08-20T22:08:05.6740045Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:05.6805430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:05.6873955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:05.6941656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:05.7008287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:05.7073499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:05.7145587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:05.7218203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:05.7284824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:05.7352564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:05.7422938Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:05.7491033Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:05.7562251Z Entering 'third_party/mimalloc' 2024-08-20T22:08:05.7631416Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:05.7699006Z Entering 'third_party/nlohmann' 2024-08-20T22:08:05.7766784Z Entering 'third_party/onnx' 2024-08-20T22:08:05.7846975Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:05.7916370Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:05.7992401Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:05.8058666Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:05.8126351Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:05.8193392Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:05.8259790Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:05.8328603Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:05.8395739Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:05.8461725Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:05.8527137Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:05.8596999Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:05.8666261Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:05.8757320Z Entering 'third_party/pocketfft' 2024-08-20T22:08:05.8824368Z Entering 'third_party/protobuf' 2024-08-20T22:08:05.8895252Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:05.8962459Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:05.9033921Z Entering 'third_party/psimd' 2024-08-20T22:08:05.9100826Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:05.9168163Z Entering 'third_party/pybind11' 2024-08-20T22:08:05.9235943Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:05.9303336Z Entering 'third_party/sleef' 2024-08-20T22:08:05.9370342Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:05.9436090Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:05.9503746Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:05.9569456Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:05.9636291Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:05.9700107Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:05.9794435Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:08:06.0156822Z Entering 'android/libs/fbjni' 2024-08-20T22:08:06.0219000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:08:06.0241128Z Entering 'third_party/FP16' 2024-08-20T22:08:06.0305552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:08:06.0328652Z Entering 'third_party/FXdiv' 2024-08-20T22:08:06.0392546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:08:06.0414835Z Entering 'third_party/NNPACK' 2024-08-20T22:08:06.0477610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:08:06.0500433Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:06.0563168Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:08:06.0585075Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:06.0649298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:08:06.0685361Z Entering 'third_party/benchmark' 2024-08-20T22:08:06.0748951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:06.0770769Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:06.0836485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:08:06.0858056Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:06.0924933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:06.0947628Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:06.1012437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:08:06.1034573Z Entering 'third_party/cutlass' 2024-08-20T22:08:06.1098546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:06.1129811Z Entering 'third_party/eigen' 2024-08-20T22:08:06.1191602Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:08:06.1217333Z Entering 'third_party/fbgemm' 2024-08-20T22:08:06.1284137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:08:06.1305214Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:06.1366644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:08:06.1391287Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:06.1452844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:06.1472768Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:06.1534508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:06.1562157Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:06.1623491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.1643555Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:06.1708081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:08:06.1730228Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:06.1795074Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:08:06.1818371Z Entering 'third_party/fmt' 2024-08-20T22:08:06.1882817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:06.1905231Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:06.1968273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:08:06.1990978Z Entering 'third_party/gloo' 2024-08-20T22:08:06.2053868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:08:06.2076211Z Entering 'third_party/googletest' 2024-08-20T22:08:06.2139621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.2161809Z Entering 'third_party/ideep' 2024-08-20T22:08:06.2226041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:08:06.2245435Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:06.2306773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:08:06.2336106Z Entering 'third_party/ittapi' 2024-08-20T22:08:06.2398350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:08:06.2420066Z Entering 'third_party/kineto' 2024-08-20T22:08:06.2483469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:08:06.2504482Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:06.2569774Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:08:06.2589194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:06.2654041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:08:06.2674767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:06.2740383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:08:06.2759064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:06.2822187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:06.2843247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:06.2906797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:08:06.2924972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:06.2989367Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:08:06.3015931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:06.3078576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:08:06.3099868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:06.3162199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.3182657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:06.3246261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:08:06.3267893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:06.3330925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:08:06.3354185Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:06.3416887Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:08:06.3437039Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:06.3499206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.3522218Z Entering 'third_party/mimalloc' 2024-08-20T22:08:06.3585208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:08:06.3609618Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:06.3672797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:08:06.3696094Z Entering 'third_party/nlohmann' 2024-08-20T22:08:06.3758955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:08:06.3782432Z Entering 'third_party/onnx' 2024-08-20T22:08:06.3848194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:08:06.3884042Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:06.3947619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:06.3969484Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:06.4032616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:06.4057764Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:06.4122879Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:08:06.4145043Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:06.4206627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:06.4227031Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:06.4293265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.4313653Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:06.4374817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:08:06.4400112Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:06.4461451Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:08:06.4482924Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:06.4545355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:08:06.4565248Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:06.4627013Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:08:06.4650311Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:06.4712629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:08:06.4730526Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:06.4793358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:08:06.4815222Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:06.4877686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:08:06.4900928Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:06.4961981Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:08:06.5003517Z Entering 'third_party/pocketfft' 2024-08-20T22:08:06.5067631Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:08:06.5090363Z Entering 'third_party/protobuf' 2024-08-20T22:08:06.5154372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:08:06.5177653Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:06.5244221Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:06.5269856Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:06.5334407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.5358379Z Entering 'third_party/psimd' 2024-08-20T22:08:06.5422652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:08:06.5445087Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:06.5510941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:08:06.5533232Z Entering 'third_party/pybind11' 2024-08-20T22:08:06.5597404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:06.5619924Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:06.5682766Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:08:06.5705755Z Entering 'third_party/sleef' 2024-08-20T22:08:06.5770416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:08:06.5793626Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:06.5858020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:08:06.5878878Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:06.5942008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:06.5962162Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:06.6024021Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:08:06.6044877Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:06.6109983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:08:06.6130819Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:06.6192473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:06.6211551Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:06.6274402Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:08:06.7335084Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:08:06.7702564Z Entering 'android/libs/fbjni' 2024-08-20T22:08:06.7754141Z Entering 'third_party/FP16' 2024-08-20T22:08:06.7807841Z Entering 'third_party/FXdiv' 2024-08-20T22:08:06.7860723Z Entering 'third_party/NNPACK' 2024-08-20T22:08:06.7914304Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:06.7967864Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:06.8038841Z Entering 'third_party/benchmark' 2024-08-20T22:08:06.8092012Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:06.8144822Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:06.8198820Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:06.8251876Z Entering 'third_party/cutlass' 2024-08-20T22:08:06.8313247Z Entering 'third_party/eigen' 2024-08-20T22:08:06.8368090Z Entering 'third_party/fbgemm' 2024-08-20T22:08:06.8420773Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:06.8472379Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:06.8524649Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:06.8582300Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:06.8635306Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:06.8695625Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:06.8751418Z Entering 'third_party/fmt' 2024-08-20T22:08:06.8803114Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:06.8855343Z Entering 'third_party/gloo' 2024-08-20T22:08:06.8908782Z Entering 'third_party/googletest' 2024-08-20T22:08:06.8960897Z Entering 'third_party/ideep' 2024-08-20T22:08:06.9011385Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:06.9072073Z Entering 'third_party/ittapi' 2024-08-20T22:08:06.9125388Z Entering 'third_party/kineto' 2024-08-20T22:08:06.9177441Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:06.9227663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:06.9280791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:06.9335149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:06.9385871Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:06.9434892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:06.9490618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:06.9542249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:06.9593516Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:06.9645779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:06.9700571Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:06.9751700Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:06.9808270Z Entering 'third_party/mimalloc' 2024-08-20T22:08:06.9859922Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:06.9914350Z Entering 'third_party/nlohmann' 2024-08-20T22:08:06.9967212Z Entering 'third_party/onnx' 2024-08-20T22:08:07.0031515Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:07.0082857Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:07.0140533Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:07.0196007Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:07.0247199Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:07.0299742Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:07.0351185Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:07.0403798Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:07.0454377Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:07.0505177Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:07.0554439Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:07.0608911Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:07.0663014Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:07.0737709Z Entering 'third_party/pocketfft' 2024-08-20T22:08:07.0790198Z Entering 'third_party/protobuf' 2024-08-20T22:08:07.0844530Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:07.0895361Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:07.0952186Z Entering 'third_party/psimd' 2024-08-20T22:08:07.1007825Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:07.1060109Z Entering 'third_party/pybind11' 2024-08-20T22:08:07.1112874Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:07.1165111Z Entering 'third_party/sleef' 2024-08-20T22:08:07.1220252Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:07.1271492Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:07.1322660Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:07.1373232Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:07.1424307Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:07.1472511Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:07.1551419Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:08:07.1919910Z Entering 'android/libs/fbjni' 2024-08-20T22:08:07.1972158Z Entering 'third_party/FP16' 2024-08-20T22:08:07.2026064Z Entering 'third_party/FXdiv' 2024-08-20T22:08:07.2077436Z Entering 'third_party/NNPACK' 2024-08-20T22:08:07.2133981Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:07.2188475Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:07.2256741Z Entering 'third_party/benchmark' 2024-08-20T22:08:07.2309710Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:07.2361430Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:07.2414248Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:07.2466376Z Entering 'third_party/cutlass' 2024-08-20T22:08:07.2525219Z Entering 'third_party/eigen' 2024-08-20T22:08:07.2579105Z Entering 'third_party/fbgemm' 2024-08-20T22:08:07.2633231Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:07.2684561Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:07.2736737Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:07.2799988Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:07.2851512Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:07.2906754Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:07.2962296Z Entering 'third_party/fmt' 2024-08-20T22:08:07.3015576Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:07.3069783Z Entering 'third_party/gloo' 2024-08-20T22:08:07.3122025Z Entering 'third_party/googletest' 2024-08-20T22:08:07.3176140Z Entering 'third_party/ideep' 2024-08-20T22:08:07.3230789Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:07.3290421Z Entering 'third_party/ittapi' 2024-08-20T22:08:07.3342794Z Entering 'third_party/kineto' 2024-08-20T22:08:07.3393899Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:07.3445000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:07.3498924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:07.3550050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:07.3601200Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:07.3650122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:07.3707373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:07.3761047Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:07.3812969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:07.3865058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:07.3921286Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:07.3972446Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:07.4027812Z Entering 'third_party/mimalloc' 2024-08-20T22:08:07.4080883Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:07.4134181Z Entering 'third_party/nlohmann' 2024-08-20T22:08:07.4189915Z Entering 'third_party/onnx' 2024-08-20T22:08:07.4256886Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:07.4308731Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:07.4365467Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:07.4419759Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:07.4473290Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:07.4524301Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:07.4574492Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:07.4631738Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:07.4681741Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:07.4733667Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:07.4782652Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:07.4836301Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:07.4890765Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:07.4971563Z Entering 'third_party/pocketfft' 2024-08-20T22:08:07.5026299Z Entering 'third_party/protobuf' 2024-08-20T22:08:07.5081914Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:07.5133243Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:07.5188512Z Entering 'third_party/psimd' 2024-08-20T22:08:07.5242722Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:07.5296734Z Entering 'third_party/pybind11' 2024-08-20T22:08:07.5350070Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:07.5404560Z Entering 'third_party/sleef' 2024-08-20T22:08:07.5457975Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:07.5510952Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:07.5561325Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:07.5612265Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:07.5663049Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:07.5714190Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:07.5789245Z ##[endgroup] 2024-08-20T22:08:07.5839846Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:08:07.5877002Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:08:07.6082502Z Prepare all required actions 2024-08-20T22:08:07.6082952Z Getting action download info 2024-08-20T22:08:07.7412410Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:08:07.7412835Z env: 2024-08-20T22:08:07.7413111Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.7413446Z ##[endgroup] 2024-08-20T22:08:07.7473087Z ##[group]Run set -euo pipefail 2024-08-20T22:08:07.7473528Z set -euo pipefail 2024-08-20T22:08:07.7473927Z function get_ec2_metadata() { 2024-08-20T22:08:07.7474470Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:08:07.7475369Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:08:07.7476143Z  category=$1 2024-08-20T22:08:07.7476729Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:08:07.7477370Z  runner_name_str=i-08b1654f015502267 2024-08-20T22:08:07.7477859Z  if [[ -f /.inarc ]]; then 2024-08-20T22:08:07.7478363Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:08:07.7478939Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:08:07.7479646Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:08:07.7480261Z  else 2024-08-20T22:08:07.7480753Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:08:07.7481339Z  fi 2024-08-20T22:08:07.7481630Z } 2024-08-20T22:08:07.7481988Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:08:07.7482579Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:08:07.7483237Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:08:07.7483813Z echo "system info $(uname -a)" 2024-08-20T22:08:07.7493259Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:07.7493749Z env: 2024-08-20T22:08:07.7494032Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.7494377Z ##[endgroup] 2024-08-20T22:08:07.7587689Z ami-id: ami-06c68f701d8090592 2024-08-20T22:08:07.7652114Z instance-id: i-08b1654f015502267 2024-08-20T22:08:07.7709492Z instance-type: g5.4xlarge 2024-08-20T22:08:07.7722887Z system info Linux ip-10-0-55-192.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:08:07.7746843Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:07.7747760Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:07.7756686Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:07.7757178Z env: 2024-08-20T22:08:07.7757460Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.7757806Z ##[endgroup] 2024-08-20T22:08:07.7832848Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:08:07.7833453Z if systemctl is-active --quiet docker; then 2024-08-20T22:08:07.7833991Z  echo "Docker daemon is running..."; 2024-08-20T22:08:07.7834611Z else 2024-08-20T22:08:07.7835096Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:08:07.7835675Z fi 2024-08-20T22:08:07.7843716Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:07.7844203Z env: 2024-08-20T22:08:07.7844478Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.7844822Z ##[endgroup] 2024-08-20T22:08:07.7928194Z Docker daemon is running... 2024-08-20T22:08:07.7975079Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:08:07.7975625Z with: 2024-08-20T22:08:07.7975888Z shell: bash 2024-08-20T22:08:07.7976174Z timeout_minutes: 5 2024-08-20T22:08:07.7976510Z max_attempts: 3 2024-08-20T22:08:07.7976851Z retry_wait_seconds: 30 2024-08-20T22:08:07.7979936Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:08:07.7982874Z polling_interval_seconds: 1 2024-08-20T22:08:07.7983254Z warning_on_retry: true 2024-08-20T22:08:07.7983597Z continue_on_error: false 2024-08-20T22:08:07.7984074Z env: 2024-08-20T22:08:07.7984361Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:07.7984716Z AWS_RETRY_MODE: standard 2024-08-20T22:08:07.7985060Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:08:07.7985404Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:08:07.7985781Z ##[endgroup] 2024-08-20T22:08:09.1470218Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:09.1471337Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:09.1472449Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:09.1473145Z 2024-08-20T22:08:09.1473311Z Login Succeeded 2024-08-20T22:08:09.8545824Z Command completed after 1 attempt(s). 2024-08-20T22:08:09.8609800Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:09.8610524Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:09.8611182Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:09.8620180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:09.8620673Z env: 2024-08-20T22:08:09.8620956Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:09.8621308Z ##[endgroup] 2024-08-20T22:08:09.8718679Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:09.8719443Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:09.8720039Z # shellcheck disable=SC2046 2024-08-20T22:08:09.8720502Z docker stop $(docker ps -q) || true 2024-08-20T22:08:09.8720976Z # Prune all of the docker images 2024-08-20T22:08:09.8721421Z docker system prune -af 2024-08-20T22:08:09.8730368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:09.8730860Z env: 2024-08-20T22:08:09.8731131Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:09.8731480Z ##[endgroup] 2024-08-20T22:08:09.9054389Z "docker stop" requires at least 1 argument. 2024-08-20T22:08:09.9055138Z See 'docker stop --help'. 2024-08-20T22:08:09.9055375Z 2024-08-20T22:08:09.9055598Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:08:09.9055975Z 2024-08-20T22:08:09.9056125Z Stop one or more running containers 2024-08-20T22:08:09.9236339Z Total reclaimed space: 0B 2024-08-20T22:08:09.9280958Z ##[group]Run set +e 2024-08-20T22:08:09.9281309Z set +e 2024-08-20T22:08:09.9281796Z set -x 2024-08-20T22:08:09.9282088Z  2024-08-20T22:08:09.9282403Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:09.9283189Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:08:09.9284265Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:08:09.9285019Z # one is returned at random 2024-08-20T22:08:09.9285561Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:08:09.9286070Z  2024-08-20T22:08:09.9286391Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:09.9287010Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:08:09.9287962Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:08:09.9288518Z  2024-08-20T22:08:09.9289251Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:09.9289823Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:08:09.9290333Z  exit 1 2024-08-20T22:08:09.9290641Z  fi 2024-08-20T22:08:09.9290925Z fi 2024-08-20T22:08:09.9291202Z  2024-08-20T22:08:09.9291561Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:08:09.9292075Z  # Clean up any old records first 2024-08-20T22:08:09.9292589Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:08:09.9293048Z fi 2024-08-20T22:08:09.9293318Z  2024-08-20T22:08:09.9293752Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:08:09.9294300Z cat /etc/hosts 2024-08-20T22:08:09.9303302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:09.9303807Z env: 2024-08-20T22:08:09.9304187Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:09.9304539Z ##[endgroup] 2024-08-20T22:08:09.9333341Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:09.9339578Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:08:09.9340036Z ++ tail -n1 2024-08-20T22:08:09.9659053Z + RESOLVED_IP=18.160.10.22 2024-08-20T22:08:09.9659504Z + '[' -z 18.160.10.22 ']' 2024-08-20T22:08:09.9659952Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:08:09.9672694Z 18.160.10.28 download.pytorch.org 2024-08-20T22:08:09.9674932Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T22:08:10.0488973Z + echo '18.160.10.22 download.pytorch.org' 2024-08-20T22:08:10.0489546Z + sudo tee -a /etc/hosts 2024-08-20T22:08:10.0812676Z 18.160.10.22 download.pytorch.org 2024-08-20T22:08:10.0835071Z + cat /etc/hosts 2024-08-20T22:08:10.0846025Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:08:10.0858733Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:08:10.0859241Z 18.160.10.22 download.pytorch.org 2024-08-20T22:08:10.1010347Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:08:10.1010953Z with: 2024-08-20T22:08:10.1012049Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1013265Z docker-build-dir: .ci/docker 2024-08-20T22:08:10.1013643Z working-directory: . 2024-08-20T22:08:10.1014105Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.1014623Z force-push: false 2024-08-20T22:08:10.1014923Z env: 2024-08-20T22:08:10.1015197Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.1015535Z ##[endgroup] 2024-08-20T22:08:10.1035074Z ##[group]Run set -ex 2024-08-20T22:08:10.1035435Z set -ex 2024-08-20T22:08:10.1035728Z  2024-08-20T22:08:10.1036292Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:08:10.1037281Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:08:10.1038268Z # job could then download the pre-built image as usual 2024-08-20T22:08:10.1039021Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:08:10.1039701Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1040347Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1040916Z  2024-08-20T22:08:10.1041430Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:08:10.1042059Z  exit 0 2024-08-20T22:08:10.1042367Z else 2024-08-20T22:08:10.1042734Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1043195Z fi 2024-08-20T22:08:10.1043480Z  2024-08-20T22:08:10.1043959Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:08:10.1044812Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:08:10.1045611Z  # use it as it is, but first let's extract the tag 2024-08-20T22:08:10.1046326Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:08:10.1047059Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1047817Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1048379Z else 2024-08-20T22:08:10.1048825Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:08:10.1049507Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1050415Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1051166Z fi 2024-08-20T22:08:10.1059868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.1060363Z env: 2024-08-20T22:08:10.1060650Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.1061008Z REPO_NAME: pytorch 2024-08-20T22:08:10.1062131Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1063325Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:10.1063823Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.1064419Z ##[endgroup] 2024-08-20T22:08:10.1093576Z + [[ ! -d .ci/docker ]] 2024-08-20T22:08:10.1093999Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:08:10.1094373Z + echo skip=false 2024-08-20T22:08:10.1096396Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:08:10.1101824Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1103069Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:08:10.1127694Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1128371Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1129818Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1164854Z ##[group]Run set +e 2024-08-20T22:08:10.1165395Z set +e 2024-08-20T22:08:10.1165853Z set -x 2024-08-20T22:08:10.1166289Z  2024-08-20T22:08:10.1166679Z login() { 2024-08-20T22:08:10.1167412Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:10.1168322Z } 2024-08-20T22:08:10.1168713Z  2024-08-20T22:08:10.1169284Z retry () { 2024-08-20T22:08:10.1169784Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:10.1170346Z } 2024-08-20T22:08:10.1170719Z  2024-08-20T22:08:10.1171201Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:10.1171677Z  2024-08-20T22:08:10.1172328Z # Check if image already exists, if it does then skip building it 2024-08-20T22:08:10.1173198Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:08:10.1173792Z  exit 0 2024-08-20T22:08:10.1174153Z fi 2024-08-20T22:08:10.1174608Z  2024-08-20T22:08:10.1175198Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:08:10.1176130Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:08:10.1176980Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:08:10.1177741Z  # if we're on the base branch then use the parent commit 2024-08-20T22:08:10.1178453Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:08:10.1179025Z else 2024-08-20T22:08:10.1179623Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:08:10.1180422Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:08:10.1181066Z fi 2024-08-20T22:08:10.1181439Z  2024-08-20T22:08:10.1181839Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:08:10.1182480Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1183048Z  2024-08-20T22:08:10.1183926Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:08:10.1184740Z  exit 0 2024-08-20T22:08:10.1185159Z fi 2024-08-20T22:08:10.1185582Z  2024-08-20T22:08:10.1186190Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:08:10.1187237Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:08:10.1188221Z  exit 1 2024-08-20T22:08:10.1188620Z fi 2024-08-20T22:08:10.1189534Z  2024-08-20T22:08:10.1190149Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:08:10.1191211Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:08:10.1192166Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:08:10.1193265Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:08:10.1194468Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:08:10.1195363Z fi 2024-08-20T22:08:10.1195901Z  2024-08-20T22:08:10.1196359Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:10.1206315Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.1206872Z env: 2024-08-20T22:08:10.1207248Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.1207804Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:10.1208340Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:10.1209696Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1211124Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.1211821Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.1212498Z ##[endgroup] 2024-08-20T22:08:10.1242860Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.1243697Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.1245875Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:10.1247570Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.6612317Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:10.6613280Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:10.6614306Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:10.6614998Z 2024-08-20T22:08:10.6615158Z Login Succeeded 2024-08-20T22:08:10.6636955Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.8826314Z { 2024-08-20T22:08:10.8826867Z "schemaVersion": 2, 2024-08-20T22:08:10.8827999Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:08:10.8828642Z "config": { 2024-08-20T22:08:10.8829624Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:08:10.8830299Z "size": 48711, 2024-08-20T22:08:10.8831361Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:08:10.8832176Z }, 2024-08-20T22:08:10.8832558Z "layers": [ 2024-08-20T22:08:10.8832940Z { 2024-08-20T22:08:10.8833674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8834336Z "size": 28580681, 2024-08-20T22:08:10.8835073Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:08:10.8835860Z }, 2024-08-20T22:08:10.8836224Z { 2024-08-20T22:08:10.8837297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8838502Z "size": 7943451, 2024-08-20T22:08:10.8839434Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:08:10.8840406Z }, 2024-08-20T22:08:10.8840942Z { 2024-08-20T22:08:10.8841900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8842852Z "size": 55728572, 2024-08-20T22:08:10.8843921Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:08:10.8845521Z }, 2024-08-20T22:08:10.8845970Z { 2024-08-20T22:08:10.8846537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8847245Z "size": 186, 2024-08-20T22:08:10.8847930Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:08:10.8848625Z }, 2024-08-20T22:08:10.8849021Z { 2024-08-20T22:08:10.8849549Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8850273Z "size": 6886, 2024-08-20T22:08:10.8850871Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:08:10.8853169Z }, 2024-08-20T22:08:10.8853895Z { 2024-08-20T22:08:10.8854435Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8855074Z "size": 1329001756, 2024-08-20T22:08:10.8855813Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:08:10.8856527Z }, 2024-08-20T22:08:10.8856892Z { 2024-08-20T22:08:10.8857600Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8858326Z "size": 62501, 2024-08-20T22:08:10.8859026Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:08:10.8859760Z }, 2024-08-20T22:08:10.8860097Z { 2024-08-20T22:08:10.8860669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8861313Z "size": 1684, 2024-08-20T22:08:10.8861957Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:08:10.8862666Z }, 2024-08-20T22:08:10.8863035Z { 2024-08-20T22:08:10.8863576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8864589Z "size": 1523, 2024-08-20T22:08:10.8865266Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:08:10.8865990Z }, 2024-08-20T22:08:10.8866406Z { 2024-08-20T22:08:10.8866925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8867601Z "size": 2528295403, 2024-08-20T22:08:10.8868360Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:08:10.8869060Z }, 2024-08-20T22:08:10.8869364Z { 2024-08-20T22:08:10.8869959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8870602Z "size": 86016, 2024-08-20T22:08:10.8871208Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:08:10.8871987Z }, 2024-08-20T22:08:10.8872336Z { 2024-08-20T22:08:10.8872947Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8873585Z "size": 1822, 2024-08-20T22:08:10.8874233Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:08:10.8875010Z }, 2024-08-20T22:08:10.8875358Z { 2024-08-20T22:08:10.8875848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8876566Z "size": 246761563, 2024-08-20T22:08:10.8877238Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:08:10.8878004Z }, 2024-08-20T22:08:10.8878385Z { 2024-08-20T22:08:10.8878906Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8879573Z "size": 544, 2024-08-20T22:08:10.8880209Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:08:10.8880934Z }, 2024-08-20T22:08:10.8881306Z { 2024-08-20T22:08:10.8881878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8882481Z "size": 1258, 2024-08-20T22:08:10.8883126Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:08:10.8883889Z }, 2024-08-20T22:08:10.8884238Z { 2024-08-20T22:08:10.8884749Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8885446Z "size": 484, 2024-08-20T22:08:10.8886089Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:08:10.8886741Z }, 2024-08-20T22:08:10.8887162Z { 2024-08-20T22:08:10.8887709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8888360Z "size": 91711766, 2024-08-20T22:08:10.8889400Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:08:10.8890132Z }, 2024-08-20T22:08:10.8890481Z { 2024-08-20T22:08:10.8891075Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8891704Z "size": 3390, 2024-08-20T22:08:10.8892502Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:08:10.8893300Z }, 2024-08-20T22:08:10.8893643Z { 2024-08-20T22:08:10.8894133Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8894881Z "size": 1909, 2024-08-20T22:08:10.8895507Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:08:10.8896159Z }, 2024-08-20T22:08:10.8896569Z { 2024-08-20T22:08:10.8897104Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8897867Z "size": 701, 2024-08-20T22:08:10.8898467Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:10.8899179Z }, 2024-08-20T22:08:10.8899605Z { 2024-08-20T22:08:10.8900138Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8900750Z "size": 2864254879, 2024-08-20T22:08:10.8901521Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:08:10.8902370Z }, 2024-08-20T22:08:10.8902707Z { 2024-08-20T22:08:10.8903289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8904019Z "size": 380, 2024-08-20T22:08:10.8904671Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:08:10.8905387Z }, 2024-08-20T22:08:10.8905739Z { 2024-08-20T22:08:10.8906294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8906992Z "size": 12900, 2024-08-20T22:08:10.8907628Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:08:10.8908422Z }, 2024-08-20T22:08:10.8908849Z { 2024-08-20T22:08:10.8909394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8910033Z "size": 803, 2024-08-20T22:08:10.8910722Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:08:10.8911397Z }, 2024-08-20T22:08:10.8911747Z { 2024-08-20T22:08:10.8912364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8912972Z "size": 106, 2024-08-20T22:08:10.8913595Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:08:10.8914369Z }, 2024-08-20T22:08:10.8914733Z { 2024-08-20T22:08:10.8915216Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8915932Z "size": 503, 2024-08-20T22:08:10.8916564Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:08:10.8917311Z }, 2024-08-20T22:08:10.8917643Z { 2024-08-20T22:08:10.8918168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8918885Z "size": 121477267, 2024-08-20T22:08:10.8919494Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:08:10.8920200Z }, 2024-08-20T22:08:10.8920612Z { 2024-08-20T22:08:10.8921142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8921788Z "size": 109, 2024-08-20T22:08:10.8922495Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:08:10.8923205Z }, 2024-08-20T22:08:10.8923547Z { 2024-08-20T22:08:10.8924108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8924760Z "size": 490, 2024-08-20T22:08:10.8925398Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:08:10.8926151Z }, 2024-08-20T22:08:10.8926461Z { 2024-08-20T22:08:10.8927006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8927746Z "size": 297, 2024-08-20T22:08:10.8928339Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:08:10.8929081Z }, 2024-08-20T22:08:10.8929601Z { 2024-08-20T22:08:10.8930145Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8930751Z "size": 103, 2024-08-20T22:08:10.8931636Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:08:10.8932377Z }, 2024-08-20T22:08:10.8932718Z { 2024-08-20T22:08:10.8933266Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8933937Z "size": 1473, 2024-08-20T22:08:10.8934575Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:08:10.8935285Z }, 2024-08-20T22:08:10.8935677Z { 2024-08-20T22:08:10.8936210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8936924Z "size": 594729516, 2024-08-20T22:08:10.8937549Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:08:10.8938273Z }, 2024-08-20T22:08:10.8938684Z { 2024-08-20T22:08:10.8939218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8940168Z + exit 0 2024-08-20T22:08:10.8940542Z "size": 164, 2024-08-20T22:08:10.8941130Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:08:10.8941906Z }, 2024-08-20T22:08:10.8942281Z { 2024-08-20T22:08:10.8942816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8943499Z "size": 564, 2024-08-20T22:08:10.8944207Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:08:10.8944929Z }, 2024-08-20T22:08:10.8945284Z { 2024-08-20T22:08:10.8945817Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8946480Z "size": 35874203, 2024-08-20T22:08:10.8947192Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:08:10.8947871Z }, 2024-08-20T22:08:10.8948242Z { 2024-08-20T22:08:10.8948855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8949508Z "size": 104, 2024-08-20T22:08:10.8950115Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:08:10.8950878Z }, 2024-08-20T22:08:10.8951242Z { 2024-08-20T22:08:10.8951732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8952424Z "size": 425, 2024-08-20T22:08:10.8953091Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:08:10.8953821Z }, 2024-08-20T22:08:10.8954180Z { 2024-08-20T22:08:10.8954734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8955385Z "size": 20262183, 2024-08-20T22:08:10.8956105Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:08:10.8956787Z }, 2024-08-20T22:08:10.8957130Z { 2024-08-20T22:08:10.8957773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8958420Z "size": 438, 2024-08-20T22:08:10.8959015Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:08:10.8959778Z }, 2024-08-20T22:08:10.8960120Z { 2024-08-20T22:08:10.8960680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8961393Z "size": 701, 2024-08-20T22:08:10.8962022Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:10.8962825Z }, 2024-08-20T22:08:10.8963135Z { 2024-08-20T22:08:10.8963655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8964371Z "size": 141, 2024-08-20T22:08:10.8965010Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:08:10.8965679Z }, 2024-08-20T22:08:10.8966089Z { 2024-08-20T22:08:10.8966608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8967330Z "size": 135, 2024-08-20T22:08:10.8968049Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:08:10.8968746Z }, 2024-08-20T22:08:10.8969137Z { 2024-08-20T22:08:10.8969681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8970336Z "size": 4838068375, 2024-08-20T22:08:10.8971007Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:08:10.8971757Z }, 2024-08-20T22:08:10.8972065Z { 2024-08-20T22:08:10.8972606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8973298Z "size": 194, 2024-08-20T22:08:10.8973951Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:08:10.8974623Z }, 2024-08-20T22:08:10.8975028Z { 2024-08-20T22:08:10.8975579Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8976220Z "size": 564, 2024-08-20T22:08:10.8976937Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:08:10.8977814Z }, 2024-08-20T22:08:10.8978165Z { 2024-08-20T22:08:10.8978734Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8979389Z "size": 43162921, 2024-08-20T22:08:10.8980036Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:08:10.8980819Z }, 2024-08-20T22:08:10.8981123Z { 2024-08-20T22:08:10.8981653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8982371Z "size": 106, 2024-08-20T22:08:10.8982990Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:08:10.8983719Z }, 2024-08-20T22:08:10.8984213Z { 2024-08-20T22:08:10.8984742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8985373Z "size": 1380, 2024-08-20T22:08:10.8986050Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:08:10.8986751Z }, 2024-08-20T22:08:10.8987157Z { 2024-08-20T22:08:10.8987696Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8988338Z "size": 701, 2024-08-20T22:08:10.8989342Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:10.8990091Z }, 2024-08-20T22:08:10.8990422Z { 2024-08-20T22:08:10.8991001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8991655Z "size": 139, 2024-08-20T22:08:10.8992265Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:08:10.8993019Z }, 2024-08-20T22:08:10.8993368Z { 2024-08-20T22:08:10.8993919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8994573Z "size": 118, 2024-08-20T22:08:10.8995214Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:08:10.8995936Z }, 2024-08-20T22:08:10.8996355Z { 2024-08-20T22:08:10.8996902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.8997595Z "size": 4324672728, 2024-08-20T22:08:10.8998311Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:08:10.8999046Z }, 2024-08-20T22:08:10.8999359Z { 2024-08-20T22:08:10.8999943Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9000612Z "size": 174, 2024-08-20T22:08:10.9001193Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:08:10.9001936Z }, 2024-08-20T22:08:10.9002304Z { 2024-08-20T22:08:10.9002830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9003510Z "size": 907, 2024-08-20T22:08:10.9004184Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:08:10.9005050Z }, 2024-08-20T22:08:10.9005476Z { 2024-08-20T22:08:10.9005963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9006635Z "size": 701, 2024-08-20T22:08:10.9007314Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:10.9007983Z }, 2024-08-20T22:08:10.9008347Z { 2024-08-20T22:08:10.9008927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9009571Z "size": 134, 2024-08-20T22:08:10.9010201Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:08:10.9011024Z }, 2024-08-20T22:08:10.9011377Z { 2024-08-20T22:08:10.9011971Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9012579Z "size": 32, 2024-08-20T22:08:10.9013232Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:10.9013979Z }, 2024-08-20T22:08:10.9014297Z { 2024-08-20T22:08:10.9014852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9015674Z "size": 158, 2024-08-20T22:08:10.9016344Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:08:10.9017015Z }, 2024-08-20T22:08:10.9017444Z { 2024-08-20T22:08:10.9018044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9018692Z "size": 1841, 2024-08-20T22:08:10.9019332Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:08:10.9020053Z }, 2024-08-20T22:08:10.9020392Z { 2024-08-20T22:08:10.9020967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9021594Z "size": 7529776, 2024-08-20T22:08:10.9022236Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:08:10.9023008Z }, 2024-08-20T22:08:10.9023317Z { 2024-08-20T22:08:10.9023977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9024690Z "size": 163, 2024-08-20T22:08:10.9025322Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:08:10.9025981Z }, 2024-08-20T22:08:10.9026409Z { 2024-08-20T22:08:10.9026938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9027590Z "size": 7943, 2024-08-20T22:08:10.9028305Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:08:10.9029017Z }, 2024-08-20T22:08:10.9029384Z { 2024-08-20T22:08:10.9029929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9030579Z "size": 8068, 2024-08-20T22:08:10.9031270Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:08:10.9032036Z }, 2024-08-20T22:08:10.9032345Z { 2024-08-20T22:08:10.9032898Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9033590Z "size": 302, 2024-08-20T22:08:10.9034213Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:08:10.9034893Z }, 2024-08-20T22:08:10.9035290Z { 2024-08-20T22:08:10.9035843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9036539Z "size": 7629638, 2024-08-20T22:08:10.9037142Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:08:10.9037904Z }, 2024-08-20T22:08:10.9038302Z { 2024-08-20T22:08:10.9038793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9039727Z "size": 108, 2024-08-20T22:08:10.9040456Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:08:10.9041198Z }, 2024-08-20T22:08:10.9041548Z { 2024-08-20T22:08:10.9042192Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9043029Z "size": 54145661, 2024-08-20T22:08:10.9043772Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:08:10.9044552Z }, 2024-08-20T22:08:10.9044971Z { 2024-08-20T22:08:10.9045534Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9046248Z "size": 473, 2024-08-20T22:08:10.9046919Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:08:10.9047624Z }, 2024-08-20T22:08:10.9048068Z { 2024-08-20T22:08:10.9048599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9049249Z "size": 1374858766, 2024-08-20T22:08:10.9049953Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:08:10.9050659Z }, 2024-08-20T22:08:10.9051014Z { 2024-08-20T22:08:10.9051632Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9052282Z "size": 106, 2024-08-20T22:08:10.9052939Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:08:10.9053826Z }, 2024-08-20T22:08:10.9054133Z { 2024-08-20T22:08:10.9054686Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9055379Z "size": 559, 2024-08-20T22:08:10.9056000Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:08:10.9056678Z }, 2024-08-20T22:08:10.9057079Z { 2024-08-20T22:08:10.9057682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9058342Z "size": 46248488, 2024-08-20T22:08:10.9059041Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:08:10.9059774Z }, 2024-08-20T22:08:10.9060119Z { 2024-08-20T22:08:10.9060651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9061315Z "size": 111, 2024-08-20T22:08:10.9061955Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:08:10.9062704Z }, 2024-08-20T22:08:10.9063104Z { 2024-08-20T22:08:10.9063665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9064454Z "size": 32, 2024-08-20T22:08:10.9065094Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:10.9065825Z }, 2024-08-20T22:08:10.9066239Z { 2024-08-20T22:08:10.9066769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9067487Z "size": 32, 2024-08-20T22:08:10.9068094Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:10.9068811Z }, 2024-08-20T22:08:10.9069234Z { 2024-08-20T22:08:10.9069725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9070367Z "size": 32, 2024-08-20T22:08:10.9071101Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:10.9071832Z }, 2024-08-20T22:08:10.9072150Z { 2024-08-20T22:08:10.9072794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:10.9073450Z "size": 32, 2024-08-20T22:08:10.9074118Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:10.9074841Z } 2024-08-20T22:08:10.9075191Z ] 2024-08-20T22:08:10.9075568Z } 2024-08-20T22:08:10.9381337Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:10.9381824Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:10.9382366Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:08:10.9392351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.9392849Z env: 2024-08-20T22:08:10.9393130Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.9394276Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.9395440Z ##[endgroup] 2024-08-20T22:08:10.9429946Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.9480566Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:08:10.9481136Z with: 2024-08-20T22:08:10.9482203Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.9483525Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.9484039Z env: 2024-08-20T22:08:10.9484316Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.9484662Z ##[endgroup] 2024-08-20T22:08:10.9502296Z ##[group]Run set -x 2024-08-20T22:08:10.9502631Z set -x 2024-08-20T22:08:10.9502933Z set +e 2024-08-20T22:08:10.9503228Z  2024-08-20T22:08:10.9503503Z login() { 2024-08-20T22:08:10.9504246Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:10.9505143Z } 2024-08-20T22:08:10.9505426Z  2024-08-20T22:08:10.9505730Z retry () { 2024-08-20T22:08:10.9506129Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:10.9506577Z } 2024-08-20T22:08:10.9506854Z  2024-08-20T22:08:10.9507191Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:10.9507633Z  2024-08-20T22:08:10.9507907Z set -e 2024-08-20T22:08:10.9508382Z # ignore output since only exit code is used for conditional 2024-08-20T22:08:10.9509082Z # only pull docker image if it's not available locally 2024-08-20T22:08:10.9509843Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:08:10.9510535Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:08:10.9510976Z fi 2024-08-20T22:08:10.9518966Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.9519471Z env: 2024-08-20T22:08:10.9519753Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.9520895Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:10.9522191Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.9522708Z ##[endgroup] 2024-08-20T22:08:10.9550186Z + set +e 2024-08-20T22:08:10.9550936Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.9551594Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:10.9554544Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:10.9555337Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:11.4947411Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:11.4948374Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:11.4949240Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:11.4949767Z 2024-08-20T22:08:11.4949893Z Login Succeeded 2024-08-20T22:08:11.4971946Z + set -e 2024-08-20T22:08:11.4973260Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.5130494Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.5132636Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:11.7600559Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:08:11.7603338Z 7a2c55901189: Pulling fs layer 2024-08-20T22:08:11.7603797Z 224fe954d725: Pulling fs layer 2024-08-20T22:08:11.7604807Z 75722010b82e: Pulling fs layer 2024-08-20T22:08:11.7605475Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:08:11.7605985Z b57676e46aee: Pulling fs layer 2024-08-20T22:08:11.7606518Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:08:11.7607078Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:08:11.7607609Z 0c1227890755: Pulling fs layer 2024-08-20T22:08:11.7608133Z d8d1234baab3: Pulling fs layer 2024-08-20T22:08:11.7608673Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:08:11.7610687Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:08:11.7611187Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:08:11.7611903Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:08:11.7612476Z 4373aa743caf: Pulling fs layer 2024-08-20T22:08:11.7612968Z d527cbbb87e3: Waiting 2024-08-20T22:08:11.7613420Z a41a8d1c11c8: Waiting 2024-08-20T22:08:11.7613872Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:08:11.7614585Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:08:11.7614957Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:08:11.7615346Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:08:11.7615711Z d8d1234baab3: Waiting 2024-08-20T22:08:11.7616048Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:08:11.7616528Z 7ed32bc8e469: Waiting 2024-08-20T22:08:11.7617014Z a8c1e85b5e14: Waiting 2024-08-20T22:08:11.7617468Z ca426ffc9d2b: Waiting 2024-08-20T22:08:11.7617927Z 4373aa743caf: Waiting 2024-08-20T22:08:11.7618384Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:08:11.7618908Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:08:11.7619432Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:08:11.7619928Z ac23c5fcc1ef: Waiting 2024-08-20T22:08:11.7620388Z 33b639226eee: Pulling fs layer 2024-08-20T22:08:11.7620899Z 34c47d63c5de: Waiting 2024-08-20T22:08:11.7621341Z 0bf61114d37b: Waiting 2024-08-20T22:08:11.7621779Z d3547bd43146: Pulling fs layer 2024-08-20T22:08:11.7622323Z 5645520256fa: Pulling fs layer 2024-08-20T22:08:11.7622805Z 75b2abb16c68: Waiting 2024-08-20T22:08:11.7623242Z d3547bd43146: Waiting 2024-08-20T22:08:11.7623661Z ec1e7978c1fe: Waiting 2024-08-20T22:08:11.7624225Z ef1e3ef1e85f: Waiting 2024-08-20T22:08:11.7624697Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:08:11.7625223Z 385d0421d67c: Pulling fs layer 2024-08-20T22:08:11.7625739Z 1fef4bd19792: Waiting 2024-08-20T22:08:11.7626178Z 0c1227890755: Waiting 2024-08-20T22:08:11.7626602Z 5645520256fa: Waiting 2024-08-20T22:08:11.7627036Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:08:11.7627608Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:08:11.7628125Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:08:11.7628627Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:08:11.7629109Z 7cbfce999d76: Waiting 2024-08-20T22:08:11.7629568Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:08:11.7630069Z 7754ee068b33: Pulling fs layer 2024-08-20T22:08:11.7630527Z 6658e558ebe4: Waiting 2024-08-20T22:08:11.7630958Z 0479fbac3361: Pulling fs layer 2024-08-20T22:08:11.7631481Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:08:11.7631983Z afc429e87b59: Pulling fs layer 2024-08-20T22:08:11.7632462Z 019620471c25: Pulling fs layer 2024-08-20T22:08:11.7632972Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:08:11.7633476Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:08:11.7633953Z 829ed7d91e38: Waiting 2024-08-20T22:08:11.7634361Z f1d6874dee9c: Waiting 2024-08-20T22:08:11.7634790Z 560cd6122a38: Pulling fs layer 2024-08-20T22:08:11.7635266Z 385d0421d67c: Waiting 2024-08-20T22:08:11.7635685Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:08:11.7636150Z 560cd6122a38: Waiting 2024-08-20T22:08:11.7636595Z da6a838a66c7: Pulling fs layer 2024-08-20T22:08:11.7636990Z 0479fbac3361: Waiting 2024-08-20T22:08:11.7638653Z cbb79a705353: Pulling fs layer 2024-08-20T22:08:11.7639161Z 1811ff4d3556: Waiting 2024-08-20T22:08:11.7641689Z da6a838a66c7: Waiting 2024-08-20T22:08:11.7642163Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:08:11.7642640Z 829c0e19f4b7: Waiting 2024-08-20T22:08:11.7643078Z cbb79a705353: Waiting 2024-08-20T22:08:11.7644849Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:08:11.7645345Z 7754ee068b33: Waiting 2024-08-20T22:08:11.7646076Z 4a0412373305: Pulling fs layer 2024-08-20T22:08:11.7646564Z 4d4fa6802ab3: Waiting 2024-08-20T22:08:11.7646975Z 0f085d4fa279: Waiting 2024-08-20T22:08:11.7647431Z 4a0412373305: Waiting 2024-08-20T22:08:11.7647875Z 302c8c177944: Pulling fs layer 2024-08-20T22:08:11.7648363Z 5a4eb19d7219: Waiting 2024-08-20T22:08:11.7648803Z 02970347dff6: Pulling fs layer 2024-08-20T22:08:11.7649300Z 302c8c177944: Waiting 2024-08-20T22:08:11.7649747Z 916c5e858016: Pulling fs layer 2024-08-20T22:08:11.7650192Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:08:11.7650568Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:08:11.7650921Z 02970347dff6: Waiting 2024-08-20T22:08:11.7651234Z 916c5e858016: Waiting 2024-08-20T22:08:11.7651544Z c42b3602a1fb: Waiting 2024-08-20T22:08:11.7651949Z f957c1624481: Pulling fs layer 2024-08-20T22:08:11.7652300Z 11fad972ef6c: Waiting 2024-08-20T22:08:11.7652627Z 768c31dca016: Pulling fs layer 2024-08-20T22:08:11.7653131Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:08:11.7653485Z f957c1624481: Waiting 2024-08-20T22:08:11.7653810Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:08:11.7654182Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:08:11.7654553Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:08:11.7654917Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:08:11.7655286Z 8c32d0ecebbe: Waiting 2024-08-20T22:08:11.7655602Z 4f4fb700ef54: Waiting 2024-08-20T22:08:11.7655930Z b10de02255c0: Pulling fs layer 2024-08-20T22:08:11.7656299Z e56eeff2db93: Pulling fs layer 2024-08-20T22:08:11.7656651Z 37513e9b43d7: Waiting 2024-08-20T22:08:11.7657000Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:08:11.7657381Z a7a5392e2bf6: Waiting 2024-08-20T22:08:11.7657713Z b10de02255c0: Waiting 2024-08-20T22:08:11.7658043Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:08:11.7658419Z ee1273153675: Pulling fs layer 2024-08-20T22:08:11.7658790Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:08:11.7659159Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:08:11.7659529Z ee1273153675: Waiting 2024-08-20T22:08:11.7659844Z e22979f9e0ee: Waiting 2024-08-20T22:08:11.7660171Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:08:11.7660544Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:08:11.7660902Z 7ebe49c3a23e: Waiting 2024-08-20T22:08:11.7661231Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:08:11.7661581Z 5d115b29b5d5: Waiting 2024-08-20T22:08:11.7661909Z d79fd7d52698: Pulling fs layer 2024-08-20T22:08:11.7662285Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:08:11.7662648Z d79fd7d52698: Waiting 2024-08-20T22:08:11.7662970Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:08:11.7663325Z 7f2d7a04fbe6: Waiting 2024-08-20T22:08:11.7663642Z 9752d6fbbcab: Waiting 2024-08-20T22:08:11.9047460Z 224fe954d725: Verifying Checksum 2024-08-20T22:08:11.9047932Z 224fe954d725: Download complete 2024-08-20T22:08:11.9756554Z d527cbbb87e3: Download complete 2024-08-20T22:08:12.0567649Z b57676e46aee: Verifying Checksum 2024-08-20T22:08:12.0568261Z b57676e46aee: Download complete 2024-08-20T22:08:12.0937844Z 7a2c55901189: Verifying Checksum 2024-08-20T22:08:12.0938259Z 7a2c55901189: Download complete 2024-08-20T22:08:12.1554649Z a41a8d1c11c8: Download complete 2024-08-20T22:08:12.2362323Z 0c1227890755: Download complete 2024-08-20T22:08:12.2996405Z d8d1234baab3: Download complete 2024-08-20T22:08:12.3866715Z 75722010b82e: Verifying Checksum 2024-08-20T22:08:12.3867181Z 75722010b82e: Download complete 2024-08-20T22:08:12.4596964Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:08:12.4597443Z ec1e7978c1fe: Download complete 2024-08-20T22:08:12.5434140Z ac23c5fcc1ef: Download complete 2024-08-20T22:08:13.2597542Z 7a2c55901189: Pull complete 2024-08-20T22:08:13.5674188Z 224fe954d725: Pull complete 2024-08-20T22:08:14.4876415Z 75722010b82e: Pull complete 2024-08-20T22:08:14.5053451Z d527cbbb87e3: Pull complete 2024-08-20T22:08:14.5219643Z b57676e46aee: Pull complete 2024-08-20T22:08:15.0654488Z ca426ffc9d2b: Download complete 2024-08-20T22:08:15.1564157Z 4373aa743caf: Verifying Checksum 2024-08-20T22:08:15.1564672Z 4373aa743caf: Download complete 2024-08-20T22:08:15.2579943Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:08:15.2581533Z 75b2abb16c68: Download complete 2024-08-20T22:08:15.3350825Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:08:15.3351295Z ef1e3ef1e85f: Download complete 2024-08-20T22:08:16.3241394Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:08:16.3242019Z 0bf61114d37b: Download complete 2024-08-20T22:08:16.3945229Z 2df849f72ba4: Download complete 2024-08-20T22:08:16.4570503Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:08:16.4571136Z 34c47d63c5de: Download complete 2024-08-20T22:08:16.5371343Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:08:16.5371830Z 1fef4bd19792: Download complete 2024-08-20T22:08:25.3946322Z a8c1e85b5e14: Download complete 2024-08-20T22:08:25.4621615Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:08:25.4622067Z b119c4fc6c58: Download complete 2024-08-20T22:08:25.5447956Z 33b639226eee: Verifying Checksum 2024-08-20T22:08:25.5448596Z 33b639226eee: Download complete 2024-08-20T22:08:25.6485861Z d3547bd43146: Verifying Checksum 2024-08-20T22:08:25.6486467Z d3547bd43146: Download complete 2024-08-20T22:08:25.7214991Z 5645520256fa: Verifying Checksum 2024-08-20T22:08:25.7215457Z 5645520256fa: Download complete 2024-08-20T22:08:25.8285101Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:08:25.8285744Z 829ed7d91e38: Download complete 2024-08-20T22:08:27.1049699Z 385d0421d67c: Verifying Checksum 2024-08-20T22:08:27.1050277Z 385d0421d67c: Download complete 2024-08-20T22:08:27.1868663Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:08:27.1869275Z 7cbfce999d76: Download complete 2024-08-20T22:08:27.2509410Z 6658e558ebe4: Download complete 2024-08-20T22:08:27.3284081Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:08:27.3284622Z 33444a5a7e01: Download complete 2024-08-20T22:08:27.3992546Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:08:27.3992995Z 7c47122f4e09: Download complete 2024-08-20T22:08:27.4694073Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:08:27.4694683Z 1811ff4d3556: Download complete 2024-08-20T22:08:33.4576300Z 7754ee068b33: Verifying Checksum 2024-08-20T22:08:33.4576953Z 7754ee068b33: Download complete 2024-08-20T22:08:33.5383724Z 0479fbac3361: Verifying Checksum 2024-08-20T22:08:33.5384383Z 0479fbac3361: Download complete 2024-08-20T22:08:33.6108916Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:08:33.6111342Z f1d6874dee9c: Download complete 2024-08-20T22:08:34.0189489Z afc429e87b59: Verifying Checksum 2024-08-20T22:08:34.0190102Z afc429e87b59: Download complete 2024-08-20T22:08:34.0955287Z 019620471c25: Download complete 2024-08-20T22:08:34.1588172Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:08:34.1591804Z 27aeb1b2cd5e: Download complete 2024-08-20T22:08:34.4131298Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:08:34.4131933Z 829c0e19f4b7: Download complete 2024-08-20T22:08:34.4876322Z 560cd6122a38: Download complete 2024-08-20T22:08:34.5648748Z 5a4eb19d7219: Download complete 2024-08-20T22:08:34.6503298Z da6a838a66c7: Verifying Checksum 2024-08-20T22:08:34.6503911Z da6a838a66c7: Download complete 2024-08-20T22:08:37.6301325Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:08:37.6302002Z 7ed32bc8e469: Download complete 2024-08-20T22:08:37.7046597Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:08:37.7047261Z 4d4fa6802ab3: Download complete 2024-08-20T22:08:37.7864856Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:08:37.7865513Z 0f085d4fa279: Download complete 2024-08-20T22:08:38.2793270Z 4a0412373305: Verifying Checksum 2024-08-20T22:08:38.2793892Z 4a0412373305: Download complete 2024-08-20T22:08:38.3541476Z 302c8c177944: Verifying Checksum 2024-08-20T22:08:38.3541933Z 302c8c177944: Download complete 2024-08-20T22:08:38.4178564Z 02970347dff6: Download complete 2024-08-20T22:08:38.5118595Z 916c5e858016: Verifying Checksum 2024-08-20T22:08:38.5119205Z 916c5e858016: Download complete 2024-08-20T22:08:38.5946375Z a8c1e85b5e14: Pull complete 2024-08-20T22:08:38.6105793Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:08:38.6106338Z c42b3602a1fb: Download complete 2024-08-20T22:08:38.7707898Z a41a8d1c11c8: Pull complete 2024-08-20T22:08:38.8448526Z 0c1227890755: Pull complete 2024-08-20T22:08:38.9153011Z d8d1234baab3: Pull complete 2024-08-20T22:08:45.2309666Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:08:45.2310321Z 6efb96fc87ea: Download complete 2024-08-20T22:08:45.2936611Z f957c1624481: Verifying Checksum 2024-08-20T22:08:45.2937210Z f957c1624481: Download complete 2024-08-20T22:08:45.3585760Z 768c31dca016: Download complete 2024-08-20T22:08:45.4335780Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:08:45.4336328Z 8c32d0ecebbe: Download complete 2024-08-20T22:08:45.4416072Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:08:45.4416685Z 4f4fb700ef54: Download complete 2024-08-20T22:08:45.5119764Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:08:45.5120418Z 78ee77eddef2: Download complete 2024-08-20T22:08:45.6059149Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:08:45.6059747Z 37513e9b43d7: Download complete 2024-08-20T22:08:45.7668970Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:08:45.7669778Z a7a5392e2bf6: Download complete 2024-08-20T22:08:45.8406414Z b10de02255c0: Verifying Checksum 2024-08-20T22:08:45.8407039Z b10de02255c0: Download complete 2024-08-20T22:08:45.9202593Z e56eeff2db93: Download complete 2024-08-20T22:08:46.0046141Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:08:46.0046825Z 70ac3e38d0fd: Download complete 2024-08-20T22:08:46.0753434Z eb09cbf3e416: Download complete 2024-08-20T22:08:46.1971527Z ee1273153675: Verifying Checksum 2024-08-20T22:08:46.1972089Z ee1273153675: Download complete 2024-08-20T22:08:46.2727512Z e22979f9e0ee: Download complete 2024-08-20T22:08:46.8654290Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:08:46.8654938Z 6c1f0aabce7c: Download complete 2024-08-20T22:08:46.9362990Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:08:46.9363463Z 7ebe49c3a23e: Download complete 2024-08-20T22:09:14.0044182Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:09:14.0044653Z 5d115b29b5d5: Download complete 2024-08-20T22:09:14.0887796Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:09:14.0889170Z 7f2d7a04fbe6: Download complete 2024-08-20T22:09:14.1674138Z d79fd7d52698: Download complete 2024-08-20T22:09:15.2921173Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:09:15.2921805Z 9752d6fbbcab: Download complete 2024-08-20T22:09:15.3626211Z 83808b2eff4d: Download complete 2024-08-20T22:09:53.2745262Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:09:53.2745736Z 11fad972ef6c: Download complete 2024-08-20T22:09:54.4420344Z cbb79a705353: Verifying Checksum 2024-08-20T22:09:54.4420940Z cbb79a705353: Download complete 2024-08-20T22:10:09.5770638Z 7ed32bc8e469: Pull complete 2024-08-20T22:10:09.7666874Z ec1e7978c1fe: Pull complete 2024-08-20T22:10:09.9782022Z ac23c5fcc1ef: Pull complete 2024-08-20T22:10:18.6650507Z ca426ffc9d2b: Pull complete 2024-08-20T22:10:18.8561488Z 4373aa743caf: Pull complete 2024-08-20T22:10:19.0835970Z 75b2abb16c68: Pull complete 2024-08-20T22:10:19.2849377Z ef1e3ef1e85f: Pull complete 2024-08-20T22:10:21.7574935Z 0bf61114d37b: Pull complete 2024-08-20T22:10:21.8406120Z 2df849f72ba4: Pull complete 2024-08-20T22:10:21.9710692Z 34c47d63c5de: Pull complete 2024-08-20T22:10:22.1109448Z 1fef4bd19792: Pull complete 2024-08-20T22:11:20.6994988Z 6efb96fc87ea: Pull complete 2024-08-20T22:11:20.8221387Z b119c4fc6c58: Pull complete 2024-08-20T22:11:20.8971386Z 33b639226eee: Pull complete 2024-08-20T22:11:20.9909719Z d3547bd43146: Pull complete 2024-08-20T22:11:21.1066296Z 5645520256fa: Pull complete 2024-08-20T22:11:21.2267125Z 829ed7d91e38: Pull complete 2024-08-20T22:11:24.6019156Z 385d0421d67c: Pull complete 2024-08-20T22:11:24.6894819Z 7cbfce999d76: Pull complete 2024-08-20T22:11:24.7609380Z 6658e558ebe4: Pull complete 2024-08-20T22:11:24.8521008Z 33444a5a7e01: Pull complete 2024-08-20T22:11:24.9795427Z 7c47122f4e09: Pull complete 2024-08-20T22:11:25.1255754Z 1811ff4d3556: Pull complete 2024-08-20T22:11:35.6850214Z 7754ee068b33: Pull complete 2024-08-20T22:11:35.8905829Z 0479fbac3361: Pull complete 2024-08-20T22:11:36.0447753Z f1d6874dee9c: Pull complete 2024-08-20T22:11:37.0834306Z afc429e87b59: Pull complete 2024-08-20T22:11:37.2795966Z 019620471c25: Pull complete 2024-08-20T22:11:37.4414708Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:11:37.8987168Z 829c0e19f4b7: Pull complete 2024-08-20T22:11:38.1066681Z 560cd6122a38: Pull complete 2024-08-20T22:11:38.4067814Z 5a4eb19d7219: Pull complete 2024-08-20T22:11:38.5486369Z da6a838a66c7: Pull complete 2024-08-20T22:12:17.3757603Z cbb79a705353: Pull complete 2024-08-20T22:12:17.5571583Z 4d4fa6802ab3: Pull complete 2024-08-20T22:12:17.7503221Z 0f085d4fa279: Pull complete 2024-08-20T22:12:20.1823391Z 4a0412373305: Pull complete 2024-08-20T22:12:20.2027371Z 302c8c177944: Pull complete 2024-08-20T22:12:20.2210598Z 02970347dff6: Pull complete 2024-08-20T22:12:20.2576333Z 916c5e858016: Pull complete 2024-08-20T22:12:20.2773770Z c42b3602a1fb: Pull complete 2024-08-20T22:13:42.1223737Z 11fad972ef6c: Pull complete 2024-08-20T22:13:42.2958472Z f957c1624481: Pull complete 2024-08-20T22:13:42.3793537Z 768c31dca016: Pull complete 2024-08-20T22:13:42.6814847Z 8c32d0ecebbe: Pull complete 2024-08-20T22:13:42.7697035Z 4f4fb700ef54: Pull complete 2024-08-20T22:13:42.8754761Z 78ee77eddef2: Pull complete 2024-08-20T22:13:43.0800242Z 37513e9b43d7: Pull complete 2024-08-20T22:13:43.4633427Z a7a5392e2bf6: Pull complete 2024-08-20T22:13:43.5399120Z b10de02255c0: Pull complete 2024-08-20T22:13:43.6999135Z e56eeff2db93: Pull complete 2024-08-20T22:13:43.8917568Z 70ac3e38d0fd: Pull complete 2024-08-20T22:13:44.0867613Z eb09cbf3e416: Pull complete 2024-08-20T22:13:45.3725648Z ee1273153675: Pull complete 2024-08-20T22:13:45.5326160Z e22979f9e0ee: Pull complete 2024-08-20T22:13:48.1819017Z 6c1f0aabce7c: Pull complete 2024-08-20T22:13:48.2711428Z 7ebe49c3a23e: Pull complete 2024-08-20T22:14:13.3966811Z 5d115b29b5d5: Pull complete 2024-08-20T22:14:13.5790687Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:14:13.7613490Z d79fd7d52698: Pull complete 2024-08-20T22:14:14.6555280Z 9752d6fbbcab: Pull complete 2024-08-20T22:14:14.8470939Z 83808b2eff4d: Pull complete 2024-08-20T22:14:15.4291362Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:14:15.4838064Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:15.5012702Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:15.5164318Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:15.5165300Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:15.5178297Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:14:15.5178805Z env: 2024-08-20T22:14:15.5179090Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:15.5179439Z ##[endgroup] 2024-08-20T22:14:15.5392882Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:14:15.5393436Z with: 2024-08-20T22:14:15.5393716Z driver-version: 550.54.15 2024-08-20T22:14:15.5394052Z env: 2024-08-20T22:14:15.5394332Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:15.5394679Z ##[endgroup] 2024-08-20T22:14:15.5469871Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:14:15.5470425Z with: 2024-08-20T22:14:15.5470702Z timeout_minutes: 10 2024-08-20T22:14:15.5471025Z max_attempts: 3 2024-08-20T22:14:15.5503555Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:14:15.5536352Z retry_wait_seconds: 10 2024-08-20T22:14:15.5536711Z polling_interval_seconds: 1 2024-08-20T22:14:15.5537072Z warning_on_retry: true 2024-08-20T22:14:15.5537417Z continue_on_error: false 2024-08-20T22:14:15.5537747Z env: 2024-08-20T22:14:15.5538074Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:15.5538419Z DRIVER_VERSION: 550.54.15 2024-08-20T22:14:15.5538767Z ##[endgroup] 2024-08-20T22:14:15.9819874Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:14:15.9820840Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:14:15.9821450Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:14:16.9730598Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:14:16.9731602Z No packages marked for removal. 2024-08-20T22:14:16.9867938Z Dependencies resolved. 2024-08-20T22:14:16.9877919Z Nothing to do. 2024-08-20T22:14:16.9879316Z Complete! 2024-08-20T22:14:17.0732884Z + install_nvidia_driver_common 2024-08-20T22:14:17.0737514Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:14:17.0738481Z Before installing NVIDIA driver 2024-08-20T22:14:17.0739116Z + lspci 2024-08-20T22:14:17.0848116Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:17.0848883Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:17.0850060Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:17.0851329Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:17.0852185Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:17.0852976Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:17.0853709Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:17.0854517Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:17.0855095Z + lsmod 2024-08-20T22:14:17.0896236Z Module Size Used by 2024-08-20T22:14:17.0896784Z ib_core 454656 0 2024-08-20T22:14:17.0897159Z veth 36864 0 2024-08-20T22:14:17.0897530Z nvidia_modeset 1351680 0 2024-08-20T22:14:17.0898080Z video 65536 1 nvidia_modeset 2024-08-20T22:14:17.0898749Z wmi 36864 1 video 2024-08-20T22:14:17.0899152Z nvidia_uvm 4706304 0 2024-08-20T22:14:17.0899597Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:17.0900080Z drm 602112 1 nvidia 2024-08-20T22:14:17.0900523Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:17.0901228Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:17.0901848Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:17.0902284Z xt_conntrack 16384 1 2024-08-20T22:14:17.0902666Z nft_chain_nat 16384 3 2024-08-20T22:14:17.0903084Z xt_MASQUERADE 20480 1 2024-08-20T22:14:17.0903739Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:17.0904460Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:17.0905303Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:17.0906090Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:17.0906556Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:17.0907028Z xfrm_user 57344 1 2024-08-20T22:14:17.0907536Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:17.0908109Z xt_addrtype 16384 2 2024-08-20T22:14:17.0908620Z nft_compat 20480 4 2024-08-20T22:14:17.0909231Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:17.0910055Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:17.0910796Z br_netfilter 36864 0 2024-08-20T22:14:17.0911332Z bridge 307200 1 br_netfilter 2024-08-20T22:14:17.0911878Z stp 16384 1 bridge 2024-08-20T22:14:17.0912577Z llc 16384 2 bridge,stp 2024-08-20T22:14:17.0912994Z overlay 167936 0 2024-08-20T22:14:17.0913367Z tls 114688 0 2024-08-20T22:14:17.0913737Z nls_ascii 16384 1 2024-08-20T22:14:17.0914107Z nls_cp437 20480 1 2024-08-20T22:14:17.0914565Z vfat 24576 1 2024-08-20T22:14:17.0915120Z fat 86016 1 vfat 2024-08-20T22:14:17.0915720Z sunrpc 692224 1 2024-08-20T22:14:17.0916281Z ena 167936 0 2024-08-20T22:14:17.0916724Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:17.0917175Z ptp 36864 1 ena 2024-08-20T22:14:17.0917739Z pps_core 24576 1 ptp 2024-08-20T22:14:17.0918341Z aesni_intel 393216 0 2024-08-20T22:14:17.0918736Z i8042 45056 0 2024-08-20T22:14:17.0919111Z serio 28672 3 i8042 2024-08-20T22:14:17.0919542Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:17.0920091Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:17.0920586Z button 24576 0 2024-08-20T22:14:17.0920963Z sch_fq_codel 20480 17 2024-08-20T22:14:17.0921342Z dm_mod 188416 0 2024-08-20T22:14:17.0921858Z fuse 163840 1 2024-08-20T22:14:17.0922402Z configfs 57344 1 2024-08-20T22:14:17.0922985Z dax 45056 1 dm_mod 2024-08-20T22:14:17.0923424Z loop 36864 0 2024-08-20T22:14:17.0923797Z dmi_sysfs 20480 0 2024-08-20T22:14:17.0924164Z crc32_pclmul 16384 0 2024-08-20T22:14:17.0924542Z crc32c_intel 24576 0 2024-08-20T22:14:17.0925234Z efivarfs 24576 1 2024-08-20T22:14:17.0925842Z + modinfo nvidia 2024-08-20T22:14:17.0926814Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:17.0927544Z alias: char-major-195-* 2024-08-20T22:14:17.0928076Z version: 550.54.15 2024-08-20T22:14:17.0928434Z supported: external 2024-08-20T22:14:17.0928782Z license: NVIDIA 2024-08-20T22:14:17.0929166Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:17.0929770Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:17.0930418Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:17.0930890Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:17.0931389Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:17.0931869Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:17.0932377Z depends: i2c-core,drm 2024-08-20T22:14:17.0932738Z retpoline: Y 2024-08-20T22:14:17.0933048Z name: nvidia 2024-08-20T22:14:17.0933654Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:17.0934353Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:17.0935003Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:17.0935658Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:17.0936097Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:17.0936530Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:17.0936986Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:17.0937418Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:17.0937847Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:17.0938373Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:17.0938928Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:17.0939404Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:17.0939827Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:17.0940242Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:17.0940696Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:17.0941222Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:17.0941801Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:17.0942482Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:17.0943089Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:17.0943687Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:17.0944383Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:17.0944973Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:17.0945462Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:17.0945994Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:17.0946531Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:17.0947025Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:17.0947487Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:17.0947963Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:17.0948437Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:17.0948887Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:17.0949395Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:17.0949910Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:17.0950387Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:17.0950873Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:17.0951375Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:17.0951852Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:17.0952351Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:17.0952828Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:17.0953238Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:17.0953698Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:17.0954168Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:17.0954626Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:17.0955101Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:17.0955722Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:17.0956189Z parm: rm_firmware_active:charp 2024-08-20T22:14:17.0956613Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:14:17.0956997Z ++ command -v nvidia-smi 2024-08-20T22:14:17.0957405Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:14:17.0957770Z + set +e 2024-08-20T22:14:17.0958278Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:14:17.1205159Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:14:17.1205600Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:17.1206027Z + '[' 0 -ne 0 ']' 2024-08-20T22:14:17.1206394Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:14:17.1206856Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:14:17.1207870Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:14:17.1208601Z + set -e 2024-08-20T22:14:17.1208925Z + '[' 1 -eq 0 ']' 2024-08-20T22:14:17.1209496Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:14:17.1210202Z + post_install_nvidia_driver_common 2024-08-20T22:14:17.1211123Z + sudo modprobe nvidia 2024-08-20T22:14:17.2491665Z + echo 'After installing NVIDIA driver' 2024-08-20T22:14:17.2492216Z + lspci 2024-08-20T22:14:17.2492525Z After installing NVIDIA driver 2024-08-20T22:14:17.2599057Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:17.2599958Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:17.2600987Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:17.2601779Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:17.2602591Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:17.2603383Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:17.2604114Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:17.2605230Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:17.2605830Z + lsmod 2024-08-20T22:14:17.2635548Z Module Size Used by 2024-08-20T22:14:17.2635976Z ib_core 454656 0 2024-08-20T22:14:17.2636475Z veth 36864 0 2024-08-20T22:14:17.2636970Z nvidia_modeset 1351680 0 2024-08-20T22:14:17.2637580Z video 65536 1 nvidia_modeset 2024-08-20T22:14:17.2638163Z wmi 36864 1 video 2024-08-20T22:14:17.2638700Z nvidia_uvm 4706304 0 2024-08-20T22:14:17.2639154Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:17.2639642Z drm 602112 1 nvidia 2024-08-20T22:14:17.2640090Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:17.2640622Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:17.2641204Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:17.2641631Z xt_conntrack 16384 1 2024-08-20T22:14:17.2642024Z nft_chain_nat 16384 3 2024-08-20T22:14:17.2642400Z xt_MASQUERADE 20480 1 2024-08-20T22:14:17.2642846Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:17.2643336Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:17.2643909Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:17.2644541Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:17.2645027Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:17.2645479Z xfrm_user 57344 1 2024-08-20T22:14:17.2645874Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:17.2646287Z xt_addrtype 16384 2 2024-08-20T22:14:17.2646662Z nft_compat 20480 4 2024-08-20T22:14:17.2647115Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:17.2647723Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:17.2648474Z br_netfilter 36864 0 2024-08-20T22:14:17.2648903Z bridge 307200 1 br_netfilter 2024-08-20T22:14:17.2649348Z stp 16384 1 bridge 2024-08-20T22:14:17.2649770Z llc 16384 2 bridge,stp 2024-08-20T22:14:17.2650185Z overlay 167936 0 2024-08-20T22:14:17.2650552Z tls 114688 0 2024-08-20T22:14:17.2650919Z nls_ascii 16384 1 2024-08-20T22:14:17.2651283Z nls_cp437 20480 1 2024-08-20T22:14:17.2651647Z vfat 24576 1 2024-08-20T22:14:17.2652024Z fat 86016 1 vfat 2024-08-20T22:14:17.2652414Z sunrpc 692224 1 2024-08-20T22:14:17.2652772Z ena 167936 0 2024-08-20T22:14:17.2653146Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:17.2653530Z ptp 36864 1 ena 2024-08-20T22:14:17.2653931Z pps_core 24576 1 ptp 2024-08-20T22:14:17.2654320Z aesni_intel 393216 0 2024-08-20T22:14:17.2654684Z i8042 45056 0 2024-08-20T22:14:17.2655218Z serio 28672 3 i8042 2024-08-20T22:14:17.2655717Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:17.2656269Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:17.2656797Z button 24576 0 2024-08-20T22:14:17.2657240Z sch_fq_codel 20480 17 2024-08-20T22:14:17.2657702Z dm_mod 188416 0 2024-08-20T22:14:17.2658149Z fuse 163840 1 2024-08-20T22:14:17.2658533Z configfs 57344 1 2024-08-20T22:14:17.2658965Z dax 45056 1 dm_mod 2024-08-20T22:14:17.2659366Z loop 36864 0 2024-08-20T22:14:17.2659811Z dmi_sysfs 20480 0 2024-08-20T22:14:17.2660185Z crc32_pclmul 16384 0 2024-08-20T22:14:17.2660557Z crc32c_intel 24576 0 2024-08-20T22:14:17.2660928Z efivarfs 24576 1 2024-08-20T22:14:17.2661287Z + modinfo nvidia 2024-08-20T22:14:17.2661967Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:17.2662835Z alias: char-major-195-* 2024-08-20T22:14:17.2663224Z version: 550.54.15 2024-08-20T22:14:17.2663564Z supported: external 2024-08-20T22:14:17.2663993Z license: NVIDIA 2024-08-20T22:14:17.2664372Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:17.2664851Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:17.2665362Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:17.2665827Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:17.2666327Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:17.2666810Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:17.2667305Z depends: i2c-core,drm 2024-08-20T22:14:17.2667663Z retpoline: Y 2024-08-20T22:14:17.2667975Z name: nvidia 2024-08-20T22:14:17.2668575Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:17.2669288Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:17.2669933Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:17.2670525Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:17.2670964Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:17.2671401Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:17.2671859Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:17.2672293Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:17.2672722Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:17.2673249Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:17.2673806Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:17.2674281Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:17.2674714Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:17.2675155Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:17.2676710Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:17.2677255Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:17.2677838Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:17.2678395Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:17.2679045Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:17.2679700Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:17.2680307Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:17.2680895Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:17.2681387Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:17.2681912Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:17.2682459Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:17.2682957Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:17.2683422Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:17.2683903Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:17.2684372Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:17.2684819Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:17.2685320Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:17.2685835Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:17.2686310Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:17.2686803Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:17.2687291Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:17.2687776Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:17.2688275Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:17.2689042Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:17.2689453Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:17.2689922Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:17.2690392Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:17.2691000Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:17.2691472Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:17.2691944Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:17.2692401Z parm: rm_firmware_active:charp 2024-08-20T22:14:17.2692812Z + set +e 2024-08-20T22:14:17.2693138Z + nvidia-smi 2024-08-20T22:14:17.2866356Z Tue Aug 20 22:14:17 2024 2024-08-20T22:14:17.2866998Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.2867932Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:14:17.2868735Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.2869536Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:14:17.2870428Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:14:17.2871138Z | | | MIG M. | 2024-08-20T22:14:17.2871683Z |=========================================+========================+======================| 2024-08-20T22:14:17.3155169Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:14:17.3155999Z | 0% 26C P8 22W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:14:17.3156663Z | | | N/A | 2024-08-20T22:14:17.3157532Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.3158841Z 2024-08-20T22:14:17.3159513Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.3160523Z | Processes: | 2024-08-20T22:14:17.3161263Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:14:17.3161953Z | ID ID Usage | 2024-08-20T22:14:17.3162514Z |=========================================================================================| 2024-08-20T22:14:17.3243064Z | No running processes found | 2024-08-20T22:14:17.3243952Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.5889515Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:14:17.6079411Z NVIDIA A10G 2024-08-20T22:14:17.6143593Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:17.6144105Z + '[' 0 -eq 0 ']' 2024-08-20T22:14:17.6144627Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:14:17.6145091Z + set -e 2024-08-20T22:14:17.6145412Z INFO: Ignoring allowed status 0 2024-08-20T22:14:17.6154404Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:14:17.6158844Z + sudo yum install -y yum-utils 2024-08-20T22:14:18.1655816Z Last metadata expiration check: 0:33:58 ago on Tue Aug 20 21:40:20 2024. 2024-08-20T22:14:18.1875736Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:14:18.2179244Z Dependencies resolved. 2024-08-20T22:14:18.2302086Z Nothing to do. 2024-08-20T22:14:18.2302457Z Complete! 2024-08-20T22:14:18.3253305Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:14:18.3254681Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:18.3256114Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:18.5570302Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:18.6298727Z + sudo yum install -y nvidia-docker2 2024-08-20T22:14:19.0935851Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:14:19.1156904Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:14:19.1453483Z Dependencies resolved. 2024-08-20T22:14:19.1577873Z Nothing to do. 2024-08-20T22:14:19.1578253Z Complete! 2024-08-20T22:14:19.2505847Z + sudo systemctl restart docker 2024-08-20T22:15:10.8164651Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:15:10.8407434Z Tue Aug 20 22:15:10 2024 2024-08-20T22:15:10.8408148Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:10.8410329Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:10.8411441Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:10.8412320Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:10.8413235Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:10.8413926Z | | | MIG M. | 2024-08-20T22:15:10.8414480Z |=========================================+========================+======================| 2024-08-20T22:15:10.8637870Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:10.8638814Z | 0% 26C P8 21W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:15:10.8639470Z | | | N/A | 2024-08-20T22:15:10.8640219Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:10.8642952Z 2024-08-20T22:15:10.8644121Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:10.8644761Z | Processes: | 2024-08-20T22:15:10.8645477Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:10.8646172Z | ID ID Usage | 2024-08-20T22:15:10.8646744Z |=========================================================================================| 2024-08-20T22:15:10.8648548Z | No running processes found | 2024-08-20T22:15:10.8649322Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:11.6854660Z Command completed after 1 attempt(s). 2024-08-20T22:15:11.6935889Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:11.6936676Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:11.6937359Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:15:11.6938004Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:11.6951309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:11.6951804Z env: 2024-08-20T22:15:11.6952081Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:11.6952524Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:11.6953006Z ##[endgroup] 2024-08-20T22:15:12.1167907Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:12.1434580Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T22:15:12.1439972Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T22:15:12.2891777Z Prepare all required actions 2024-08-20T22:15:12.2892552Z Getting action download info 2024-08-20T22:15:12.4344794Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:15:12.6149489Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:15:12.7427519Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:15:12.7428007Z with: 2024-08-20T22:15:12.7428363Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:12.7428847Z s3-bucket: gha-artifacts 2024-08-20T22:15:12.7429192Z env: 2024-08-20T22:15:12.7429471Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:12.7429932Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:12.7430427Z ##[endgroup] 2024-08-20T22:15:12.7464662Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:12.7465115Z with: 2024-08-20T22:15:12.7465505Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:12.7465964Z s3-bucket: gha-artifacts 2024-08-20T22:15:12.7466339Z region: us-east-1 2024-08-20T22:15:12.7466648Z env: 2024-08-20T22:15:12.7466932Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:12.7467381Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:12.7467867Z ##[endgroup] 2024-08-20T22:15:13.2210849Z (node:223123) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:13.2212010Z 2024-08-20T22:15:13.2212428Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:13.2213591Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:13.2214981Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:13.2938225Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:15:13.2939978Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:20.8286573Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:20.8293891Z Artifact download has finished successfully 2024-08-20T22:15:20.8653123Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:15:20.8653562Z unzip -o artifacts.zip 2024-08-20T22:15:20.8663105Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:20.8663613Z env: 2024-08-20T22:15:20.8663893Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:20.8664458Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:20.8664949Z ##[endgroup] 2024-08-20T22:15:20.8754775Z Archive: artifacts.zip 2024-08-20T22:15:20.8756564Z creating: dist/ 2024-08-20T22:15:23.0045602Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:23.0046495Z creating: build/custom_test_artifacts/ 2024-08-20T22:15:23.0047221Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:15:23.0047964Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:15:23.0048813Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:23.0054840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:23.0056120Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:23.0057315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:23.0058308Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:23.0059488Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:23.0060733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:23.0062574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:23.0063847Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:23.0065204Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:23.0066639Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:23.0068061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:23.0069479Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:23.0070960Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:23.0072406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:23.0073959Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:23.0075224Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:23.0076237Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:23.0116336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:23.0156044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:23.0157568Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:23.0203506Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:23.0205405Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:23.0207368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:23.0209350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:23.0211161Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:23.0213048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:23.0214905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:23.0216751Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:23.0218406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:23.0219713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:23.0220955Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:23.0222174Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:23.0223408Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:23.0224780Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:23.0226012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:23.0279069Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:23.0339412Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:23.0341016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:23.0342354Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:23.0343593Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:23.0345000Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:23.0346272Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:15:23.0347633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:23.0349198Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:15:23.0350678Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:15:23.0351765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:15:23.0352863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:23.0354008Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:15:23.0355107Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:23.0356205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:15:23.0357290Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:15:23.0371696Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:15:23.0515775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:15:23.0517166Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:15:23.0518682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:23.0520298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:15:23.0521663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:15:23.0523141Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:15:23.0524692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:23.0525854Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:15:23.0526999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:23.0528143Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:15:23.0529264Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:15:23.0545967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:15:23.0629443Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:15:23.0631090Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:23.0632483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:23.0633997Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:15:23.0635313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:15:23.0636223Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:23.0637166Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:15:23.0638324Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:15:23.0639366Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:15:23.0640420Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:15:23.0760360Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:15:23.0823808Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:15:23.0824887Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:15:23.0825608Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:15:23.0826433Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:23.0832721Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:23.0834053Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:23.0835148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:23.0836143Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:23.0837438Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:23.0838715Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:23.0840097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:23.0841103Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:23.0842321Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:23.0843729Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:23.0845105Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:23.0846648Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:23.0847823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:23.0849333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:23.0850918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:23.0851989Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:23.0853043Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:23.0892294Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:23.0932069Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:23.0933553Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:23.0978951Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:23.0980822Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:23.0982992Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:23.0985341Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:23.0987233Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:23.0989335Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:23.0990982Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:23.0992877Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:23.0994544Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:23.0995847Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:23.0997081Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:23.0998305Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:23.0999538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:23.1000731Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:23.1001961Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:23.1054802Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:23.1115499Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:23.1117114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:23.1118466Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:23.1119642Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:23.1120869Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:23.1122148Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:15:23.1123654Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:15:23.1125235Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:15:23.1126702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:15:23.1127795Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:15:23.1128903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:15:23.1130014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:15:23.1131146Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:15:23.1132259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:15:23.1133390Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:15:23.1147877Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:15:23.1212559Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:15:23.1214369Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:23.1215727Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:23.1216960Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:15:23.1218109Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:15:23.1219012Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:23.1219944Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:15:23.1221091Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:15:23.1222153Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:15:23.1223088Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:15:23.1272592Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:15:23.1273691Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:15:23.1274486Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:15:23.1275374Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:23.1281509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:23.1282782Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:23.1284173Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:23.1285260Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:23.1286445Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:23.1287824Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:23.1289643Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:23.1290746Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:23.1291876Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:23.1293411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:23.1295007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:23.1296597Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:23.1297908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:23.1299520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:23.1301203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:23.1302350Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:23.1303495Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:23.1342442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:23.1382014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:23.1385055Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:23.1429438Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:23.1431429Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:23.1433493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:23.1435643Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:23.1437511Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:23.1439502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:23.1441484Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:23.1443279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:23.1444881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:23.1446226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:23.1447533Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:23.1448812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:23.1450107Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:23.1451365Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:23.1452657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:23.1505897Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:23.1565011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:23.1566704Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:23.1568141Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:23.1569405Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:23.1570757Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:23.1572070Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:15:23.1573663Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:23.1575342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:15:23.1576733Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:15:23.1577889Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:15:23.1579087Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:23.1580786Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:15:23.1582240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:23.1583460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:15:23.1584771Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:15:23.1586039Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:15:23.1707332Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:15:23.1708929Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:15:23.1710542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:23.1712273Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:15:23.1713980Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:15:23.1715562Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:15:23.1716812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:23.1718056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:15:23.1719303Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:23.1720545Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:15:23.1721782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:15:23.1737248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:15:23.1793312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:15:23.1795054Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:23.1796633Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:23.1797894Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:15:23.1799205Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:15:23.1800258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:23.1801250Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:15:23.1802525Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:15:23.1803712Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:15:23.1804755Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:15:23.1907900Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:15:23.1951811Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:15:23.1953343Z creating: build/lib/ 2024-08-20T22:15:23.2040164Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:15:23.2496029Z inflating: build/lib/libprotobuf.a 2024-08-20T22:15:23.2505951Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:15:23.2514241Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:15:23.2522618Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:15:23.2523470Z inflating: build/lib/libclog.a 2024-08-20T22:15:23.2541600Z inflating: build/lib/libnnpack.a 2024-08-20T22:15:23.2544241Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:15:23.2609728Z inflating: build/lib/libgtest.a 2024-08-20T22:15:23.2684858Z inflating: build/lib/libbenchmark.a 2024-08-20T22:15:23.2749120Z inflating: build/lib/libasmjit.a 2024-08-20T22:15:23.2756802Z inflating: build/lib/libittnotify.a 2024-08-20T22:15:23.2784795Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:15:23.2913064Z inflating: build/lib/libgloo.a 2024-08-20T22:15:23.2933401Z inflating: build/lib/libfmt.a 2024-08-20T22:15:23.3029237Z inflating: build/lib/libc10.so 2024-08-20T22:15:23.3030460Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:15:23.3032356Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:15:23.3537183Z inflating: build/lib/libprotoc.a 2024-08-20T22:15:23.3556305Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:15:23.3574354Z inflating: build/lib/libgmock.a 2024-08-20T22:15:23.3575145Z inflating: build/lib/libgtest_main.a 2024-08-20T22:15:23.3575938Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:15:23.3967276Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:15:24.4154825Z inflating: build/lib/libdnnl.a 2024-08-20T22:15:24.4728968Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:15:24.4786449Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:15:24.4787212Z inflating: build/lib/libgmock_main.a 2024-08-20T22:15:24.6053911Z inflating: build/lib/libfbgemm.a 2024-08-20T22:15:24.6309811Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:15:24.6816782Z inflating: build/lib/libkineto.a 2024-08-20T22:15:24.7001174Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:15:24.7043888Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:15:24.7738511Z inflating: build/lib/libonnx.a 2024-08-20T22:15:27.2281116Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:15:27.2285530Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:15:27.2290519Z inflating: build/lib/libshm.so 2024-08-20T22:15:29.2996595Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:15:29.2997692Z inflating: build/lib/libtorch.so 2024-08-20T22:15:29.3000849Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:15:30.1199557Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:15:30.3204677Z inflating: build/lib/libtorch_python.so 2024-08-20T22:15:30.3277169Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:15:30.3298543Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:15:30.3324618Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:15:30.3351130Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:15:30.3385987Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:15:30.3386630Z creating: build/bin/ 2024-08-20T22:15:30.3437908Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:15:30.3491066Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:15:30.3542007Z inflating: build/bin/c10_Device_test 2024-08-20T22:15:30.3602879Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:15:30.3657669Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:15:30.3708262Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:15:30.3760445Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:15:30.3815981Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:15:30.3873648Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:15:30.3931370Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:15:30.4002613Z inflating: build/bin/c10_cow_test 2024-08-20T22:15:30.4056242Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:15:30.4105826Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:15:30.4156438Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:15:30.4207695Z inflating: build/bin/c10_Half_test 2024-08-20T22:15:30.4264894Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:15:30.4320615Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:15:30.4371980Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:15:30.4427720Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:15:30.4479998Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:15:30.4532502Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:15:30.4583945Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:15:30.4636771Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:15:30.4694800Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:15:30.4746383Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:15:30.4803884Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:15:30.4860074Z inflating: build/bin/c10_complex_test 2024-08-20T22:15:30.4913839Z inflating: build/bin/c10_exception_test 2024-08-20T22:15:30.4964586Z inflating: build/bin/c10_flags_test 2024-08-20T22:15:30.5015649Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:15:30.5181618Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:15:30.5234771Z inflating: build/bin/c10_irange_test 2024-08-20T22:15:30.5288218Z inflating: build/bin/c10_lazy_test 2024-08-20T22:15:30.5346442Z inflating: build/bin/c10_logging_test 2024-08-20T22:15:30.5423126Z inflating: build/bin/c10_optional_test 2024-08-20T22:15:30.5485445Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:15:30.5541571Z inflating: build/bin/c10_registry_test 2024-08-20T22:15:30.5693760Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:15:30.5746232Z inflating: build/bin/c10_ssize_test 2024-08-20T22:15:30.5798724Z inflating: build/bin/c10_string_util_test 2024-08-20T22:15:30.5849709Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:15:30.5909408Z inflating: build/bin/c10_string_view_test 2024-08-20T22:15:30.5958600Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:15:30.6015350Z inflating: build/bin/c10_typeid_test 2024-08-20T22:15:30.6465308Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:15:30.6916213Z inflating: build/bin/protoc 2024-08-20T22:15:30.6970719Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:15:30.7024527Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:15:30.7076895Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:15:30.7130097Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:15:30.7183317Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:15:30.7236019Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:15:30.7285233Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:15:30.7339426Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:15:30.7672072Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:15:30.8018001Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:15:30.8379532Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:15:30.8432512Z inflating: build/bin/BackoffTest 2024-08-20T22:15:30.8485856Z inflating: build/bin/FileStoreTest 2024-08-20T22:15:30.8542213Z inflating: build/bin/TCPStoreTest 2024-08-20T22:15:30.8596184Z inflating: build/bin/HashStoreTest 2024-08-20T22:15:30.8609946Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:15:30.8664411Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:15:30.8668778Z inflating: build/bin/torch_shm_manager 2024-08-20T22:15:30.8671799Z inflating: build/bin/example_allreduce 2024-08-20T22:15:30.8727818Z inflating: build/bin/test_dist_autograd 2024-08-20T22:15:30.8797220Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:15:30.8800028Z inflating: build/bin/parallel_benchmark 2024-08-20T22:15:30.8867600Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:15:30.8876592Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:15:30.9224829Z inflating: build/bin/test_lazy 2024-08-20T22:15:31.0413521Z inflating: build/bin/test_api 2024-08-20T22:15:31.0487936Z inflating: build/bin/Dict_test 2024-08-20T22:15:31.0540815Z inflating: build/bin/Dimname_test 2024-08-20T22:15:31.0606323Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:15:31.0664058Z inflating: build/bin/NamedTensor_test 2024-08-20T22:15:31.0724138Z inflating: build/bin/apply_utils_test 2024-08-20T22:15:31.0784578Z inflating: build/bin/atest 2024-08-20T22:15:31.0848925Z inflating: build/bin/basic 2024-08-20T22:15:31.0903623Z inflating: build/bin/broadcast_test 2024-08-20T22:15:31.0954776Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:15:31.1013498Z inflating: build/bin/cpu_generator_test 2024-08-20T22:15:31.1067217Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:15:31.1161033Z inflating: build/bin/cpu_rng_test 2024-08-20T22:15:31.1211591Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:15:31.1263075Z inflating: build/bin/dlconvertor_test 2024-08-20T22:15:31.1321268Z inflating: build/bin/extension_backend_test 2024-08-20T22:15:31.1377810Z inflating: build/bin/half_test 2024-08-20T22:15:31.1473799Z inflating: build/bin/ivalue_test 2024-08-20T22:15:31.1523909Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:15:31.1579429Z inflating: build/bin/math_kernel_test 2024-08-20T22:15:31.1633530Z inflating: build/bin/memory_format_test 2024-08-20T22:15:31.1687588Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:15:31.1741147Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:15:31.1797261Z inflating: build/bin/native_test 2024-08-20T22:15:31.1848900Z inflating: build/bin/operator_name_test 2024-08-20T22:15:31.1901143Z inflating: build/bin/operators_test 2024-08-20T22:15:31.1953306Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:15:31.2021734Z inflating: build/bin/pow_test 2024-08-20T22:15:31.2078757Z inflating: build/bin/quantized_test 2024-08-20T22:15:31.2129362Z inflating: build/bin/reduce_ops_test 2024-08-20T22:15:31.2180960Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:15:31.2238152Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:15:31.2297438Z inflating: build/bin/scalar_test 2024-08-20T22:15:31.2349612Z inflating: build/bin/StorageUtils_test 2024-08-20T22:15:31.2402910Z inflating: build/bin/stride_properties_test 2024-08-20T22:15:31.2482485Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:15:31.2537123Z inflating: build/bin/test_parallel 2024-08-20T22:15:31.2540145Z inflating: build/bin/thread_init_test 2024-08-20T22:15:31.2596816Z inflating: build/bin/type_ptr_test 2024-08-20T22:15:31.2657179Z inflating: build/bin/type_test 2024-08-20T22:15:31.2710110Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:15:31.2711508Z inflating: build/bin/verify_api_visibility 2024-08-20T22:15:31.2781467Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:15:31.2834645Z inflating: build/bin/weakref_test 2024-08-20T22:15:31.2885949Z inflating: build/bin/wrapdim_test 2024-08-20T22:15:31.2938183Z inflating: build/bin/xla_tensor_test 2024-08-20T22:15:31.2998799Z inflating: build/bin/IListRef_test 2024-08-20T22:15:31.3106305Z inflating: build/bin/List_test 2024-08-20T22:15:31.3174036Z inflating: build/bin/KernelFunction_test 2024-08-20T22:15:31.3296444Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:15:31.3393829Z inflating: build/bin/kernel_function_test 2024-08-20T22:15:31.3523432Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:15:31.3627766Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:15:31.3690205Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:15:31.3786821Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:15:31.3838136Z inflating: build/bin/CppSignature_test 2024-08-20T22:15:31.3887597Z inflating: build/bin/op_allowlist_test 2024-08-20T22:15:31.3944234Z inflating: build/bin/backend_fallback_test 2024-08-20T22:15:31.4249168Z inflating: build/bin/op_registration_test 2024-08-20T22:15:31.4314165Z inflating: build/bin/inline_container_test 2024-08-20T22:15:31.4367498Z inflating: build/bin/cuda_apply_test 2024-08-20T22:15:31.4420136Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:15:31.4478914Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:15:31.4534852Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:15:31.4604865Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:15:31.4664547Z inflating: build/bin/cuda_complex_test 2024-08-20T22:15:31.4715220Z inflating: build/bin/cuda_device_test 2024-08-20T22:15:31.4773593Z inflating: build/bin/cuda_cub_test 2024-08-20T22:15:31.4824670Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:15:31.4890484Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:15:31.4947071Z inflating: build/bin/cuda_generator_test 2024-08-20T22:15:31.4997162Z inflating: build/bin/cuda_half_test 2024-08-20T22:15:31.5049464Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:15:31.5098636Z inflating: build/bin/cuda_optional_test 2024-08-20T22:15:31.5150742Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:15:31.5203969Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:15:31.5254241Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:15:31.5315640Z inflating: build/bin/cuda_stream_test 2024-08-20T22:15:31.5365216Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:15:31.5418097Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:15:31.5432688Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:15:31.5498809Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:15:31.5556715Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:15:31.5620791Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:15:31.5683401Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:15:31.6518047Z inflating: build/bin/test_tensorexpr 2024-08-20T22:15:31.7101207Z inflating: build/bin/test_jit 2024-08-20T22:15:31.7101809Z creating: .additional_ci_files/ 2024-08-20T22:15:31.7163529Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:15:31.7404580Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:15:31.7439287Z ##[group]Run rm artifacts.zip 2024-08-20T22:15:31.7439686Z rm artifacts.zip 2024-08-20T22:15:31.7448704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:31.7449219Z env: 2024-08-20T22:15:31.7449513Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:31.7449975Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:31.7450469Z ##[endgroup] 2024-08-20T22:15:31.8548111Z ##[group]Run df -H 2024-08-20T22:15:31.8548435Z df -H 2024-08-20T22:15:31.8557169Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:31.8557680Z env: 2024-08-20T22:15:31.8557970Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:31.8558430Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:31.8558911Z ##[endgroup] 2024-08-20T22:15:31.8617570Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:15:31.8618303Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:15:31.8618821Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:15:31.8619326Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:15:31.8619805Z /dev/nvme0n1p1 161G 76G 86G 47% / 2024-08-20T22:15:31.8620469Z tmpfs 34G 959k 34G 1% /tmp 2024-08-20T22:15:31.8621507Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:15:31.8622219Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:15:31.8661582Z Prepare all required actions 2024-08-20T22:15:31.8662020Z Getting action download info 2024-08-20T22:15:31.9963501Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:15:31.9963976Z with: 2024-08-20T22:15:31.9964229Z env: 2024-08-20T22:15:31.9964496Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:31.9964949Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:31.9965430Z ##[endgroup] 2024-08-20T22:15:31.9999167Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:31.9999610Z with: 2024-08-20T22:15:31.9999883Z name: td_results 2024-08-20T22:15:32.0000199Z s3-bucket: gha-artifacts 2024-08-20T22:15:32.0000553Z region: us-east-1 2024-08-20T22:15:32.0000855Z env: 2024-08-20T22:15:32.0001134Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.0001587Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.0002084Z ##[endgroup] 2024-08-20T22:15:32.4807077Z (node:223143) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:32.4807775Z 2024-08-20T22:15:32.4808043Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:32.4808802Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:32.4809658Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:32.5409123Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/ 2024-08-20T22:15:32.5416544Z Artifact download has finished successfully 2024-08-20T22:15:32.5737445Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:15:32.5737943Z mkdir -p .additional_ci_files 2024-08-20T22:15:32.5738501Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:15:32.5748650Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:32.5749170Z env: 2024-08-20T22:15:32.5749447Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.5749925Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.5750416Z ##[endgroup] 2024-08-20T22:15:32.5852449Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:15:32.5866154Z ##[error]Process completed with exit code 1. 2024-08-20T22:15:32.5901027Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:15:32.5901515Z .github/scripts/parse_ref.py 2024-08-20T22:15:32.5910367Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:32.5910715Z env: 2024-08-20T22:15:32.5911001Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.5911466Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.5911945Z ##[endgroup] 2024-08-20T22:15:32.6207769Z Prepare all required actions 2024-08-20T22:15:32.6247215Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:15:32.6247675Z with: 2024-08-20T22:15:32.6248580Z github-token: *** 2024-08-20T22:15:32.6248915Z env: 2024-08-20T22:15:32.6249189Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.6249644Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.6250131Z ##[endgroup] 2024-08-20T22:15:32.6267173Z ##[group]Run set -eux 2024-08-20T22:15:32.6267511Z set -eux 2024-08-20T22:15:32.6268099Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:15:32.6277341Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:32.6277837Z env: 2024-08-20T22:15:32.6278121Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:32.6278578Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:32.6279217Z GITHUB_TOKEN: *** 2024-08-20T22:15:32.6279528Z ##[endgroup] 2024-08-20T22:15:32.6308499Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-08b1654f015502267 2024-08-20T22:15:33.2952966Z setting job-id=29026299278 2024-08-20T22:15:33.2954205Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:33.3171280Z Prepare all required actions 2024-08-20T22:15:33.3171731Z Getting action download info 2024-08-20T22:15:33.4576564Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:15:33.4577056Z with: 2024-08-20T22:15:33.4577514Z github-token: *** 2024-08-20T22:15:33.4584098Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:15:33.4591721Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:33.4592616Z env: 2024-08-20T22:15:33.4592898Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:33.4593359Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:33.4593845Z ##[endgroup] 2024-08-20T22:15:33.4637219Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:33.4637770Z with: 2024-08-20T22:15:33.4638038Z shell: bash 2024-08-20T22:15:33.4638328Z timeout_minutes: 10 2024-08-20T22:15:33.4638654Z max_attempts: 5 2024-08-20T22:15:33.4638968Z retry_wait_seconds: 30 2024-08-20T22:15:33.4640094Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:33.4641266Z polling_interval_seconds: 1 2024-08-20T22:15:33.4641650Z warning_on_retry: true 2024-08-20T22:15:33.4642002Z continue_on_error: false 2024-08-20T22:15:33.4642361Z env: 2024-08-20T22:15:33.4642630Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:33.4643080Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:33.4643720Z GITHUB_TOKEN: *** 2024-08-20T22:15:33.4644040Z ##[endgroup] 2024-08-20T22:15:33.7033274Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:34.2871836Z Collecting requests==2.27.1 2024-08-20T22:15:34.3209211Z Downloading requests-2.27.1-py2.py3-none-any.whl.metadata (5.0 kB) 2024-08-20T22:15:34.3662097Z Collecting pyyaml==6.0.1 2024-08-20T22:15:34.3687900Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB) 2024-08-20T22:15:34.4133733Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.27.1) 2024-08-20T22:15:34.4157488Z Downloading urllib3-1.26.19-py2.py3-none-any.whl.metadata (49 kB) 2024-08-20T22:15:34.4326616Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:15:34.4975448Z Collecting charset-normalizer~=2.0.0 (from requests==2.27.1) 2024-08-20T22:15:34.5000978Z Downloading charset_normalizer-2.0.12-py3-none-any.whl.metadata (11 kB) 2024-08-20T22:15:34.5059919Z Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (3.7) 2024-08-20T22:15:34.5135749Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:15:34.5225180Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (724 kB) 2024-08-20T22:15:34.5372357Z ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 725.0/725.0 kB 51.4 MB/s eta 0:00:00 2024-08-20T22:15:34.5386139Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:15:34.5450960Z Downloading urllib3-1.26.19-py2.py3-none-any.whl (143 kB) 2024-08-20T22:15:34.6117544Z Installing collected packages: urllib3, pyyaml, charset-normalizer, requests 2024-08-20T22:15:34.6118506Z Attempting uninstall: urllib3 2024-08-20T22:15:34.6138634Z Found existing installation: urllib3 2.2.2 2024-08-20T22:15:34.6181224Z Uninstalling urllib3-2.2.2: 2024-08-20T22:15:34.6190611Z Successfully uninstalled urllib3-2.2.2 2024-08-20T22:15:34.7352296Z Attempting uninstall: charset-normalizer 2024-08-20T22:15:34.7386233Z Found existing installation: charset-normalizer 3.3.2 2024-08-20T22:15:34.7418142Z Uninstalling charset-normalizer-3.3.2: 2024-08-20T22:15:34.7438390Z Successfully uninstalled charset-normalizer-3.3.2 2024-08-20T22:15:34.7706344Z Attempting uninstall: requests 2024-08-20T22:15:34.7731937Z Found existing installation: requests 2.32.3 2024-08-20T22:15:34.7761511Z Uninstalling requests-2.32.3: 2024-08-20T22:15:34.7768087Z Successfully uninstalled requests-2.32.3 2024-08-20T22:15:34.8134033Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:15:34.8135430Z conda 24.7.1 requires requests<3,>=2.28.0, but you have requests 2.27.1 which is incompatible. 2024-08-20T22:15:34.8136504Z Successfully installed charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 urllib3-1.26.19 2024-08-20T22:15:35.5196430Z Command completed after 1 attempt(s). 2024-08-20T22:15:35.5278007Z ##[group]Run set -x 2024-08-20T22:15:35.5278339Z set -x 2024-08-20T22:15:35.5278635Z  2024-08-20T22:15:35.5279173Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:35.5279849Z # in runner workspace 2024-08-20T22:15:35.5280372Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:15:35.5291397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:35.5291903Z env: 2024-08-20T22:15:35.5292182Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:35.5292660Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:35.5293146Z ##[endgroup] 2024-08-20T22:15:35.5324536Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:15:35.5567755Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:35.5568306Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:35.5568773Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:15:35.5569178Z  2024-08-20T22:15:35.5569728Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:35.5570404Z # in runner workspace 2024-08-20T22:15:35.5570987Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:15:35.5571627Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:15:35.5572086Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:15:35.5579116Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:15:35.5586448Z  --selected-test-configs "" \ 2024-08-20T22:15:35.5586957Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:15:35.5587384Z  --tag "${TAG}" \ 2024-08-20T22:15:35.5587778Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:15:35.5588219Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:15:35.5588648Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:15:35.5597500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:35.5598010Z env: 2024-08-20T22:15:35.5598287Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:35.5598749Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:35.5599430Z GITHUB_TOKEN: *** 2024-08-20T22:15:35.5600259Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:35.5601142Z PR_NUMBER: 2024-08-20T22:15:35.5601453Z TAG: ciflow/inductor/133712 2024-08-20T22:15:35.5601823Z EVENT_NAME: push 2024-08-20T22:15:35.5602137Z SCHEDULE: 2024-08-20T22:15:35.5602425Z HEAD_BRANCH: 2024-08-20T22:15:35.5602729Z ##[endgroup] 2024-08-20T22:15:35.5630387Z Workflow: inductor-periodic 2024-08-20T22:15:35.5631408Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:35.8171695Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:15:35.9708151Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:15:35.9708639Z echo "Filtered matrix:" 2024-08-20T22:15:35.9716623Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:15:35.9723672Z  2024-08-20T22:15:35.9723957Z echo 2024-08-20T22:15:35.9724342Z echo "Is the current job unstable? False" 2024-08-20T22:15:35.9724796Z  2024-08-20T22:15:35.9725081Z echo 2024-08-20T22:15:35.9725454Z echo "Is keep-going label set? False" 2024-08-20T22:15:35.9725891Z  2024-08-20T22:15:35.9726167Z echo 2024-08-20T22:15:35.9726487Z echo "Renabled issues? " 2024-08-20T22:15:35.9735320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:35.9735811Z env: 2024-08-20T22:15:35.9736094Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:35.9736556Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:35.9737067Z ##[endgroup] 2024-08-20T22:15:35.9766797Z Filtered matrix: 2024-08-20T22:15:35.9774381Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:15:35.9781045Z 2024-08-20T22:15:35.9781207Z Is the current job unstable? False 2024-08-20T22:15:35.9781491Z 2024-08-20T22:15:35.9781861Z Is keep-going label set? False 2024-08-20T22:15:35.9782119Z 2024-08-20T22:15:35.9782241Z Renabled issues? 2024-08-20T22:15:35.9824932Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:35.9825640Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:35.9833949Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:35.9834437Z env: 2024-08-20T22:15:35.9834709Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:35.9835160Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:35.9835644Z JOB_TIMEOUT: 240 2024-08-20T22:15:35.9835944Z ##[endgroup] 2024-08-20T22:15:35.9916898Z ##[group]Run set -x 2024-08-20T22:15:35.9917332Z set -x 2024-08-20T22:15:35.9917631Z  2024-08-20T22:15:35.9918166Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:15:35.9918708Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:15:35.9919268Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:15:35.9919772Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:15:35.9920186Z else 2024-08-20T22:15:35.9920526Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:35.9920948Z fi 2024-08-20T22:15:35.9921229Z  2024-08-20T22:15:35.9921692Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:15:35.9922439Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:15:35.9923114Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:15:35.9923694Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:15:35.9924155Z container_name=$(docker run \ 2024-08-20T22:15:35.9924576Z  ${GPU_FLAG:-} \ 2024-08-20T22:15:35.9924961Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:15:35.9925363Z  -e PR_NUMBER \ 2024-08-20T22:15:35.9925723Z  -e GITHUB_ACTIONS \ 2024-08-20T22:15:35.9926115Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:15:35.9926521Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:15:35.9926910Z  -e GITHUB_JOB \ 2024-08-20T22:15:35.9927297Z  -e GITHUB_RUN_ID \ 2024-08-20T22:15:35.9927709Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:15:35.9928114Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:15:35.9928515Z  -e JOB_ID \ 2024-08-20T22:15:35.9928854Z  -e JOB_NAME \ 2024-08-20T22:15:35.9929205Z  -e BASE_SHA \ 2024-08-20T22:15:35.9929551Z  -e BRANCH \ 2024-08-20T22:15:35.9929880Z  -e SHA1 \ 2024-08-20T22:15:35.9930229Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:15:35.9930637Z  -e IN_WHEEL_TEST \ 2024-08-20T22:15:35.9931017Z  -e SHARD_NUMBER \ 2024-08-20T22:15:35.9931392Z  -e TEST_CONFIG \ 2024-08-20T22:15:35.9931766Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:15:35.9932164Z  -e REENABLED_ISSUES \ 2024-08-20T22:15:35.9932581Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:15:35.9933000Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:15:35.9933401Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:15:35.9933796Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:15:35.9934176Z  -e NO_TD \ 2024-08-20T22:15:35.9934518Z  -e TD_DISTRIBUTED \ 2024-08-20T22:15:35.9934902Z  -e PR_LABELS \ 2024-08-20T22:15:35.9935312Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:15:35.9935762Z  -e SCCACHE_BUCKET \ 2024-08-20T22:15:35.9936159Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:15:35.9936573Z  -e XLA_CUDA \ 2024-08-20T22:15:35.9936982Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:15:35.9937489Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:15:35.9938019Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:15:35.9938534Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:15:35.9939004Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:15:35.9939453Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:15:35.9939896Z  -e DASHBOARD_TAG \ 2024-08-20T22:15:35.9940373Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:15:35.9940933Z  --security-opt seccomp=unconfined \ 2024-08-20T22:15:35.9941399Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:15:35.9941792Z  --ipc=host \ 2024-08-20T22:15:35.9942155Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:15:35.9942554Z  --tty \ 2024-08-20T22:15:35.9942868Z  --detach \ 2024-08-20T22:15:35.9943358Z  --name="${container_name}" \ 2024-08-20T22:15:35.9943785Z  --user jenkins \ 2024-08-20T22:15:35.9944275Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:15:35.9944995Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:15:35.9945436Z  "${DOCKER_IMAGE}" 2024-08-20T22:15:35.9945793Z ) 2024-08-20T22:15:35.9946201Z # Propagate download.pytorch.org IP to container 2024-08-20T22:15:35.9947163Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:15:35.9948127Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:15:35.9949034Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:15:35.9957472Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:35.9957816Z env: 2024-08-20T22:15:35.9958094Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:35.9958562Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:35.9959164Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:35.9959672Z PR_NUMBER: 2024-08-20T22:15:35.9959997Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:15:35.9960432Z GITHUB_WORKFLOW: inductor-periodic 2024-08-20T22:15:35.9960838Z GITHUB_JOB: test 2024-08-20T22:15:35.9961153Z GITHUB_RUN_ID: 10479310721 2024-08-20T22:15:35.9961518Z GITHUB_RUN_NUMBER: 54565 2024-08-20T22:15:35.9961873Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:15:35.9962213Z JOB_ID: 29026299278 2024-08-20T22:15:35.9963048Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:35.9963939Z BRANCH: 2024-08-20T22:15:35.9964283Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:35.9964809Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:35.9965290Z TEST_CONFIG: aot_eager_huggingface 2024-08-20T22:15:35.9965700Z SHARD_NUMBER: 1 2024-08-20T22:15:35.9966016Z NUM_TEST_SHARDS: 1 2024-08-20T22:15:35.9966342Z REENABLED_ISSUES: 2024-08-20T22:15:35.9966682Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:15:35.9967079Z VERBOSE_TEST_LOGS: False 2024-08-20T22:15:35.9967439Z TEST_SHOWLOCALS: False 2024-08-20T22:15:35.9967783Z NO_TEST_TIMEOUT: False 2024-08-20T22:15:35.9968123Z NO_TD: False 2024-08-20T22:15:35.9968426Z TD_DISTRIBUTED: False 2024-08-20T22:15:35.9968844Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:15:35.9969360Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-08-20T22:15:35.9969785Z SHM_SIZE: 2g 2024-08-20T22:15:35.9970879Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:35.9972057Z XLA_CUDA: 2024-08-20T22:15:35.9972535Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:35.9973153Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:15:35.9973591Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:15:35.9974007Z DASHBOARD_TAG: 2024-08-20T22:15:35.9974529Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:15:35.9974911Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:15:35.9975285Z ##[endgroup] 2024-08-20T22:15:36.0002522Z + [[ aot_eager_huggingface == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:15:36.0003424Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:15:36.0004113Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:36.0011364Z +++ nproc --ignore=2 2024-08-20T22:15:36.0042962Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:47.6381842Z + container_name=ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T22:15:47.6384976Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:15:47.6386939Z + docker exec -i ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:15:47.7921321Z + echo DOCKER_CONTAINER_ID=ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T22:15:47.7925902Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:47.7929491Z + docker exec -t ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:15:48.1936244Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:15:48.5179982Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:15:48.5182989Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:15:48.5186526Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:15:48.5190572Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:15:48.5194582Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:15:48.5200092Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:15:48.5217500Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:15:48.5231810Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:15:48.5250383Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:15:48.5638887Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:15:49.1427218Z Installing collected packages: torch 2024-08-20T22:15:58.9686778Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:15:58.9688634Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:15:58.9689690Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:15:59.0517788Z + export TERM=vt100 2024-08-20T22:15:59.0518750Z + TERM=vt100 2024-08-20T22:15:59.0521103Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:15:59.0588587Z + source .ci/pytorch/common.sh 2024-08-20T22:15:59.0592778Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:15:59.0601561Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:15:59.0603735Z +++ declare -f -t trap_add 2024-08-20T22:15:59.0609301Z ++ set -ex 2024-08-20T22:15:59.0610030Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:15:59.0610565Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:15:59.0611086Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:15:59.0614017Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:15:59.0837997Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:15:59.0838436Z + trap_add cleanup_workspace EXIT 2024-08-20T22:15:59.0838863Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:15:59.0839229Z + shift 2024-08-20T22:15:59.0841751Z + for trap_add_name in "$@" 2024-08-20T22:15:59.0847403Z +++ trap -p EXIT 2024-08-20T22:15:59.0850385Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:15:59.0850834Z +++ extract_trap_cmd 2024-08-20T22:15:59.0851323Z +++ printf '%s\n' '' 2024-08-20T22:15:59.0851732Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:15:59.0854333Z + trap -- ' 2024-08-20T22:15:59.0855190Z cleanup_workspace' EXIT 2024-08-20T22:15:59.0855716Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:15:59.8334020Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:15:59.8356463Z + echo 'Environment variables:' 2024-08-20T22:15:59.8356968Z Environment variables: 2024-08-20T22:15:59.8357315Z + env 2024-08-20T22:15:59.8545797Z INSTALLED_DB=yes 2024-08-20T22:15:59.8546302Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:15:59.8546815Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:15:59.8547403Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:15:59.8548229Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:15:59.8548912Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:15:59.8549434Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:15:59.8549961Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:15:59.8550551Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:59.8551047Z HOSTNAME=ed7e90a491ee 2024-08-20T22:15:59.8551888Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8552940Z GITHUB_ACTION=__self 2024-08-20T22:15:59.8553422Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:15:59.8557849Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:15:59.8562700Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:15:59.8563209Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:15:59.8563616Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:15:59.8563973Z TEST_CONFIG=aot_eager_huggingface 2024-08-20T22:15:59.8564375Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:15:59.8564854Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:15:59.8565322Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:15:59.8565765Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:15:59.8566431Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:15:59.8566848Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:15:59.8567297Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:15:59.8567943Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:15:59.8568405Z GITHUB_REF_TYPE=tag 2024-08-20T22:15:59.8568734Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:15:59.8569121Z NCCL_VERSION=2.17.1-1 2024-08-20T22:15:59.8569510Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8569953Z XLA_CUDA= 2024-08-20T22:15:59.8570447Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:15:59.8572731Z *** 2024-08-20T22:15:59.8573033Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:15:59.8573445Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:15:59.8573850Z GITHUB_ACTIONS=true 2024-08-20T22:15:59.8574181Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:59.8574709Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:15:59.8575252Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:15:59.8575731Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8576244Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:15:59.8576725Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8577664Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:15:59.8578424Z UCC_HOME=/usr 2024-08-20T22:15:59.8578764Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:15:59.8579160Z VERBOSE_TEST_LOGS=False 2024-08-20T22:15:59.8579500Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:15:59.8579968Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:15:59.8580460Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:15:59.8580925Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:15:59.8581304Z SHARD_NUMBER=1 2024-08-20T22:15:59.8581615Z GITHUB_REF_PROTECTED=false 2024-08-20T22:15:59.8581967Z HOME=/var/lib/jenkins 2024-08-20T22:15:59.8582326Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:15:59.8582769Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:15:59.8583246Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:15:59.8583790Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:15:59.8584208Z CUDA_VERSION=12.1.1 2024-08-20T22:15:59.8584746Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:15:59.8585183Z NUM_TEST_SHARDS=1 2024-08-20T22:15:59.8585477Z UCX_HOME=/usr 2024-08-20T22:15:59.8585994Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:15:59.8587069Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8588493Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:59.8590261Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8591387Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:15:59.8592038Z GITHUB_EVENT_NAME=push 2024-08-20T22:15:59.8592376Z DASHBOARD_TAG= 2024-08-20T22:15:59.8592680Z GITHUB_RUN_ID=10479310721 2024-08-20T22:15:59.8593143Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:15:59.8593731Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:15:59.8594721Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8595618Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:15:59.8596025Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:15:59.8596401Z PR_NUMBER= 2024-08-20T22:15:59.8596687Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:15:59.8597019Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:15:59.8597466Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:15:59.8597921Z TERM=vt100 2024-08-20T22:15:59.8598251Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:15:59.8598665Z INSTALLED_VISION=yes 2024-08-20T22:15:59.8599309Z BRANCH= 2024-08-20T22:15:59.8599598Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:15:59.8599995Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:15:59.8600534Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:15:59.8601295Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:15:59.8602037Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:15:59.8602512Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:15:59.8602981Z REENABLED_ISSUES= 2024-08-20T22:15:59.8603277Z SHLVL=1 2024-08-20T22:15:59.8603545Z MAX_JOBS=14 2024-08-20T22:15:59.8603900Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:15:59.8604251Z NVARCH=x86_64 2024-08-20T22:15:59.8604550Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:15:59.8604998Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8605534Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:15:59.8606024Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:15:59.8606693Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:59.8607256Z GITHUB_JOB=test 2024-08-20T22:15:59.8607659Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:15:59.8608217Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:15:59.8608726Z NO_TEST_TIMEOUT=False 2024-08-20T22:15:59.8609051Z TD_DISTRIBUTED=False 2024-08-20T22:15:59.8609435Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:15:59.8609870Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:15:59.8610287Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:15:59.8610650Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:15:59.8610997Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:15:59.8611465Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:15:59.8612493Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:15:59.8613606Z GITHUB_BASE_REF= 2024-08-20T22:15:59.8613935Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:15:59.8614386Z CI=true 2024-08-20T22:15:59.8614701Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:15:59.8615192Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:15:59.8615576Z JOB_ID=29026299278 2024-08-20T22:15:59.8615890Z INSTALLED_PROTOBUF=yes 2024-08-20T22:15:59.8616214Z GITHUB_HEAD_REF= 2024-08-20T22:15:59.8616517Z GITHUB_ACTION_REF= 2024-08-20T22:15:59.8616952Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:15:59.8617409Z TEST_SHOWLOCALS=False 2024-08-20T22:15:59.8617801Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:15:59.8618205Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:15:59.8619120Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8619922Z NO_TD=False 2024-08-20T22:15:59.8620224Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:15:59.8620589Z _=/usr/bin/env 2024-08-20T22:15:59.8621071Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:15:59.8796957Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:15:59.8798067Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:15:59.8799034Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:15:59.8800104Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:15:59.8800711Z + BUILD_DIR=build 2024-08-20T22:15:59.8801041Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:15:59.8801437Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:15:59.8801783Z + SHARD_NUMBER=1 2024-08-20T22:15:59.8802082Z + NUM_TEST_SHARDS=1 2024-08-20T22:15:59.8802403Z + export VALGRIND=ON 2024-08-20T22:15:59.8802727Z + VALGRIND=ON 2024-08-20T22:15:59.8803310Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:15:59.8803836Z + [[ 0 == \1 ]] 2024-08-20T22:15:59.8804134Z + [[ False == \1 ]] 2024-08-20T22:15:59.8805017Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:15:59.8805618Z ++ realpath build/custom_test_artifacts 2024-08-20T22:15:59.8940579Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:15:59.8941493Z + [[ -n '' ]] 2024-08-20T22:15:59.8941860Z + echo 'Environment variables' 2024-08-20T22:15:59.8942233Z Environment variables 2024-08-20T22:15:59.8942550Z + env 2024-08-20T22:15:59.8950014Z INSTALLED_DB=yes 2024-08-20T22:15:59.8950552Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:15:59.8951016Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:15:59.8951557Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:15:59.8952379Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:15:59.8953154Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:15:59.8953860Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:15:59.8954558Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:15:59.8955325Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:59.8955914Z HOSTNAME=ed7e90a491ee 2024-08-20T22:15:59.8956847Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8957649Z GITHUB_ACTION=__self 2024-08-20T22:15:59.8958002Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:15:59.8961766Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:15:59.8965599Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:15:59.8966113Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:15:59.8966492Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:15:59.8966900Z TEST_CONFIG=aot_eager_huggingface 2024-08-20T22:15:59.8967311Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:15:59.8967792Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:15:59.8968251Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:15:59.8968694Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:15:59.8969101Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:15:59.8969496Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:15:59.8969975Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:15:59.8970504Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:15:59.8970962Z GITHUB_REF_TYPE=tag 2024-08-20T22:15:59.8971289Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:15:59.8971664Z NCCL_VERSION=2.17.1-1 2024-08-20T22:15:59.8972063Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8972509Z XLA_CUDA= 2024-08-20T22:15:59.8973023Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:15:59.8973425Z *** 2024-08-20T22:15:59.8973711Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:15:59.8974106Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:15:59.8974473Z GITHUB_ACTIONS=true 2024-08-20T22:15:59.8974799Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:59.8975302Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:15:59.8975844Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:15:59.8976313Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8976826Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:15:59.8977310Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.8978240Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:15:59.8978990Z UCC_HOME=/usr 2024-08-20T22:15:59.8979511Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:15:59.8979921Z VERBOSE_TEST_LOGS=False 2024-08-20T22:15:59.8980264Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:15:59.8980810Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:15:59.8981307Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:15:59.8981774Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:15:59.8982155Z SHARD_NUMBER=1 2024-08-20T22:15:59.8982464Z GITHUB_REF_PROTECTED=false 2024-08-20T22:15:59.8982863Z HOME=/var/lib/jenkins 2024-08-20T22:15:59.8983271Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:15:59.8983708Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:15:59.8984179Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:15:59.8984850Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:15:59.8985267Z CUDA_VERSION=12.1.1 2024-08-20T22:15:59.8985693Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:15:59.8986124Z NUM_TEST_SHARDS=1 2024-08-20T22:15:59.8986425Z UCX_HOME=/usr 2024-08-20T22:15:59.8986945Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:15:59.8988026Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8989816Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:59.8991200Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8992303Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:15:59.8992945Z GITHUB_EVENT_NAME=push 2024-08-20T22:15:59.8993270Z DASHBOARD_TAG= 2024-08-20T22:15:59.8993599Z GITHUB_RUN_ID=10479310721 2024-08-20T22:15:59.8994091Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:15:59.8994630Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:15:59.8995619Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.8996525Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:15:59.8996949Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:15:59.8997327Z PR_NUMBER= 2024-08-20T22:15:59.8997609Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:15:59.8997939Z VALGRIND=ON 2024-08-20T22:15:59.8998241Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:15:59.8998676Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:15:59.8999136Z TERM=vt100 2024-08-20T22:15:59.8999475Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:15:59.8999887Z INSTALLED_VISION=yes 2024-08-20T22:15:59.9000197Z BRANCH= 2024-08-20T22:15:59.9000486Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:15:59.9000888Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:15:59.9001312Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:15:59.9002063Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:15:59.9002817Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:15:59.9003314Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:15:59.9003784Z REENABLED_ISSUES= 2024-08-20T22:15:59.9004085Z SHLVL=1 2024-08-20T22:15:59.9004348Z MAX_JOBS=14 2024-08-20T22:15:59.9004666Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:15:59.9005013Z NVARCH=x86_64 2024-08-20T22:15:59.9005312Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:15:59.9005772Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:59.9006305Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:15:59.9006789Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:15:59.9007451Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:59.9008019Z GITHUB_JOB=test 2024-08-20T22:15:59.9008425Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:15:59.9008974Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:15:59.9009483Z NO_TEST_TIMEOUT=False 2024-08-20T22:15:59.9009972Z TD_DISTRIBUTED=False 2024-08-20T22:15:59.9010363Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:15:59.9010797Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:15:59.9011339Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:15:59.9011710Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:15:59.9012053Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:15:59.9012408Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:15:59.9013438Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:15:59.9014542Z GITHUB_BASE_REF= 2024-08-20T22:15:59.9014860Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:15:59.9015231Z CI=true 2024-08-20T22:15:59.9015549Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:15:59.9015964Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:15:59.9016333Z JOB_ID=29026299278 2024-08-20T22:15:59.9016648Z INSTALLED_PROTOBUF=yes 2024-08-20T22:15:59.9016983Z GITHUB_HEAD_REF= 2024-08-20T22:15:59.9017280Z GITHUB_ACTION_REF= 2024-08-20T22:15:59.9017723Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:15:59.9018187Z TEST_SHOWLOCALS=False 2024-08-20T22:15:59.9018589Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:15:59.9018989Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:15:59.9019897Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_b2d9d143-cf95-4e3a-a1af-01988df310cb 2024-08-20T22:15:59.9020702Z NO_TD=False 2024-08-20T22:15:59.9021007Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:15:59.9021368Z _=/usr/bin/env 2024-08-20T22:15:59.9021713Z + echo 'Testing pytorch' 2024-08-20T22:15:59.9022057Z Testing pytorch 2024-08-20T22:15:59.9022378Z + export LANG=C.UTF-8 2024-08-20T22:15:59.9022720Z + LANG=C.UTF-8 2024-08-20T22:15:59.9969790Z + PR_NUMBER= 2024-08-20T22:15:59.9970200Z + [[ aot_eager_huggingface == \d\e\f\a\u\l\t ]] 2024-08-20T22:15:59.9970847Z + [[ aot_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:15:59.9971448Z + [[ aot_eager_huggingface == \s\l\o\w ]] 2024-08-20T22:15:59.9972147Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:15:59.9972852Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:15:59.9973414Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:15:59.9973933Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:15:59.9974387Z + [[ aot_eager_huggingface == *crossref* ]] 2024-08-20T22:15:59.9974975Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:15:59.9975608Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:15:59.9976273Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:15:59.9976837Z + pip_install --user ninja==1.10.2 2024-08-20T22:15:59.9977379Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:16:01.0213243Z Collecting ninja==1.10.2 2024-08-20T22:16:01.0379846Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:16:01.0851173Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:16:01.7089180Z Installing collected packages: ninja 2024-08-20T22:16:01.7171822Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:16:01.7173169Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:16:01.7727291Z Successfully installed ninja-1.10.2 2024-08-20T22:16:01.8505619Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:01.8507811Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:01.8509804Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:01.8510319Z + install_tlparse 2024-08-20T22:16:01.8510709Z + pip_install --user tlparse==0.3.25 2024-08-20T22:16:01.8511474Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:16:02.2815704Z Collecting tlparse==0.3.25 2024-08-20T22:16:02.2990715Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:16:02.3508953Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:16:03.0104337Z Installing collected packages: tlparse 2024-08-20T22:16:03.0896310Z Successfully installed tlparse-0.3.25 2024-08-20T22:16:03.1678721Z ++ python -m site --user-base 2024-08-20T22:16:03.1923438Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:03.1925651Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:16:03.1926573Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:16:03.1927358Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:03.1928320Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:16:03.1929705Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:16:03.1930600Z + cd test 2024-08-20T22:16:03.1931139Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:16:05.4702991Z + [[ aot_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:16:05.4703640Z + [[ aot_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:16:05.4707268Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:16:05.4707748Z + [[ aot_eager_huggingface == *pr_time_benchmarks* ]] 2024-08-20T22:16:05.4708305Z + [[ aot_eager_huggingface == *dynamo_eager* ]] 2024-08-20T22:16:05.4708847Z + [[ aot_eager_huggingface == *aot_eager* ]] 2024-08-20T22:16:05.4709621Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-08-20T22:16:05.4710103Z + [[ aot_eager_huggingface == *dynamic* ]] 2024-08-20T22:16:05.4710559Z + [[ aot_eager_huggingface == *cpu* ]] 2024-08-20T22:16:05.4711059Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:16:05.4741341Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:05.4742078Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:16:05.4744041Z + cd test 2024-08-20T22:16:05.4744834Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:16:06.9335624Z PyTorch built with: 2024-08-20T22:16:06.9336387Z - GCC 9.4 2024-08-20T22:16:06.9336883Z - C++ Version: 201703 2024-08-20T22:16:06.9337947Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:06.9339066Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:06.9339744Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:06.9340272Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:16:06.9340773Z - NNPACK is enabled 2024-08-20T22:16:06.9341164Z - CPU capability usage: AVX2 2024-08-20T22:16:06.9341582Z - CUDA Runtime 12.1 2024-08-20T22:16:06.9342108Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:16:06.9342723Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:16:06.9343179Z - Magma 2.6.1 2024-08-20T22:16:06.9351459Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:16:06.9357972Z 2024-08-20T22:16:07.2513076Z + cd test 2024-08-20T22:16:07.2513866Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:16:08.5870469Z ATen/Parallel: 2024-08-20T22:16:08.5870974Z at::get_num_threads() : 8 2024-08-20T22:16:08.5871508Z at::get_num_interop_threads() : 16 2024-08-20T22:16:08.5872067Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:08.5872565Z omp_get_max_threads() : 8 2024-08-20T22:16:08.5873501Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:08.5874300Z mkl_get_max_threads() : 8 2024-08-20T22:16:08.5874930Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:08.5875614Z std::thread::hardware_concurrency() : 16 2024-08-20T22:16:08.5876048Z Environment variables: 2024-08-20T22:16:08.5876395Z OMP_NUM_THREADS : [not set] 2024-08-20T22:16:08.5876769Z MKL_NUM_THREADS : [not set] 2024-08-20T22:16:08.5877145Z ATen parallel backend: OpenMP 2024-08-20T22:16:08.5877399Z 2024-08-20T22:16:08.8607512Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:08.8608428Z + [[ aot_eager_huggingface == *backward* ]] 2024-08-20T22:16:08.8609076Z + [[ aot_eager_huggingface == *xla* ]] 2024-08-20T22:16:08.8609683Z + [[ aot_eager_huggingface == *executorch* ]] 2024-08-20T22:16:08.8610223Z + [[ aot_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:16:08.8610915Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:08.8611472Z + [[ aot_eager_huggingface == distributed ]] 2024-08-20T22:16:08.8611989Z + [[ aot_eager_huggingface == *inductor_distributed* ]] 2024-08-20T22:16:08.8612671Z + [[ aot_eager_huggingface == *inductor-halide* ]] 2024-08-20T22:16:08.8613297Z + [[ aot_eager_huggingface == *inductor-micro-benchmark* ]] 2024-08-20T22:16:08.8614015Z + [[ aot_eager_huggingface == *huggingface* ]] 2024-08-20T22:16:08.8614605Z + install_torchvision 2024-08-20T22:16:08.8614981Z + local orig_preload 2024-08-20T22:16:08.8615304Z + local commit 2024-08-20T22:16:08.8615659Z ++ get_pinned_commit vision 2024-08-20T22:16:08.8616060Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:16:08.8637479Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:08.8638106Z + orig_preload= 2024-08-20T22:16:08.8638595Z + '[' -n '' ']' 2024-08-20T22:16:08.8639505Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:08.8640946Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:09.1998498Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:09.2003415Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-jb2_lu1v 2024-08-20T22:16:09.2034329Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-jb2_lu1v 2024-08-20T22:16:10.7309270Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:16:10.7338020Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:12.0579695Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:12.3525937Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:14.8435236Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:16:14.8467806Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:16:14.8471548Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:16:14.8475660Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:16:14.8547621Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:16:14.8550744Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:16:14.8553591Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:16:14.8556517Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:16:14.8559305Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:16:14.8564255Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:16:14.8578038Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:16:14.9056005Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:16:14.9118665Z Building wheels for collected packages: torchvision 2024-08-20T22:17:32.4619475Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:17:32.4621334Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2115987 sha256=a669eea271c7e4dae909ebe18525f9718d98b73ab254b0ab98906cba5d1e7b17 2024-08-20T22:17:32.4622977Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:17:32.4623840Z Successfully built torchvision 2024-08-20T22:17:32.9496373Z Installing collected packages: torchvision 2024-08-20T22:17:33.3666787Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:17:33.5145616Z + '[' -n '' ']' 2024-08-20T22:17:33.5145968Z + id=0 2024-08-20T22:17:33.5149379Z + test_dynamo_benchmark huggingface 0 2024-08-20T22:17:33.5150758Z ++ pwd 2024-08-20T22:17:33.5153186Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:33.5153734Z + local suite=huggingface 2024-08-20T22:17:33.5154076Z + shift 2024-08-20T22:17:33.5154349Z + local shard_id=0 2024-08-20T22:17:33.5154647Z + shift 2024-08-20T22:17:33.5154983Z + [[ aot_eager_huggingface == *perf_compare* ]] 2024-08-20T22:17:33.5157906Z + [[ aot_eager_huggingface == *perf* ]] 2024-08-20T22:17:33.5158507Z + [[ aot_eager_huggingface == *cpu* ]] 2024-08-20T22:17:33.5160200Z + [[ aot_eager_huggingface == *aot_inductor* ]] 2024-08-20T22:17:33.5161066Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-08-20T22:17:33.5161660Z ++ pwd 2024-08-20T22:17:33.5162447Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:33.5163125Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:33.5208679Z + local name=inference 2024-08-20T22:17:33.5209102Z + shift 2024-08-20T22:17:33.5209434Z + local suite=huggingface 2024-08-20T22:17:33.5209777Z + shift 2024-08-20T22:17:33.5210122Z + local shard_id=0 2024-08-20T22:17:33.5210500Z + shift 2024-08-20T22:17:33.5210781Z + partition_flags=() 2024-08-20T22:17:33.5211124Z + local partition_flags 2024-08-20T22:17:33.5211610Z + [[ -n 1 ]] 2024-08-20T22:17:33.5211936Z + [[ -n 0 ]] 2024-08-20T22:17:33.5212558Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:17:33.5213237Z + [[ aot_eager_huggingface == *perf_compare* ]] 2024-08-20T22:17:33.5213729Z + [[ aot_eager_huggingface == *perf* ]] 2024-08-20T22:17:33.5214205Z + [[ aot_eager_huggingface == *aot_inductor* ]] 2024-08-20T22:17:33.5214684Z + [[ aot_eager_huggingface == *_avx2* ]] 2024-08-20T22:17:33.5215139Z + [[ aot_eager_huggingface == *_avx512* ]] 2024-08-20T22:17:33.5218226Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-08-20T22:17:37.5909630Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:17:37.5911228Z warnings.warn( 2024-08-20T22:17:37.6507720Z 2024-08-20T22:17:37.6508000Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-08-20T22:35:45.0384244Z pass 2024-08-20T22:35:45.0886377Z TIMING: entire_frame_compile:10.76027 backend_compile:7.31225 total_wall_time:10.76027 2024-08-20T22:35:45.0887576Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:28532 | ProxyTorchDispatchMode.__torch_dispatch__:14646 | FakeTensor.__torch_dispatch__:518 2024-08-20T22:35:45.0888844Z Dynamo produced 2 graphs covering 533 ops with 5 graph breaks (4 unique) 2024-08-20T22:35:49.2595644Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:35:49.2597342Z warnings.warn( 2024-08-20T22:35:49.4744026Z 2024-08-20T22:35:52.4222077Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:35:52.4222950Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:35:52.4223479Z cuda train AllenaiLongformerBase 2024-08-20T22:36:00.1261768Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T22:36:00.1263381Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T22:36:00.1264701Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-20T22:36:00.1265922Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T22:36:00.1267104Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-20T22:36:00.1268100Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:36:00.1269280Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-20T22:36:00.1270968Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522 2024-08-20T22:36:00.1272526Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(**cloned_inputs) 2024-08-20T22:36:00.1274453Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-08-20T22:36:00.1276071Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.longformer( 2024-08-20T22:36:00.1277803Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-08-20T22:36:00.1279376Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] encoder_outputs = self.encoder( 2024-08-20T22:36:00.1281118Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward 2024-08-20T22:36:00.1282787Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] is_global_attn = is_index_global_attn.flatten().any().item() 2024-08-20T22:36:00.1283777Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:36:00.1284497Z W0820 22:36:00.125000 6177 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-20T22:36:33.4928020Z W0820 22:36:33.491000 6177 torch/_logging/_internal.py:1058] [16/0] Profiler function will be ignored 2024-08-20T22:36:52.2751178Z pass 2024-08-20T22:36:52.3397557Z TIMING: entire_frame_compile:52.25164 backend_compile:31.61425 total_wall_time:52.25164 2024-08-20T22:36:52.3400394Z STATS: call_* op count: 2366 | FakeTensorMode.__torch_dispatch__:91163 | ProxyTorchDispatchMode.__torch_dispatch__:48592 | FakeTensor.__torch_dispatch__:3764 2024-08-20T22:36:52.3401615Z Dynamo produced 7 graphs covering 2366 ops with 9 graph breaks (5 unique) 2024-08-20T22:36:57.3102169Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:36:57.3103935Z warnings.warn( 2024-08-20T22:36:57.4988174Z 2024-08-20T22:37:01.5144013Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:01.5144627Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:37:01.5147667Z cuda train BartForCausalLM 2024-08-20T22:37:11.6178042Z pass 2024-08-20T22:37:11.7660125Z TIMING: entire_frame_compile:2.35615 backend_compile:1.42839 total_wall_time:2.35615 2024-08-20T22:37:11.7662480Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:2599 | FakeTensor.__torch_dispatch__:65 | ProxyTorchDispatchMode.__torch_dispatch__:1122 2024-08-20T22:37:11.7664730Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique) 2024-08-20T22:37:15.6964626Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:15.6966308Z warnings.warn( 2024-08-20T22:37:15.8842892Z 2024-08-20T22:37:22.9548163Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:22.9548636Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:37:22.9549123Z cuda train BartForConditionalGeneration 2024-08-20T22:37:43.0283018Z pass 2024-08-20T22:37:43.2779912Z TIMING: entire_frame_compile:3.75369 backend_compile:2.238 total_wall_time:3.75369 2024-08-20T22:37:43.2781100Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:5785 | FakeTensor.__torch_dispatch__:150 | ProxyTorchDispatchMode.__torch_dispatch__:2721 2024-08-20T22:37:43.2782193Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique) 2024-08-20T22:37:47.2735532Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:37:47.2737151Z warnings.warn( 2024-08-20T22:37:47.4593555Z 2024-08-20T22:37:49.5675087Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:49.5675573Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:37:49.5676043Z cuda train BertForMaskedLM 2024-08-20T22:38:04.6901476Z W0820 22:38:04.689000 6463 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:38:22.7191873Z pass 2024-08-20T22:38:22.8416060Z TIMING: entire_frame_compile:30.03829 backend_compile:16.17252 total_wall_time:30.03829 2024-08-20T22:38:22.8417596Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:46009 | ProxyTorchDispatchMode.__torch_dispatch__:21485 | FakeTensor.__torch_dispatch__:3686 2024-08-20T22:38:22.8418738Z Dynamo produced 2 graphs covering 995 ops with 5 graph breaks (4 unique) 2024-08-20T22:38:27.3266395Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:38:27.3268029Z warnings.warn( 2024-08-20T22:38:27.5270657Z 2024-08-20T22:38:29.3919887Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:38:29.3920378Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:38:29.3920842Z cuda train BertForQuestionAnswering 2024-08-20T22:38:44.1071561Z W0820 22:38:44.106000 6559 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:39:01.8757027Z pass 2024-08-20T22:39:01.9945667Z TIMING: entire_frame_compile:29.7215 backend_compile:15.9866 total_wall_time:29.7215 2024-08-20T22:39:01.9948566Z STATS: call_* op count: 993 | FakeTensorMode.__torch_dispatch__:45714 | ProxyTorchDispatchMode.__torch_dispatch__:21302 | FakeTensor.__torch_dispatch__:3638 2024-08-20T22:39:01.9949718Z Dynamo produced 2 graphs covering 993 ops with 5 graph breaks (4 unique) 2024-08-20T22:39:06.4888244Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:06.4890206Z warnings.warn( 2024-08-20T22:39:06.6521113Z 2024-08-20T22:39:36.0715972Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:36.0718505Z loading model: 0it [00:29, ?it/s] 2024-08-20T22:39:36.0718995Z cuda train BlenderbotForCausalLM 2024-08-20T22:39:36.0723964Z Traceback (most recent call last): 2024-08-20T22:39:36.0724699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:39:36.0725463Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:39:36.0728293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:39:36.0729174Z pred = mod(**cloned_inputs) 2024-08-20T22:39:36.0730343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:39:36.0731274Z return self._call_impl(*args, **kwargs) 2024-08-20T22:39:36.0732219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:39:36.0733124Z return forward_call(*args, **kwargs) 2024-08-20T22:39:36.0734211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-08-20T22:39:36.0735164Z outputs = self.model.decoder( 2024-08-20T22:39:36.0736113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:39:36.0737118Z return self._call_impl(*args, **kwargs) 2024-08-20T22:39:36.0738275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:39:36.0739110Z return forward_call(*args, **kwargs) 2024-08-20T22:39:36.0740208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-08-20T22:39:36.0741172Z layer_outputs = decoder_layer( 2024-08-20T22:39:36.0742145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:39:36.0743096Z return self._call_impl(*args, **kwargs) 2024-08-20T22:39:36.0744134Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:39:36.0744970Z return forward_call(*args, **kwargs) 2024-08-20T22:39:36.0746039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-08-20T22:39:36.0747152Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-08-20T22:39:36.0748283Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:39:36.0749523Z return self._call_impl(*args, **kwargs) 2024-08-20T22:39:36.0750478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:39:36.0751472Z return forward_call(*args, **kwargs) 2024-08-20T22:39:36.0752544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-08-20T22:39:36.0753631Z query_states = self.q_proj(hidden_states) * self.scaling 2024-08-20T22:39:36.0754699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:39:36.0755595Z return self._call_impl(*args, **kwargs) 2024-08-20T22:39:36.0756540Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:39:36.0757377Z return forward_call(*args, **kwargs) 2024-08-20T22:39:36.0758273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-08-20T22:39:36.0759133Z return F.linear(input, self.weight, self.bias) 2024-08-20T22:39:36.0762492Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 230941 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:39:36.0765293Z 2024-08-20T22:39:36.0765592Z The above exception was the direct cause of the following exception: 2024-08-20T22:39:36.0766050Z 2024-08-20T22:39:36.0766197Z Traceback (most recent call last): 2024-08-20T22:39:36.0766858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:39:36.0767522Z ) = runner.load_model( 2024-08-20T22:39:36.0768185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 454, in load_model 2024-08-20T22:39:36.0768957Z self.validate_model(model, example_inputs) 2024-08-20T22:39:36.0769722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:39:36.0770497Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:39:36.0770968Z RuntimeError: Eager run failed 2024-08-20T22:39:36.0771226Z 2024-08-20T22:39:36.0771344Z eager_fail_to_run 2024-08-20T22:39:39.7575176Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:39.7577096Z warnings.warn( 2024-08-20T22:39:39.9523413Z 2024-08-20T22:39:41.5588180Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:41.5589059Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:39:41.5589526Z cuda train BlenderbotSmallForCausalLM 2024-08-20T22:39:41.5799690Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-08-20T22:39:44.8058446Z pass 2024-08-20T22:39:44.8341630Z TIMING: entire_frame_compile:2.50164 backend_compile:1.50757 total_wall_time:2.50164 2024-08-20T22:39:44.8344423Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:2991 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1286 2024-08-20T22:39:44.8346137Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:39:48.7298744Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:39:48.7300457Z warnings.warn( 2024-08-20T22:39:48.9291222Z 2024-08-20T22:39:50.8544755Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:50.8546579Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:39:50.8547273Z cuda train BlenderbotSmallForConditionalGeneration 2024-08-20T22:39:50.9020516Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:39:56.1897173Z pass 2024-08-20T22:39:56.2408125Z TIMING: entire_frame_compile:3.93364 backend_compile:2.20431 total_wall_time:3.93364 2024-08-20T22:39:56.2409466Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:6219 | ProxyTorchDispatchMode.__torch_dispatch__:2976 | FakeTensor.__torch_dispatch__:124 2024-08-20T22:39:56.2410572Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique) 2024-08-20T22:40:00.2289929Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:40:00.2291571Z warnings.warn( 2024-08-20T22:40:00.3984953Z 2024-08-20T22:40:02.5415795Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:40:02.5416267Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:40:02.5418382Z cuda train CamemBert 2024-08-20T22:40:17.7167320Z W0820 22:40:17.715000 6888 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:40:35.6163532Z pass 2024-08-20T22:40:35.7339885Z TIMING: entire_frame_compile:29.95542 backend_compile:16.13163 total_wall_time:29.95542 2024-08-20T22:40:35.7341114Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:46106 | ProxyTorchDispatchMode.__torch_dispatch__:21521 | FakeTensor.__torch_dispatch__:3686 2024-08-20T22:40:35.7342265Z Dynamo produced 2 graphs covering 1003 ops with 5 graph breaks (4 unique) 2024-08-20T22:40:40.1841518Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:40:40.1843218Z warnings.warn( 2024-08-20T22:40:40.3783815Z 2024-08-20T22:40:42.8971466Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:40:42.8971930Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:40:42.8972386Z cuda train DebertaForMaskedLM 2024-08-20T22:41:00.0282963Z W0820 22:41:00.027000 6984 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:41:14.9061802Z pass 2024-08-20T22:41:15.0180122Z TIMING: entire_frame_compile:28.62095 backend_compile:15.51476 total_wall_time:28.62095 2024-08-20T22:41:15.0181352Z STATS: call_* op count: 1318 | FakeTensorMode.__torch_dispatch__:48482 | ProxyTorchDispatchMode.__torch_dispatch__:23799 | FakeTensor.__torch_dispatch__:3020 2024-08-20T22:41:15.0182479Z Dynamo produced 2 graphs covering 1318 ops with 5 graph breaks (4 unique) 2024-08-20T22:41:19.4795516Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:41:19.4797220Z warnings.warn( 2024-08-20T22:41:19.7488374Z 2024-08-20T22:41:21.8954588Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:41:21.8955216Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:41:21.8955748Z cuda train DebertaForQuestionAnswering 2024-08-20T22:41:38.3662991Z W0820 22:41:38.365000 7079 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:41:52.9334290Z pass 2024-08-20T22:41:53.0582237Z TIMING: entire_frame_compile:28.21555 backend_compile:15.48984 total_wall_time:28.21555 2024-08-20T22:41:53.0585223Z STATS: call_* op count: 1316 | FakeTensorMode.__torch_dispatch__:48187 | ProxyTorchDispatchMode.__torch_dispatch__:23616 | FakeTensor.__torch_dispatch__:2976 2024-08-20T22:41:53.0586541Z Dynamo produced 2 graphs covering 1316 ops with 5 graph breaks (4 unique) 2024-08-20T22:41:57.5611591Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:41:57.5613309Z warnings.warn( 2024-08-20T22:41:57.7408869Z 2024-08-20T22:42:10.6415016Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:10.6415483Z loading model: 0it [00:12, ?it/s] 2024-08-20T22:42:10.6420348Z cuda train DebertaV2ForMaskedLM 2024-08-20T22:42:10.6421042Z pass_due_to_skip 2024-08-20T22:42:10.8673220Z TIMING: 2024-08-20T22:42:10.8674046Z STATS: call_* op count: 0 2024-08-20T22:42:10.8674807Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:42:14.5593415Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:42:14.5595034Z warnings.warn( 2024-08-20T22:42:14.7453650Z 2024-08-20T22:42:26.0674587Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:26.0675205Z loading model: 0it [00:11, ?it/s] 2024-08-20T22:42:26.0675807Z cuda train DebertaV2ForQuestionAnswering 2024-08-20T22:42:36.7191242Z ERROR:common: 2024-08-20T22:42:36.7191691Z Traceback (most recent call last): 2024-08-20T22:42:36.7192456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in check_accuracy 2024-08-20T22:42:36.7193214Z correct_result = self.run_n_iterations( 2024-08-20T22:42:36.7193992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-20T22:42:36.7198152Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-20T22:42:36.7199283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass 2024-08-20T22:42:36.7200118Z pred = mod(**cloned_inputs) 2024-08-20T22:42:36.7201328Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7202322Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7203416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7204544Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7205767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-08-20T22:42:36.7206783Z outputs = self.deberta( 2024-08-20T22:42:36.7207881Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7208846Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7209890Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7210891Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7212275Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-08-20T22:42:36.7213462Z encoder_outputs = self.encoder( 2024-08-20T22:42:36.7214449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7215625Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7216730Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7217885Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7219042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-08-20T22:42:36.7220073Z output_states = layer_module( 2024-08-20T22:42:36.7221187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7222159Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7223188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7224217Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7225449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-08-20T22:42:36.7226480Z attention_output = self.attention( 2024-08-20T22:42:36.7227539Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7228513Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7229583Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7230496Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7231711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-08-20T22:42:36.7232717Z self_output = self.self( 2024-08-20T22:42:36.7233736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:42:36.7234661Z return self._call_impl(*args, **kwargs) 2024-08-20T22:42:36.7235661Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:42:36.7236496Z return forward_call(*args, **kwargs) 2024-08-20T22:42:36.7237652Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-08-20T22:42:36.7238828Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-08-20T22:42:36.7239887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply 2024-08-20T22:42:36.7240831Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-08-20T22:42:36.7242027Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-08-20T22:42:36.7242996Z output = torch.softmax(output, self.dim) 2024-08-20T22:42:36.7246262Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 231628 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:42:36.7519140Z eager_1st_run_OOM 2024-08-20T22:42:36.7519455Z TIMING: 2024-08-20T22:42:36.7519742Z STATS: call_* op count: 0 2024-08-20T22:42:36.7520278Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T22:42:40.4500664Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:42:40.4502467Z warnings.warn( 2024-08-20T22:42:40.6909248Z 2024-08-20T22:42:42.1767209Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:42.1768290Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:42:42.1768882Z cuda train DistilBertForMaskedLM 2024-08-20T22:42:49.5121630Z W0820 22:42:49.511000 7259 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:42:58.8931737Z pass 2024-08-20T22:42:58.9580237Z TIMING: entire_frame_compile:15.55518 backend_compile:8.93405 total_wall_time:15.55518 2024-08-20T22:42:58.9581444Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:24058 | ProxyTorchDispatchMode.__torch_dispatch__:11147 | FakeTensor.__torch_dispatch__:1936 2024-08-20T22:42:58.9582561Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique) 2024-08-20T22:43:03.1376953Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:03.1378690Z warnings.warn( 2024-08-20T22:43:03.3106881Z 2024-08-20T22:43:04.5442884Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:04.5443757Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:43:04.5444679Z cuda train DistilBertForQuestionAnswering 2024-08-20T22:43:11.7366405Z W0820 22:43:11.735000 7355 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:43:20.8450048Z pass 2024-08-20T22:43:20.9083781Z TIMING: entire_frame_compile:15.234 backend_compile:8.53374 total_wall_time:15.234 2024-08-20T22:43:20.9084979Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:23806 | ProxyTorchDispatchMode.__torch_dispatch__:10978 | FakeTensor.__torch_dispatch__:1892 2024-08-20T22:43:20.9086119Z Dynamo produced 2 graphs covering 539 ops with 5 graph breaks (4 unique) 2024-08-20T22:43:25.0479418Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:25.0481081Z warnings.warn( 2024-08-20T22:43:25.2372786Z 2024-08-20T22:43:27.3093480Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:27.3093950Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:43:27.3094409Z cuda train DistillGPT2 2024-08-20T22:43:35.3199779Z W0820 22:43:35.319000 7450 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:43:42.1636611Z pass 2024-08-20T22:43:42.2261505Z TIMING: entire_frame_compile:12.83694 backend_compile:7.39427 total_wall_time:12.83694 2024-08-20T22:43:42.2262724Z STATS: call_* op count: 571 | FakeTensorMode.__torch_dispatch__:20519 | ProxyTorchDispatchMode.__torch_dispatch__:9304 | FakeTensor.__torch_dispatch__:1442 2024-08-20T22:43:42.2263932Z Dynamo produced 2 graphs covering 571 ops with 5 graph breaks (4 unique) 2024-08-20T22:43:46.3906636Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:43:46.3908280Z warnings.warn( 2024-08-20T22:43:46.5917919Z 2024-08-20T22:43:46.5927268Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T22:43:47.4096833Z 2024-08-20T22:43:47.4097268Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:47.4101524Z cuda train ElectraForCausalLM 2024-08-20T22:44:01.1121236Z pass 2024-08-20T22:44:01.1334725Z TIMING: entire_frame_compile:12.31106 backend_compile:6.73142 total_wall_time:12.31106 2024-08-20T22:44:01.1336119Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:27303 | ProxyTorchDispatchMode.__torch_dispatch__:12488 | FakeTensor.__torch_dispatch__:12 2024-08-20T22:44:01.1337523Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-08-20T22:44:05.1965881Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:05.1967563Z warnings.warn( 2024-08-20T22:44:05.4089487Z 2024-08-20T22:44:06.2145358Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:06.2145819Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:06.2146286Z cuda train ElectraForQuestionAnswering 2024-08-20T22:44:19.6325579Z W0820 22:44:19.631000 7641 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:44:37.4849999Z pass 2024-08-20T22:44:37.5039714Z TIMING: entire_frame_compile:29.91811 backend_compile:16.08063 total_wall_time:29.91811 2024-08-20T22:44:37.5040947Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:46099 | ProxyTorchDispatchMode.__torch_dispatch__:21467 | FakeTensor.__torch_dispatch__:3686 2024-08-20T22:44:37.5042081Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T22:44:42.0054272Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:44:42.0055949Z warnings.warn( 2024-08-20T22:44:42.2283427Z 2024-08-20T22:44:44.6621178Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:44.6621824Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:44:44.6626062Z cuda train GPT2ForSequenceClassification 2024-08-20T22:45:02.2522681Z W0820 22:45:02.251000 7737 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:45:15.8001344Z pass 2024-08-20T22:45:15.9204843Z TIMING: entire_frame_compile:26.65306 backend_compile:13.38719 total_wall_time:26.65306 2024-08-20T22:45:15.9206054Z STATS: call_* op count: 1101 | FakeTensorMode.__torch_dispatch__:42781 | ProxyTorchDispatchMode.__torch_dispatch__:18351 | FakeTensor.__torch_dispatch__:2810 2024-08-20T22:45:15.9207201Z Dynamo produced 5 graphs covering 1101 ops with 7 graph breaks (5 unique) 2024-08-20T22:45:20.3401221Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:20.3402856Z warnings.warn( 2024-08-20T22:45:20.5262535Z 2024-08-20T22:45:22.0322047Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:45:22.2520110Z 2024-08-20T22:45:22.2520694Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:45:22.2521235Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:45:22.2533432Z cuda train GoogleFnet 2024-08-20T22:45:23.5927266Z WARNING:common:Model GoogleFnet supports float32 only 2024-08-20T22:45:30.3155114Z W0820 22:45:30.314000 7833 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:45:39.7371404Z pass 2024-08-20T22:45:39.8003818Z TIMING: entire_frame_compile:15.47643 backend_compile:8.36368 total_wall_time:15.47643 2024-08-20T22:45:39.8005300Z STATS: call_* op count: 573 | FakeTensorMode.__torch_dispatch__:21478 | ProxyTorchDispatchMode.__torch_dispatch__:10146 | FakeTensor.__torch_dispatch__:1994 2024-08-20T22:45:39.8006435Z Dynamo produced 2 graphs covering 573 ops with 5 graph breaks (4 unique) 2024-08-20T22:45:43.9985762Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:45:43.9987385Z warnings.warn( 2024-08-20T22:45:44.2873847Z 2024-08-20T22:45:46.4831970Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:46.4832669Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:45:46.4833226Z cuda train LayoutLMForMaskedLM 2024-08-20T22:46:01.9433126Z W0820 22:46:01.942000 7929 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:46:20.2753883Z pass 2024-08-20T22:46:20.4024846Z TIMING: entire_frame_compile:30.65938 backend_compile:16.45876 total_wall_time:30.65938 2024-08-20T22:46:20.4026064Z STATS: call_* op count: 1033 | FakeTensorMode.__torch_dispatch__:47242 | ProxyTorchDispatchMode.__torch_dispatch__:21936 | FakeTensor.__torch_dispatch__:3762 2024-08-20T22:46:20.4027209Z Dynamo produced 2 graphs covering 1033 ops with 5 graph breaks (4 unique) 2024-08-20T22:46:24.9236190Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:46:24.9237839Z warnings.warn( 2024-08-20T22:46:25.1117772Z 2024-08-20T22:46:27.2993529Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:27.2994033Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:46:27.2995756Z cuda train LayoutLMForSequenceClassification 2024-08-20T22:46:44.7101007Z W0820 22:46:44.709000 8025 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:47:03.5258348Z pass 2024-08-20T22:47:03.6493588Z TIMING: entire_frame_compile:33.39789 backend_compile:16.80642 total_wall_time:33.39789 2024-08-20T22:47:03.6494830Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:50859 | ProxyTorchDispatchMode.__torch_dispatch__:22181 | FakeTensor.__torch_dispatch__:3776 2024-08-20T22:47:03.6495953Z Dynamo produced 5 graphs covering 1030 ops with 7 graph breaks (5 unique) 2024-08-20T22:47:08.1827491Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:47:08.1829222Z warnings.warn( 2024-08-20T22:47:08.3629099Z 2024-08-20T22:47:19.1477143Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:19.1477618Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:47:19.1482526Z cuda train M2M100ForConditionalGeneration 2024-08-20T22:47:19.2257619Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:47:50.6088164Z pass 2024-08-20T22:47:50.8277577Z TIMING: entire_frame_compile:29.14969 backend_compile:16.46639 total_wall_time:29.14969 2024-08-20T22:47:50.8280662Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:67674 | ProxyTorchDispatchMode.__torch_dispatch__:32498 | FakeTensor.__torch_dispatch__:114 2024-08-20T22:47:50.8281823Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-08-20T22:47:55.4158561Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:47:55.4160194Z warnings.warn( 2024-08-20T22:47:55.6023762Z 2024-08-20T22:47:59.6534379Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:59.6534992Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:47:59.6535974Z cuda train MBartForCausalLM 2024-08-20T22:47:59.6831676Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:48:03.4602728Z pass 2024-08-20T22:48:03.5472692Z TIMING: entire_frame_compile:2.55185 backend_compile:1.67666 total_wall_time:2.55185 2024-08-20T22:48:03.5473866Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3031 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1295 2024-08-20T22:48:03.5474964Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T22:48:07.5114124Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:48:07.5115740Z warnings.warn( 2024-08-20T22:48:07.7400298Z 2024-08-20T22:48:15.6156312Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:15.6156925Z loading model: 0it [00:07, ?it/s] 2024-08-20T22:48:15.6158969Z cuda train MBartForConditionalGeneration 2024-08-20T22:48:15.7153078Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:48:23.2231152Z pass 2024-08-20T22:48:23.4497707Z TIMING: entire_frame_compile:4.05783 backend_compile:2.4256 total_wall_time:4.05783 2024-08-20T22:48:23.4498938Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:6556 | FakeTensor.__torch_dispatch__:126 | ProxyTorchDispatchMode.__torch_dispatch__:3088 2024-08-20T22:48:23.4500039Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique) 2024-08-20T22:48:27.4323146Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:48:27.4324868Z warnings.warn( 2024-08-20T22:48:27.6065039Z 2024-08-20T22:48:31.8077185Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:31.8077762Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:48:31.8078328Z cuda train MT5ForConditionalGeneration 2024-08-20T22:48:56.5170627Z W0820 22:48:56.516000 8408 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:49:16.0181114Z pass 2024-08-20T22:49:16.3307162Z TIMING: entire_frame_compile:40.95834 backend_compile:22.99261 total_wall_time:40.95834 2024-08-20T22:49:16.3310255Z STATS: call_* op count: 1763 | FakeTensorMode.__torch_dispatch__:73137 | ProxyTorchDispatchMode.__torch_dispatch__:36442 | FakeTensor.__torch_dispatch__:3534 2024-08-20T22:49:16.3311428Z Dynamo produced 2 graphs covering 1763 ops with 5 graph breaks (4 unique) 2024-08-20T22:49:21.1174969Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:49:21.1176688Z warnings.warn( 2024-08-20T22:49:21.3103567Z 2024-08-20T22:49:21.3142342Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-08-20T22:49:26.1368548Z 2024-08-20T22:49:26.1369273Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:49:26.1374214Z cuda train MegatronBertForCausalLM 2024-08-20T22:49:57.0489201Z W0820 22:49:57.048000 8504 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:50:32.8991280Z pass 2024-08-20T22:50:33.2557783Z TIMING: entire_frame_compile:58.7215 backend_compile:31.33805 total_wall_time:58.7215 2024-08-20T22:50:33.2559065Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:89495 | ProxyTorchDispatchMode.__torch_dispatch__:42045 | FakeTensor.__torch_dispatch__:7142 2024-08-20T22:50:33.2563415Z Dynamo produced 2 graphs covering 1922 ops with 5 graph breaks (4 unique) 2024-08-20T22:50:38.3765474Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:50:38.3768752Z warnings.warn( 2024-08-20T22:50:38.5583484Z 2024-08-20T22:50:43.0776430Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:50:43.0777004Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:50:43.0777471Z cuda train MegatronBertForQuestionAnswering 2024-08-20T22:51:13.5916104Z W0820 22:51:13.590000 8600 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:51:49.3094794Z pass 2024-08-20T22:51:49.6567757Z TIMING: entire_frame_compile:58.59006 backend_compile:31.39949 total_wall_time:58.59006 2024-08-20T22:51:49.6570196Z STATS: call_* op count: 1916 | FakeTensorMode.__torch_dispatch__:89092 | ProxyTorchDispatchMode.__torch_dispatch__:41827 | FakeTensor.__torch_dispatch__:7094 2024-08-20T22:51:49.6571349Z Dynamo produced 2 graphs covering 1916 ops with 5 graph breaks (4 unique) 2024-08-20T22:51:54.8143765Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:51:54.8145384Z warnings.warn( 2024-08-20T22:51:55.0021413Z 2024-08-20T22:51:56.4711775Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:56.4712525Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:51:56.4713203Z cuda train MobileBertForMaskedLM 2024-08-20T22:52:55.1493558Z pass 2024-08-20T22:52:55.2003831Z TIMING: entire_frame_compile:54.89447 backend_compile:27.51791 total_wall_time:54.89447 2024-08-20T22:52:55.2006855Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:109311 | ProxyTorchDispatchMode.__torch_dispatch__:55808 | FakeTensor.__torch_dispatch__:14 2024-08-20T22:52:55.2008023Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-08-20T22:53:00.3007706Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:53:00.3009460Z warnings.warn( 2024-08-20T22:53:00.5137107Z 2024-08-20T22:53:01.8067716Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:01.8068408Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:53:01.8069181Z cuda train MobileBertForQuestionAnswering 2024-08-20T22:54:00.4282754Z pass 2024-08-20T22:54:00.4642635Z TIMING: entire_frame_compile:54.84468 backend_compile:27.61506 total_wall_time:54.84468 2024-08-20T22:54:00.4645591Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:109224 | ProxyTorchDispatchMode.__torch_dispatch__:55709 | FakeTensor.__torch_dispatch__:18 2024-08-20T22:54:00.4646749Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-08-20T22:54:05.5611841Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:05.5613473Z warnings.warn( 2024-08-20T22:54:05.7553838Z 2024-08-20T22:54:08.6338075Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:08.6339082Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:54:08.6339553Z cuda train OPTForCausalLM 2024-08-20T22:54:08.6535179Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-08-20T22:54:13.3209425Z pass 2024-08-20T22:54:13.3683162Z TIMING: entire_frame_compile:2.64467 backend_compile:1.76063 total_wall_time:2.64467 2024-08-20T22:54:13.3684480Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:3327 | FakeTensor.__torch_dispatch__:56 | ProxyTorchDispatchMode.__torch_dispatch__:1405 2024-08-20T22:54:13.3685584Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique) 2024-08-20T22:54:17.3586458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:17.3588128Z warnings.warn( 2024-08-20T22:54:17.5390307Z 2024-08-20T22:54:19.6518079Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:19.6518551Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:54:19.6519034Z cuda train PLBartForCausalLM 2024-08-20T22:54:19.6685383Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-08-20T22:54:22.7373919Z pass 2024-08-20T22:54:22.7816758Z TIMING: entire_frame_compile:2.37855 backend_compile:1.52102 total_wall_time:2.37855 2024-08-20T22:54:22.7817938Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3067 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1304 2024-08-20T22:54:22.7819030Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T22:54:26.7385847Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:26.7387496Z warnings.warn( 2024-08-20T22:54:26.9390789Z 2024-08-20T22:54:30.4181232Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:30.4182052Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:54:30.4182584Z cuda train PLBartForConditionalGeneration 2024-08-20T22:54:30.4719389Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:54:36.0566205Z pass 2024-08-20T22:54:36.1274080Z TIMING: entire_frame_compile:4.05151 backend_compile:2.45287 total_wall_time:4.05151 2024-08-20T22:54:36.1275262Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:6716 | FakeTensor.__torch_dispatch__:134 | ProxyTorchDispatchMode.__torch_dispatch__:3161 2024-08-20T22:54:36.1276374Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique) 2024-08-20T22:54:40.1623313Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:40.1626057Z warnings.warn( 2024-08-20T22:54:40.3468769Z 2024-08-20T22:54:46.5295641Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:46.5296127Z loading model: 0it [00:06, ?it/s] 2024-08-20T22:54:46.5296603Z cuda train PegasusForCausalLM 2024-08-20T22:54:46.5597552Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-08-20T22:54:50.0955441Z pass 2024-08-20T22:54:50.1816215Z TIMING: entire_frame_compile:2.50139 backend_compile:1.66836 total_wall_time:2.50139 2024-08-20T22:54:50.1817379Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:2936 | FakeTensor.__torch_dispatch__:56 | ProxyTorchDispatchMode.__torch_dispatch__:1276 2024-08-20T22:54:50.1820677Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-08-20T22:54:54.1160799Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:54:54.1162831Z warnings.warn( 2024-08-20T22:54:54.2985307Z 2024-08-20T22:55:04.6250381Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:04.6250989Z loading model: 0it [00:10, ?it/s] 2024-08-20T22:55:04.6251629Z cuda train PegasusForConditionalGeneration 2024-08-20T22:55:04.6999482Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-08-20T22:55:10.8501378Z pass 2024-08-20T22:55:11.0563363Z TIMING: entire_frame_compile:3.91461 backend_compile:2.3775 total_wall_time:3.91461 2024-08-20T22:55:11.0564763Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:6137 | ProxyTorchDispatchMode.__torch_dispatch__:2966 | FakeTensor.__torch_dispatch__:122 2024-08-20T22:55:11.0565883Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique) 2024-08-20T22:55:15.1000696Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:15.1002325Z warnings.warn( 2024-08-20T22:55:15.2811726Z 2024-08-20T22:55:15.2840745Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-08-20T22:55:17.4931958Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:55:17.4933916Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:55:17.7775638Z 2024-08-20T22:55:17.7776188Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:55:17.7776692Z cuda train RobertaForCausalLM 2024-08-20T22:55:33.2382229Z W0820 22:55:33.237000 9368 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:55:51.2593312Z pass 2024-08-20T22:55:51.3697848Z TIMING: entire_frame_compile:30.12256 backend_compile:16.26098 total_wall_time:30.12256 2024-08-20T22:55:51.3700319Z STATS: call_* op count: 1007 | FakeTensorMode.__torch_dispatch__:46201 | ProxyTorchDispatchMode.__torch_dispatch__:21552 | FakeTensor.__torch_dispatch__:3686 2024-08-20T22:55:51.3701475Z Dynamo produced 2 graphs covering 1007 ops with 5 graph breaks (4 unique) 2024-08-20T22:55:55.8573188Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:55:55.8574845Z warnings.warn( 2024-08-20T22:55:56.2432295Z 2024-08-20T22:55:57.9729438Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-08-20T22:55:57.9731575Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-08-20T22:55:58.2862601Z 2024-08-20T22:55:58.2863310Z loading model: 0it [00:02, ?it/s] 2024-08-20T22:55:58.2864062Z cuda train RobertaForQuestionAnswering 2024-08-20T22:56:13.1074664Z W0820 22:56:13.106000 9464 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:56:30.9999344Z pass 2024-08-20T22:56:31.1279198Z TIMING: entire_frame_compile:29.90373 backend_compile:16.13306 total_wall_time:29.90373 2024-08-20T22:56:31.1280852Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:45794 | ProxyTorchDispatchMode.__torch_dispatch__:21334 | FakeTensor.__torch_dispatch__:3638 2024-08-20T22:56:31.1284504Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique) 2024-08-20T22:56:35.6061815Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:56:35.6063540Z warnings.warn( 2024-08-20T22:56:35.8544294Z 2024-08-20T22:56:36.4972981Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:36.4973585Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:36.4974054Z cuda train Speech2Text2ForCausalLM 2024-08-20T22:56:36.6060188Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-08-20T22:56:39.6587774Z pass 2024-08-20T22:56:39.6644499Z TIMING: entire_frame_compile:2.52421 backend_compile:1.5283 total_wall_time:2.52421 2024-08-20T22:56:39.6645912Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:3128 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1342 2024-08-20T22:56:39.6647013Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique) 2024-08-20T22:56:43.6392841Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:56:43.6394588Z warnings.warn( 2024-08-20T22:56:43.9276154Z 2024-08-20T22:56:45.4629793Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:56:45.4630386Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:56:45.4630994Z cuda train T5ForConditionalGeneration 2024-08-20T22:57:04.9699934Z W0820 22:57:04.969000 9656 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:57:18.0642936Z pass 2024-08-20T22:57:18.1268214Z TIMING: entire_frame_compile:28.57086 backend_compile:16.22768 total_wall_time:28.57086 2024-08-20T22:57:18.1270055Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:50300 | ProxyTorchDispatchMode.__torch_dispatch__:24715 | FakeTensor.__torch_dispatch__:2452 2024-08-20T22:57:18.1271775Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T22:57:22.6608673Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:57:22.6610459Z warnings.warn( 2024-08-20T22:57:22.8490350Z 2024-08-20T22:57:24.3856983Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:57:24.3857699Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:57:24.3860379Z cuda train T5Small 2024-08-20T22:57:43.8960958Z W0820 22:57:43.895000 9752 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:57:56.8904476Z pass 2024-08-20T22:57:56.9530811Z TIMING: entire_frame_compile:28.48083 backend_compile:16.15475 total_wall_time:28.48083 2024-08-20T22:57:56.9533229Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:50300 | ProxyTorchDispatchMode.__torch_dispatch__:24715 | FakeTensor.__torch_dispatch__:2452 2024-08-20T22:57:56.9535927Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique) 2024-08-20T22:58:01.4501413Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:01.4503523Z warnings.warn( 2024-08-20T22:58:01.6273157Z 2024-08-20T22:58:05.6544044Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:05.6544530Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:58:05.6548302Z cuda train TrOCRForCausalLM 2024-08-20T22:58:05.6854897Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-08-20T22:58:09.2995642Z pass 2024-08-20T22:58:09.3725951Z TIMING: entire_frame_compile:2.56349 backend_compile:1.68275 total_wall_time:2.56349 2024-08-20T22:58:09.3728313Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:3025 | ProxyTorchDispatchMode.__torch_dispatch__:1295 | FakeTensor.__torch_dispatch__:58 2024-08-20T22:58:09.3730516Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-08-20T22:58:13.3236866Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:13.3238638Z warnings.warn( 2024-08-20T22:58:13.5151254Z 2024-08-20T22:58:23.4488436Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:23.4489164Z loading model: 0it [00:09, ?it/s] 2024-08-20T22:58:23.4489623Z cuda train XGLMForCausalLM 2024-08-20T22:58:23.5007910Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-08-20T22:58:28.1404306Z pass 2024-08-20T22:58:28.3065546Z TIMING: entire_frame_compile:2.6466 backend_compile:1.75994 total_wall_time:2.6466 2024-08-20T22:58:28.3067006Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:3219 | FakeTensor.__torch_dispatch__:61 | ProxyTorchDispatchMode.__torch_dispatch__:1397 2024-08-20T22:58:28.3068120Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique) 2024-08-20T22:58:32.2830532Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T22:58:32.2832165Z warnings.warn( 2024-08-20T22:58:32.4577801Z 2024-08-20T22:58:37.1237765Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:37.1238249Z loading model: 0it [00:04, ?it/s] 2024-08-20T22:58:37.1238724Z cuda train XLNetLMHeadModel 2024-08-20T22:59:19.9856266Z W0820 22:59:19.984000 10040 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T22:59:58.2843856Z pass 2024-08-20T22:59:58.6616878Z TIMING: entire_frame_compile:71.31465 backend_compile:39.27611 total_wall_time:71.31465 2024-08-20T22:59:58.6618091Z STATS: call_* op count: 1873 | FakeTensorMode.__torch_dispatch__:120728 | ProxyTorchDispatchMode.__torch_dispatch__:55611 | FakeTensor.__torch_dispatch__:6562 2024-08-20T22:59:58.6622881Z Dynamo produced 2 graphs covering 1873 ops with 5 graph breaks (4 unique) 2024-08-20T23:00:04.1568136Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-08-20T23:00:04.1571332Z warnings.warn( 2024-08-20T23:00:04.3415872Z 2024-08-20T23:00:06.6586702Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:00:06.6587204Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:00:06.6588239Z cuda train YituTechConvBert 2024-08-20T23:00:28.1166933Z W0820 23:00:28.115000 10136 torch/_logging/_internal.py:1058] [4/0] Profiler function will be ignored 2024-08-20T23:00:55.1387300Z pass 2024-08-20T23:00:55.2374381Z TIMING: entire_frame_compile:45.0545 backend_compile:24.51105 total_wall_time:45.0545 2024-08-20T23:00:55.2377315Z STATS: call_* op count: 1511 | FakeTensorMode.__torch_dispatch__:67175 | ProxyTorchDispatchMode.__torch_dispatch__:31640 | FakeTensor.__torch_dispatch__:5226 2024-08-20T23:00:55.2378465Z Dynamo produced 2 graphs covering 1511 ops with 5 graph breaks (4 unique) 2024-08-20T23:00:57.1627186Z accuracy pass_rate=93.48% 2024-08-20T23:00:57.1631945Z calls_captured gmean=0.00x mean=750.522x 2024-08-20T23:00:57.1635321Z unique_graphs gmean=0.00x mean=3.348x 2024-08-20T23:00:57.1639143Z graph_breaks gmean=0.00x mean=5.196x 2024-08-20T23:00:57.1643407Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-08-20T23:00:57.1647246Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:00:57.1651079Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:00:57.1654753Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:00:57.7986950Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_huggingface_training.csv 2024-08-20T23:00:58.0638261Z AlbertForMaskedLM PASS 2024-08-20T23:00:58.0641270Z AlbertForQuestionAnswering PASS 2024-08-20T23:00:58.0646316Z AllenaiLongformerBase PASS 2024-08-20T23:00:58.0651244Z BartForCausalLM PASS 2024-08-20T23:00:58.0656200Z BartForConditionalGeneration PASS 2024-08-20T23:00:58.0661239Z BertForMaskedLM PASS 2024-08-20T23:00:58.0665992Z BertForQuestionAnswering PASS 2024-08-20T23:00:58.0671161Z BlenderbotForCausalLM XFAIL 2024-08-20T23:00:58.0676241Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:00:58.0681188Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:00:58.0686062Z CamemBert PASS 2024-08-20T23:00:58.0691383Z DebertaForMaskedLM PASS 2024-08-20T23:00:58.0696541Z DebertaForQuestionAnswering PASS 2024-08-20T23:00:58.0701492Z DebertaV2ForMaskedLM XFAIL 2024-08-20T23:00:58.0706920Z DebertaV2ForQuestionAnswering XFAIL 2024-08-20T23:00:58.0711733Z DistilBertForMaskedLM PASS 2024-08-20T23:00:58.0716907Z DistilBertForQuestionAnswering PASS 2024-08-20T23:00:58.0721750Z DistillGPT2 PASS 2024-08-20T23:00:58.0726718Z ElectraForCausalLM PASS 2024-08-20T23:00:58.0731782Z ElectraForQuestionAnswering PASS 2024-08-20T23:00:58.0736624Z GPT2ForSequenceClassification PASS 2024-08-20T23:00:58.0741427Z GoogleFnet PASS 2024-08-20T23:00:58.0746677Z LayoutLMForMaskedLM PASS 2024-08-20T23:00:58.0751642Z LayoutLMForSequenceClassification PASS 2024-08-20T23:00:58.0756537Z M2M100ForConditionalGeneration PASS 2024-08-20T23:00:58.0761311Z MBartForCausalLM PASS 2024-08-20T23:00:58.0766348Z MBartForConditionalGeneration PASS 2024-08-20T23:00:58.0771200Z MT5ForConditionalGeneration PASS 2024-08-20T23:00:58.0776218Z MegatronBertForCausalLM PASS 2024-08-20T23:00:58.0781104Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:00:58.0786358Z MobileBertForMaskedLM PASS 2024-08-20T23:00:58.0791839Z MobileBertForQuestionAnswering PASS 2024-08-20T23:00:58.0797042Z OPTForCausalLM PASS 2024-08-20T23:00:58.0801809Z PLBartForCausalLM PASS 2024-08-20T23:00:58.0806883Z PLBartForConditionalGeneration PASS 2024-08-20T23:00:58.0811671Z PegasusForCausalLM PASS 2024-08-20T23:00:58.0816761Z PegasusForConditionalGeneration PASS 2024-08-20T23:00:58.0821815Z RobertaForCausalLM PASS 2024-08-20T23:00:58.0826785Z RobertaForQuestionAnswering PASS 2024-08-20T23:00:58.0831564Z Speech2Text2ForCausalLM PASS 2024-08-20T23:00:58.0836582Z T5ForConditionalGeneration PASS 2024-08-20T23:00:58.0841374Z T5Small PASS 2024-08-20T23:00:58.0846356Z TrOCRForCausalLM PASS 2024-08-20T23:00:58.0851230Z XGLMForCausalLM PASS 2024-08-20T23:00:58.0856247Z XLNetLMHeadModel PASS 2024-08-20T23:00:58.0861140Z YituTechConvBert PASS 2024-08-20T23:00:58.1325483Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_huggingface_training.csv 2024-08-20T23:00:58.3990020Z AlbertForMaskedLM PASS 2024-08-20T23:00:58.3994382Z AlbertForQuestionAnswering PASS 2024-08-20T23:00:58.3999542Z AllenaiLongformerBase PASS 2024-08-20T23:00:58.4005014Z BartForCausalLM PASS 2024-08-20T23:00:58.4009996Z BartForConditionalGeneration PASS 2024-08-20T23:00:58.4015122Z BertForMaskedLM PASS 2024-08-20T23:00:58.4020211Z BertForQuestionAnswering PASS 2024-08-20T23:00:58.4025222Z BlenderbotForCausalLM PASS 2024-08-20T23:00:58.4030831Z BlenderbotSmallForCausalLM PASS 2024-08-20T23:00:58.4035948Z BlenderbotSmallForConditionalGeneration PASS 2024-08-20T23:00:58.4041171Z CamemBert PASS 2024-08-20T23:00:58.4046336Z DebertaForMaskedLM PASS 2024-08-20T23:00:58.4051641Z DebertaForQuestionAnswering PASS 2024-08-20T23:00:58.4056666Z DebertaV2ForMaskedLM PASS 2024-08-20T23:00:58.4061933Z DebertaV2ForQuestionAnswering PASS 2024-08-20T23:00:58.4067268Z DistilBertForMaskedLM PASS 2024-08-20T23:00:58.4072489Z DistilBertForQuestionAnswering PASS 2024-08-20T23:00:58.4077563Z DistillGPT2 PASS 2024-08-20T23:00:58.4082819Z ElectraForCausalLM PASS 2024-08-20T23:00:58.4088067Z ElectraForQuestionAnswering PASS 2024-08-20T23:00:58.4094009Z GPT2ForSequenceClassification PASS 2024-08-20T23:00:58.4099194Z GoogleFnet PASS 2024-08-20T23:00:58.4104119Z LayoutLMForMaskedLM PASS 2024-08-20T23:00:58.4109393Z LayoutLMForSequenceClassification PASS 2024-08-20T23:00:58.4114557Z M2M100ForConditionalGeneration PASS 2024-08-20T23:00:58.4119641Z MBartForCausalLM PASS 2024-08-20T23:00:58.4124868Z MBartForConditionalGeneration PASS 2024-08-20T23:00:58.4129977Z MT5ForConditionalGeneration PASS 2024-08-20T23:00:58.4135161Z MegatronBertForCausalLM PASS 2024-08-20T23:00:58.4140231Z MegatronBertForQuestionAnswering PASS 2024-08-20T23:00:58.4145551Z MobileBertForMaskedLM PASS 2024-08-20T23:00:58.4150725Z MobileBertForQuestionAnswering PASS 2024-08-20T23:00:58.4155952Z OPTForCausalLM PASS 2024-08-20T23:00:58.4161146Z PLBartForCausalLM PASS 2024-08-20T23:00:58.4166350Z PLBartForConditionalGeneration PASS 2024-08-20T23:00:58.4171417Z PegasusForCausalLM PASS 2024-08-20T23:00:58.4176703Z PegasusForConditionalGeneration PASS 2024-08-20T23:00:58.4181765Z RobertaForCausalLM PASS 2024-08-20T23:00:58.4187317Z RobertaForQuestionAnswering PASS 2024-08-20T23:00:58.4192810Z Speech2Text2ForCausalLM PASS 2024-08-20T23:00:58.4198099Z T5ForConditionalGeneration PASS 2024-08-20T23:00:58.4203147Z T5Small PASS 2024-08-20T23:00:58.4208368Z TrOCRForCausalLM PASS 2024-08-20T23:00:58.4213460Z XGLMForCausalLM PASS 2024-08-20T23:00:58.4218712Z XLNetLMHeadModel PASS 2024-08-20T23:00:58.4223955Z YituTechConvBert PASS 2024-08-20T23:00:58.4693053Z + cleanup_workspace 2024-08-20T23:00:58.4695963Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:00:58.4697195Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:00:58.4698132Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:00:58.4698770Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:00:58.4699570Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:00:58.4700503Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:00:58.4701206Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:00:59.5970604Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:00:59.5971132Z cat test/**/*_toprint.log || true 2024-08-20T23:00:59.5984935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:00:59.5985442Z env: 2024-08-20T23:00:59.5985723Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:00:59.5986179Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:00:59.5986954Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:00:59.5987616Z ##[endgroup] 2024-08-20T23:00:59.6083837Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:00:59.6116701Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:00:59.6117221Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:00:59.6125337Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:00:59.6125835Z env: 2024-08-20T23:00:59.6126116Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:00:59.6126568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:00:59.6127344Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:00:59.6127995Z MONITOR_SCRIPT_PID: 223102 2024-08-20T23:00:59.6128353Z ##[endgroup] 2024-08-20T23:00:59.6323462Z Prepare all required actions 2024-08-20T23:00:59.6323896Z Getting action download info 2024-08-20T23:00:59.8506333Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:00:59.9949698Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:00:59.9950181Z with: 2024-08-20T23:00:59.9950719Z file-suffix: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278 2024-08-20T23:00:59.9951394Z s3-bucket: gha-artifacts 2024-08-20T23:00:59.9951728Z env: 2024-08-20T23:00:59.9952005Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:00:59.9952455Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:00:59.9953189Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:00:59.9953823Z ##[endgroup] 2024-08-20T23:00:59.9979973Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:00:59.9980613Z # Remove any previous test jsons if they exist 2024-08-20T23:00:59.9981121Z rm -f test-jsons-*.zip 2024-08-20T23:00:59.9981635Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:00:59.9990918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:00:59.9991417Z env: 2024-08-20T23:00:59.9991694Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:00:59.9992138Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:00:59.9992864Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:00:59.9993756Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278 2024-08-20T23:00:59.9994401Z ##[endgroup] 2024-08-20T23:01:00.0276941Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:01:00.0306658Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:01:00.0307349Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:01:00.0313538Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:01:00.0319154Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:01:00.0352164Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:01:00.0352975Z # Remove any previous test reports if they exist 2024-08-20T23:01:00.0353500Z rm -f test-reports-*.zip 2024-08-20T23:01:00.0354083Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:01:00.0362953Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:00.0363448Z env: 2024-08-20T23:01:00.0363731Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:00.0364183Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:00.0364918Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:00.0365811Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278 2024-08-20T23:01:00.0366467Z ##[endgroup] 2024-08-20T23:01:00.0632976Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-08-20T23:01:00.0634215Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%) 2024-08-20T23:01:00.0635598Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%) 2024-08-20T23:01:00.0636498Z adding: test/test-reports/training_huggingface.csv (deflated 74%) 2024-08-20T23:01:00.0642247Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-08-20T23:01:00.0643466Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%) 2024-08-20T23:01:00.0676845Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:01:00.0677484Z # Remove any previous usage logs if they exist 2024-08-20T23:01:00.0677987Z rm -f logs-*.zip 2024-08-20T23:01:00.0678869Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:01:00.0679675Z # so check to see if the file exists first 2024-08-20T23:01:00.0680177Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:01:00.0680727Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:01:00.0681214Z fi 2024-08-20T23:01:00.0681577Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:01:00.0682146Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:01:00.0682638Z fi 2024-08-20T23:01:00.0691844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:00.0692334Z env: 2024-08-20T23:01:00.0692613Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:00.0693067Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:00.0693798Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:00.0694693Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278 2024-08-20T23:01:00.0695357Z ##[endgroup] 2024-08-20T23:01:00.0772088Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:01:00.0861217Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:01:00.0861945Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:01:00.0862489Z rm -f debug-*.zip 2024-08-20T23:01:00.0862868Z if [ -d 'test/debug' ]; then 2024-08-20T23:01:00.0863366Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:01:00.0863948Z fi 2024-08-20T23:01:00.0872094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:00.0872589Z env: 2024-08-20T23:01:00.0872870Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:00.0873319Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:00.0874049Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:00.0874955Z FILE_SUFFIX: test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278 2024-08-20T23:01:00.0875601Z ##[endgroup] 2024-08-20T23:01:00.0970369Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:01:00.0970805Z with: 2024-08-20T23:01:00.0971260Z s3-bucket: gha-artifacts 2024-08-20T23:01:00.0971697Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:00.0972168Z retention-days: 14 2024-08-20T23:01:00.0972505Z if-no-files-found: warn 2024-08-20T23:01:00.0972861Z path: test-jsons-*.zip 2024-08-20T23:01:00.0973204Z name: artifact 2024-08-20T23:01:00.0973503Z region: us-east-1 2024-08-20T23:01:00.0973805Z env: 2024-08-20T23:01:00.0974074Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:00.0974517Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:00.0975237Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:00.0975872Z ##[endgroup] 2024-08-20T23:01:00.4517848Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:01:00.4518609Z With the provided path, there will be 1 file uploaded 2024-08-20T23:01:00.4519229Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:00.4577706Z Starting upload of test-jsons-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:00.5552407Z Finished upload of test-jsons-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:00.5839296Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:01:00.5839736Z with: 2024-08-20T23:01:00.5840024Z s3-bucket: gha-artifacts 2024-08-20T23:01:00.5840466Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:00.5840937Z retention-days: 14 2024-08-20T23:01:00.5841276Z if-no-files-found: error 2024-08-20T23:01:00.5841647Z path: test-reports-*.zip 2024-08-20T23:01:00.5842001Z name: artifact 2024-08-20T23:01:00.5842304Z region: us-east-1 2024-08-20T23:01:00.5842606Z env: 2024-08-20T23:01:00.5842881Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:00.5843532Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:00.5844264Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:00.5844907Z ##[endgroup] 2024-08-20T23:01:00.9382111Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:01:00.9382712Z With the provided path, there will be 1 file uploaded 2024-08-20T23:01:00.9383492Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:00.9439240Z Starting upload of test-reports-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:01.1573785Z Finished upload of test-reports-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:01.1859519Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:01:01.1859968Z with: 2024-08-20T23:01:01.1860260Z s3-bucket: gha-artifacts 2024-08-20T23:01:01.1860704Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:01.1861197Z retention-days: 14 2024-08-20T23:01:01.1861532Z if-no-files-found: ignore 2024-08-20T23:01:01.1861898Z path: logs-*.zip 2024-08-20T23:01:01.1862213Z name: artifact 2024-08-20T23:01:01.1862528Z region: us-east-1 2024-08-20T23:01:01.1862833Z env: 2024-08-20T23:01:01.1863115Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:01.1863570Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:01.1864413Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:01.1865056Z ##[endgroup] 2024-08-20T23:01:01.5381272Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:01:01.5381872Z With the provided path, there will be 1 file uploaded 2024-08-20T23:01:01.5382488Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:01.5438396Z Starting upload of logs-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:01.7019088Z Finished upload of logs-test-aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026299278.zip 2024-08-20T23:01:01.7306036Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:01:01.7306479Z with: 2024-08-20T23:01:01.7306768Z s3-bucket: gha-artifacts 2024-08-20T23:01:01.7307432Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:01:01.7307924Z retention-days: 14 2024-08-20T23:01:01.7308265Z if-no-files-found: ignore 2024-08-20T23:01:01.7308635Z path: debug-*.zip 2024-08-20T23:01:01.7308955Z name: artifact 2024-08-20T23:01:01.7309259Z region: us-east-1 2024-08-20T23:01:01.7309563Z env: 2024-08-20T23:01:01.7309840Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:01.7310290Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:01.7311016Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:01.7311652Z ##[endgroup] 2024-08-20T23:01:02.0784793Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:01:02.1067811Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:01:02.1068301Z # shellcheck disable=SC2156 2024-08-20T23:01:02.1069095Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:01:02.1078533Z shell: /usr/bin/bash -e {0} 2024-08-20T23:01:02.1078888Z env: 2024-08-20T23:01:02.1079168Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:02.1079622Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:02.1080351Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:02.1080989Z ##[endgroup] 2024-08-20T23:01:02.3697312Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:01:02.3697932Z with: 2024-08-20T23:01:02.3698201Z env: 2024-08-20T23:01:02.3698478Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:02.3698929Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:02.3699670Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:02.3700321Z ##[endgroup] 2024-08-20T23:01:02.3720818Z ##[group]Run set -eou pipefail 2024-08-20T23:01:02.3721223Z set -eou pipefail 2024-08-20T23:01:02.3721590Z  2024-08-20T23:01:02.3722101Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:01:02.3722730Z for _ in $(seq 1440); do 2024-08-20T23:01:02.3723206Z  # Break if no ssh session exists anymore 2024-08-20T23:01:02.3723703Z  if [ "$(who)" = "" ]; then 2024-08-20T23:01:02.3724110Z  break 2024-08-20T23:01:02.3724449Z  fi 2024-08-20T23:01:02.3724749Z  echo "." 2024-08-20T23:01:02.3725081Z  sleep 5 2024-08-20T23:01:02.3725400Z done 2024-08-20T23:01:02.3734128Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:02.3734617Z env: 2024-08-20T23:01:02.3734896Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:02.3735351Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:02.3736076Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:02.3736711Z ##[endgroup] 2024-08-20T23:01:02.3766177Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:01:02.3838816Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:01:02.3839567Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:01:02.3840151Z # shellcheck disable=SC2046 2024-08-20T23:01:02.3840614Z docker stop $(docker ps -q) || true 2024-08-20T23:01:02.3841084Z # Prune all of the docker images 2024-08-20T23:01:02.3841534Z docker system prune -af 2024-08-20T23:01:02.3849708Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:02.3850205Z env: 2024-08-20T23:01:02.3850482Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:01:02.3850930Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:01:02.3851656Z DOCKER_CONTAINER_ID: ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:02.3852460Z ##[endgroup] 2024-08-20T23:01:03.0604724Z ed7e90a491ee 2024-08-20T23:01:03.8168886Z Deleted Containers: 2024-08-20T23:01:03.8169438Z ed7e90a491ee4b0259044742eae99dac0f7c6b02c9134d1605a136a5b417686f 2024-08-20T23:01:03.8169886Z 2024-08-20T23:01:13.1402960Z Deleted Images: 2024-08-20T23:01:13.1404934Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:01:13.1407701Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:01:13.1409221Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:01:13.1410064Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:01:13.1410910Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:01:13.1411746Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:01:13.1412543Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:01:13.1413352Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:01:13.1414169Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:01:13.1415701Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:01:13.1416529Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:01:13.1417365Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:01:13.1418188Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:01:13.1418993Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:01:13.1419807Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:01:13.1420668Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:01:13.1421490Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:01:13.1422315Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:01:13.1423140Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:01:13.1424075Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:01:13.1424893Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:01:13.1425708Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:01:13.1426524Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:01:13.1427350Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:01:13.1428358Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:01:13.1429177Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:01:13.1429996Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:01:13.1430878Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:01:13.1431710Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:01:13.1432540Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:01:13.1433366Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:01:13.1434181Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:01:13.1435005Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:01:13.1436192Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:01:13.1437004Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:01:13.1437820Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:01:13.1438645Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:01:13.1439466Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:01:13.1440284Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:01:13.1441104Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:01:13.1441923Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:01:13.1442743Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:01:13.1443608Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:01:13.1444864Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:01:13.1445753Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:01:13.1446550Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:01:13.1447509Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:01:13.1448354Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:01:13.1449180Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:01:13.1449987Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:01:13.1450860Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:01:13.1451699Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:01:13.1452529Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:01:13.1453339Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:01:13.1454153Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:01:13.1454984Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:01:13.1455815Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:01:13.1456642Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:01:13.1457460Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:01:13.1458277Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:01:13.1459087Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:01:13.1459905Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:01:13.1460769Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:01:13.1461634Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:01:13.1462468Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:01:13.1463280Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:01:13.1464171Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:01:13.1464996Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:01:13.1465814Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:01:13.1466635Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:01:13.1467563Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:01:13.1468382Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:01:13.1469184Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:01:13.1469988Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:01:13.1470870Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:01:13.1471900Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:01:13.1472806Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:01:13.1473641Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:01:13.1474469Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:01:13.1475299Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:01:13.1476123Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:01:13.1476607Z 2024-08-20T23:01:13.1476766Z Total reclaimed space: 39.92GB 2024-08-20T23:01:13.1543903Z Post job cleanup. 2024-08-20T23:01:13.1595104Z Post job cleanup. 2024-08-20T23:01:13.2465973Z [command]/usr/bin/git version 2024-08-20T23:01:13.2643133Z git version 2.40.1 2024-08-20T23:01:13.2673624Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/189d0ba5-abf1-4234-b006-63f9267e75e3/.gitconfig' 2024-08-20T23:01:13.2684197Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/189d0ba5-abf1-4234-b006-63f9267e75e3' before making global git config changes 2024-08-20T23:01:13.2685415Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:01:13.2689145Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:01:13.2759629Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:01:13.2805604Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:01:13.3252928Z Entering 'android/libs/fbjni' 2024-08-20T23:01:13.3320902Z Entering 'third_party/FP16' 2024-08-20T23:01:13.3391597Z Entering 'third_party/FXdiv' 2024-08-20T23:01:13.3461293Z Entering 'third_party/NNPACK' 2024-08-20T23:01:13.3536398Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:01:13.3606252Z Entering 'third_party/XNNPACK' 2024-08-20T23:01:13.3692521Z Entering 'third_party/benchmark' 2024-08-20T23:01:13.3763747Z Entering 'third_party/cpp-httplib' 2024-08-20T23:01:13.3834467Z Entering 'third_party/cpuinfo' 2024-08-20T23:01:13.3904737Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:01:13.3974145Z Entering 'third_party/cutlass' 2024-08-20T23:01:13.4051015Z Entering 'third_party/eigen' 2024-08-20T23:01:13.4123034Z Entering 'third_party/fbgemm' 2024-08-20T23:01:13.4194546Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:01:13.4262204Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:01:13.4329424Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:01:13.4402109Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:01:13.4468120Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:01:13.4538737Z Entering 'third_party/flatbuffers' 2024-08-20T23:01:13.4611768Z Entering 'third_party/fmt' 2024-08-20T23:01:13.4682127Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:01:13.4752745Z Entering 'third_party/gloo' 2024-08-20T23:01:13.4822629Z Entering 'third_party/googletest' 2024-08-20T23:01:13.4891157Z Entering 'third_party/ideep' 2024-08-20T23:01:13.4958467Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:01:13.5036475Z Entering 'third_party/ittapi' 2024-08-20T23:01:13.5109038Z Entering 'third_party/kineto' 2024-08-20T23:01:13.5177159Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:01:13.5243367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:01:13.5314642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:01:13.5383492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:01:13.5452213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:01:13.5518380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:01:13.5591397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:01:13.5659421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:01:13.5728483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:01:13.5798231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:01:13.5869586Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:01:13.5936932Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:01:13.6008681Z Entering 'third_party/mimalloc' 2024-08-20T23:01:13.6078436Z Entering 'third_party/nccl/nccl' 2024-08-20T23:01:13.6149440Z Entering 'third_party/nlohmann' 2024-08-20T23:01:13.6219633Z Entering 'third_party/onnx' 2024-08-20T23:01:13.6302650Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:01:13.6370525Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:01:13.6443893Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:01:13.6514596Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:01:13.6582188Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:01:13.6649143Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:01:13.6717318Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:01:13.6785852Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:01:13.6851557Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:01:13.6917857Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:01:13.6983828Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:01:13.7054649Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:01:13.7126126Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:01:13.7215694Z Entering 'third_party/pocketfft' 2024-08-20T23:01:13.7286124Z Entering 'third_party/protobuf' 2024-08-20T23:01:13.7357105Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:01:13.7424666Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:01:13.7497448Z Entering 'third_party/psimd' 2024-08-20T23:01:13.7564099Z Entering 'third_party/pthreadpool' 2024-08-20T23:01:13.7634047Z Entering 'third_party/pybind11' 2024-08-20T23:01:13.7703835Z Entering 'third_party/python-peachpy' 2024-08-20T23:01:13.7773152Z Entering 'third_party/sleef' 2024-08-20T23:01:13.7846383Z Entering 'third_party/tensorpipe' 2024-08-20T23:01:13.7914817Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:01:13.7982954Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:01:13.8049784Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:01:13.8116510Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:01:13.8182981Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:01:13.8279304Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:01:13.8316499Z http.https://github.com/.extraheader 2024-08-20T23:01:13.8327315Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:01:13.8375969Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:01:13.8748470Z Entering 'android/libs/fbjni' 2024-08-20T23:01:13.8794318Z http.https://github.com/.extraheader 2024-08-20T23:01:13.8840125Z Entering 'third_party/FP16' 2024-08-20T23:01:13.8885311Z http.https://github.com/.extraheader 2024-08-20T23:01:13.8930050Z Entering 'third_party/FXdiv' 2024-08-20T23:01:13.8974579Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9019454Z Entering 'third_party/NNPACK' 2024-08-20T23:01:13.9064271Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9111263Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:01:13.9156193Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9200879Z Entering 'third_party/XNNPACK' 2024-08-20T23:01:13.9245956Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9307017Z Entering 'third_party/benchmark' 2024-08-20T23:01:13.9351937Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9396397Z Entering 'third_party/cpp-httplib' 2024-08-20T23:01:13.9443377Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9487686Z Entering 'third_party/cpuinfo' 2024-08-20T23:01:13.9532795Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9576420Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:01:13.9620944Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9665082Z Entering 'third_party/cutlass' 2024-08-20T23:01:13.9710177Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9761681Z Entering 'third_party/eigen' 2024-08-20T23:01:13.9805602Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9851737Z Entering 'third_party/fbgemm' 2024-08-20T23:01:13.9895613Z http.https://github.com/.extraheader 2024-08-20T23:01:13.9940137Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:01:13.9988332Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0032654Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:01:14.0075388Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0120414Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:01:14.0162899Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0212057Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:01:14.0254640Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0297416Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:01:14.0339909Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0385457Z Entering 'third_party/flatbuffers' 2024-08-20T23:01:14.0430389Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0477487Z Entering 'third_party/fmt' 2024-08-20T23:01:14.0524530Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0568826Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:01:14.0613869Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0658588Z Entering 'third_party/gloo' 2024-08-20T23:01:14.0703335Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0749026Z Entering 'third_party/googletest' 2024-08-20T23:01:14.0793113Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0836726Z Entering 'third_party/ideep' 2024-08-20T23:01:14.0879768Z http.https://github.com/.extraheader 2024-08-20T23:01:14.0923234Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:01:14.0965605Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1018370Z Entering 'third_party/ittapi' 2024-08-20T23:01:14.1061465Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1105428Z Entering 'third_party/kineto' 2024-08-20T23:01:14.1149446Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1193904Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:01:14.1240436Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1286527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:01:14.1329679Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1375396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:01:14.1418677Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1464750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:01:14.1505979Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1551355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:01:14.1594632Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1637273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:01:14.1680538Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1729257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:01:14.1771676Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1817304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:01:14.1859759Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1904513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:01:14.1947330Z http.https://github.com/.extraheader 2024-08-20T23:01:14.1993764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:01:14.2036923Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2084737Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:01:14.2131036Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2174306Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:01:14.2216865Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2263178Z Entering 'third_party/mimalloc' 2024-08-20T23:01:14.2307308Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2351240Z Entering 'third_party/nccl/nccl' 2024-08-20T23:01:14.2395742Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2440048Z Entering 'third_party/nlohmann' 2024-08-20T23:01:14.2483830Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2529956Z Entering 'third_party/onnx' 2024-08-20T23:01:14.2573262Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2631269Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:01:14.2674468Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2720276Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:01:14.2763902Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2812821Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:01:14.2857272Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2902699Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:01:14.2949849Z http.https://github.com/.extraheader 2024-08-20T23:01:14.2992995Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:01:14.3035333Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3078028Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:01:14.3120829Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3165067Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:01:14.3207668Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3252720Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:01:14.3300330Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3343785Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:01:14.3386153Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3429361Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:01:14.3471491Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3516489Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:01:14.3559300Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3606901Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:01:14.3649840Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3702247Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:01:14.3744602Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3809541Z Entering 'third_party/pocketfft' 2024-08-20T23:01:14.3853854Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3897866Z Entering 'third_party/protobuf' 2024-08-20T23:01:14.3942057Z http.https://github.com/.extraheader 2024-08-20T23:01:14.3988353Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:01:14.4031200Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4075152Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:01:14.4117883Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4164759Z Entering 'third_party/psimd' 2024-08-20T23:01:14.4209834Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4256578Z Entering 'third_party/pthreadpool' 2024-08-20T23:01:14.4297943Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4341740Z Entering 'third_party/pybind11' 2024-08-20T23:01:14.4384944Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4428563Z Entering 'third_party/python-peachpy' 2024-08-20T23:01:14.4472103Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4516122Z Entering 'third_party/sleef' 2024-08-20T23:01:14.4560743Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4605763Z Entering 'third_party/tensorpipe' 2024-08-20T23:01:14.4650560Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4694598Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:01:14.4736678Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4779432Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:01:14.4822447Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4864767Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:01:14.4908316Z http.https://github.com/.extraheader 2024-08-20T23:01:14.4953390Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:01:14.4996692Z http.https://github.com/.extraheader 2024-08-20T23:01:14.5038319Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:01:14.5083641Z http.https://github.com/.extraheader 2024-08-20T23:01:14.5247530Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:01:14.5267833Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:01:14.5275923Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:01:14.5276442Z ##[endgroup] 2024-08-20T23:01:21.4793701Z Cleaning up orphan processes