2024-12-18T00:18:00.3662012Z Current runner version: '2.321.0' 2024-12-18T00:18:00.3668754Z Runner name: 'i-096ab043c4c0d1de7' 2024-12-18T00:18:00.3669763Z Runner group name: 'Default' 2024-12-18T00:18:00.3670937Z Machine name: 'ip-10-0-22-90' 2024-12-18T00:18:00.3676281Z ##[group]GITHUB_TOKEN Permissions 2024-12-18T00:18:00.3678940Z Actions: read 2024-12-18T00:18:00.3679929Z Attestations: read 2024-12-18T00:18:00.3680727Z Checks: read 2024-12-18T00:18:00.3681400Z Contents: read 2024-12-18T00:18:00.3682201Z Deployments: read 2024-12-18T00:18:00.3682895Z Discussions: read 2024-12-18T00:18:00.3683626Z Issues: read 2024-12-18T00:18:00.3684402Z Metadata: read 2024-12-18T00:18:00.3685111Z Packages: read 2024-12-18T00:18:00.3685793Z Pages: read 2024-12-18T00:18:00.3686519Z PullRequests: read 2024-12-18T00:18:00.3687267Z RepositoryProjects: read 2024-12-18T00:18:00.3688074Z SecurityEvents: read 2024-12-18T00:18:00.3688850Z Statuses: read 2024-12-18T00:18:00.3689502Z ##[endgroup] 2024-12-18T00:18:00.3692939Z Secret source: Actions 2024-12-18T00:18:00.3694032Z Prepare workflow directory 2024-12-18T00:18:00.4167756Z Prepare all required actions 2024-12-18T00:18:00.4204316Z Getting action download info 2024-12-18T00:18:00.6269771Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-18T00:18:02.5876975Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T00:18:17.1150280Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-12-18T00:18:17.3431785Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-18T00:18:17.6275375Z Getting action download info 2024-12-18T00:18:17.7616049Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-18T00:18:18.0124666Z Getting action download info 2024-12-18T00:18:18.1217527Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-18T00:18:18.3174690Z Getting action download info 2024-12-18T00:18:18.4455605Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-12-18T00:18:18.7008713Z Getting action download info 2024-12-18T00:18:18.8407497Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e) 2024-12-18T00:18:20.4145677Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T00:18:20.4147532Z ##[group] Inputs 2024-12-18T00:18:20.4147923Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:18:20.4150000Z test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-12-18T00:18:20.4152354Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:18:20.4153217Z sync-tag: 2024-12-18T00:18:20.4153932Z timeout-minutes: 240 2024-12-18T00:18:20.4154198Z use-gha: 2024-12-18T00:18:20.4154423Z dashboard-tag: 2024-12-18T00:18:20.4154678Z s3-bucket: gha-artifacts 2024-12-18T00:18:20.4154959Z aws-role-to-assume: 2024-12-18T00:18:20.4155476Z disable-monitor: false 2024-12-18T00:18:20.4155754Z ##[endgroup] 2024-12-18T00:18:20.4156572Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:18:20.4690878Z A job started hook has been configured by the self-hosted runner administrator 2024-12-18T00:18:20.4788422Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-12-18T00:18:20.4799732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:18:20.4800369Z ##[endgroup] 2024-12-18T00:18:21.6396504Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-12-18T00:18:21.6397003Z Instance Type: g5.4xlarge 2024-12-18T00:18:21.6397287Z AMI Name: unknown 2024-12-18T00:18:21.6436658Z AMI ID: ami-0fff1b9a61dec8a5f 2024-12-18T00:18:27.4149742Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6 2024-12-18T00:18:27.4150194Z with: 2024-12-18T00:18:27.4150786Z github-secret: *** 2024-12-18T00:18:27.4151484Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-12-18T00:18:27.4152234Z activate-with-label: false 2024-12-18T00:18:27.4152536Z label: with-ssh 2024-12-18T00:18:27.4152784Z remove-existing-keys: true 2024-12-18T00:18:27.4153073Z fail-silently: true 2024-12-18T00:18:27.4153321Z env: 2024-12-18T00:18:27.4153542Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:18:27.4153819Z ##[endgroup] 2024-12-18T00:18:27.5340802Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-12-18T00:18:27.5342508Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2024-12-18T00:18:27.5491347Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-18T00:18:27.5491818Z with: 2024-12-18T00:18:27.5492048Z no-sudo: true 2024-12-18T00:18:27.5492296Z submodules: recursive 2024-12-18T00:18:27.5492563Z fetch-depth: 0 2024-12-18T00:18:27.5492792Z env: 2024-12-18T00:18:27.5493009Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:18:27.5493268Z ##[endgroup] 2024-12-18T00:18:27.5612939Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:18:27.5613967Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:18:27.5626224Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:18:27.5626611Z env: 2024-12-18T00:18:27.5626832Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:18:27.5627107Z ##[endgroup] 2024-12-18T00:18:27.5725564Z ##[group]Run retry () { 2024-12-18T00:18:27.5725878Z retry () { 2024-12-18T00:18:27.5726255Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-18T00:18:27.5726672Z } 2024-12-18T00:18:27.5726918Z echo "${GITHUB_WORKSPACE}" 2024-12-18T00:18:27.5727256Z if [ -z "${NO_SUDO}" ]; then 2024-12-18T00:18:27.5727680Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T00:18:27.5728039Z else 2024-12-18T00:18:27.5728329Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T00:18:27.5728646Z fi 2024-12-18T00:18:27.5728893Z mkdir "${GITHUB_WORKSPACE}" 2024-12-18T00:18:27.5737718Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:18:27.5738094Z env: 2024-12-18T00:18:27.5738320Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:18:27.5738589Z NO_SUDO: true 2024-12-18T00:18:27.5738823Z ##[endgroup] 2024-12-18T00:18:27.5770200Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:18:27.5863310Z ##[group]Run malfet/checkout@silent-checkout 2024-12-18T00:18:27.5863663Z with: 2024-12-18T00:18:27.5863940Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:18:27.5864287Z fetch-depth: 0 2024-12-18T00:18:27.5864549Z submodules: recursive 2024-12-18T00:18:27.5864822Z quiet-checkout: true 2024-12-18T00:18:27.5865110Z repository: pytorch/pytorch 2024-12-18T00:18:27.5865499Z token: *** 2024-12-18T00:18:27.5865743Z ssh-strict: true 2024-12-18T00:18:27.5866222Z persist-credentials: true 2024-12-18T00:18:27.5866502Z clean: true 2024-12-18T00:18:27.5866768Z sparse-checkout-cone-mode: true 2024-12-18T00:18:27.5867076Z lfs: false 2024-12-18T00:18:27.5867319Z set-safe-directory: true 2024-12-18T00:18:27.5867588Z env: 2024-12-18T00:18:27.5867807Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:18:27.5868075Z ##[endgroup] 2024-12-18T00:18:27.6776398Z Syncing repository: pytorch/pytorch 2024-12-18T00:18:27.6777591Z ##[group]Getting Git version info 2024-12-18T00:18:27.6778083Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-18T00:18:27.6779271Z [command]/usr/bin/git version 2024-12-18T00:18:27.6779588Z git version 2.40.1 2024-12-18T00:18:27.6807039Z ##[endgroup] 2024-12-18T00:18:27.6823360Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d920b3e5-4a29-4ede-8706-26391ae8ff66' before making global git config changes 2024-12-18T00:18:27.6824853Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T00:18:27.6828613Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:18:27.6864376Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-18T00:18:27.6868348Z ##[group]Initializing the repository 2024-12-18T00:18:27.6871398Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:18:27.6903334Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-18T00:18:27.6904067Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-18T00:18:27.6904640Z hint: of your new repositories, which will suppress this warning, call: 2024-12-18T00:18:27.6905057Z hint: 2024-12-18T00:18:27.6905377Z hint: git config --global init.defaultBranch 2024-12-18T00:18:27.6905736Z hint: 2024-12-18T00:18:27.6906075Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-18T00:18:27.6906650Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-18T00:18:27.6907240Z hint: 2024-12-18T00:18:27.6907591Z hint: git branch -m 2024-12-18T00:18:27.6908418Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-18T00:18:27.6917828Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-18T00:18:27.6946997Z ##[endgroup] 2024-12-18T00:18:27.6947483Z ##[group]Disabling automatic garbage collection 2024-12-18T00:18:27.6950326Z [command]/usr/bin/git config --local gc.auto 0 2024-12-18T00:18:27.6978706Z ##[endgroup] 2024-12-18T00:18:27.6979347Z ##[group]Setting up auth 2024-12-18T00:18:27.6984337Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T00:18:27.7012971Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T00:18:27.7367119Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T00:18:27.7393647Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T00:18:27.7730532Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T00:18:27.7775820Z ##[endgroup] 2024-12-18T00:18:27.7776283Z ##[group]Fetching the repository 2024-12-18T00:18:27.7781406Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-18T00:18:30.9856427Z remote: Enumerating objects: 1056571 2024-12-18T00:18:30.9856977Z remote: Enumerating objects: 1057277, done. 2024-12-18T00:18:30.9858063Z remote: Counting objects: 0% (1/706) 2024-12-18T00:18:30.9858833Z remote: Counting objects: 1% (8/706) 2024-12-18T00:18:30.9859898Z remote: Counting objects: 2% (15/706) 2024-12-18T00:18:30.9860759Z remote: Counting objects: 3% (22/706) 2024-12-18T00:18:30.9861427Z remote: Counting objects: 4% (29/706) 2024-12-18T00:18:30.9861810Z remote: Counting objects: 5% (36/706) 2024-12-18T00:18:30.9862172Z remote: Counting objects: 6% (43/706) 2024-12-18T00:18:30.9862754Z remote: Counting objects: 7% (50/706) 2024-12-18T00:18:30.9863495Z remote: Counting objects: 8% (57/706) 2024-12-18T00:18:30.9863993Z remote: Counting objects: 9% (64/706) 2024-12-18T00:18:30.9864543Z remote: Counting objects: 10% (71/706) 2024-12-18T00:18:30.9864981Z remote: Counting objects: 11% (78/706) 2024-12-18T00:18:30.9865586Z remote: Counting objects: 12% (85/706) 2024-12-18T00:18:30.9865966Z remote: Counting objects: 13% (92/706) 2024-12-18T00:18:30.9866384Z remote: Counting objects: 14% (99/706) 2024-12-18T00:18:30.9866920Z remote: Counting objects: 15% (106/706) 2024-12-18T00:18:30.9867372Z remote: Counting objects: 16% (113/706) 2024-12-18T00:18:30.9867844Z remote: Counting objects: 17% (121/706) 2024-12-18T00:18:30.9868513Z remote: Counting objects: 18% (128/706) 2024-12-18T00:18:30.9868917Z remote: Counting objects: 19% (135/706) 2024-12-18T00:18:30.9869720Z remote: Counting objects: 20% (142/706) 2024-12-18T00:18:30.9870130Z remote: Counting objects: 21% (149/706) 2024-12-18T00:18:30.9870497Z remote: Counting objects: 22% (156/706) 2024-12-18T00:18:30.9870864Z remote: Counting objects: 23% (163/706) 2024-12-18T00:18:30.9871357Z remote: Counting objects: 24% (170/706) 2024-12-18T00:18:30.9871801Z remote: Counting objects: 25% (177/706) 2024-12-18T00:18:30.9872167Z remote: Counting objects: 26% (184/706) 2024-12-18T00:18:30.9872613Z remote: Counting objects: 27% (191/706) 2024-12-18T00:18:30.9872983Z remote: Counting objects: 28% (198/706) 2024-12-18T00:18:30.9873346Z remote: Counting objects: 29% (205/706) 2024-12-18T00:18:30.9873713Z remote: Counting objects: 30% (212/706) 2024-12-18T00:18:30.9874078Z remote: Counting objects: 31% (219/706) 2024-12-18T00:18:30.9874489Z remote: Counting objects: 32% (226/706) 2024-12-18T00:18:30.9874868Z remote: Counting objects: 33% (233/706) 2024-12-18T00:18:30.9875240Z remote: Counting objects: 34% (241/706) 2024-12-18T00:18:30.9875602Z remote: Counting objects: 35% (248/706) 2024-12-18T00:18:30.9875958Z remote: Counting objects: 36% (255/706) 2024-12-18T00:18:30.9876319Z remote: Counting objects: 37% (262/706) 2024-12-18T00:18:30.9876683Z remote: Counting objects: 38% (269/706) 2024-12-18T00:18:30.9877048Z remote: Counting objects: 39% (276/706) 2024-12-18T00:18:30.9877417Z remote: Counting objects: 40% (283/706) 2024-12-18T00:18:30.9877773Z remote: Counting objects: 41% (290/706) 2024-12-18T00:18:30.9878138Z remote: Counting objects: 42% (297/706) 2024-12-18T00:18:30.9878499Z remote: Counting objects: 43% (304/706) 2024-12-18T00:18:30.9878859Z remote: Counting objects: 44% (311/706) 2024-12-18T00:18:30.9879231Z remote: Counting objects: 45% (318/706) 2024-12-18T00:18:30.9879599Z remote: Counting objects: 46% (325/706) 2024-12-18T00:18:30.9879957Z remote: Counting objects: 47% (332/706) 2024-12-18T00:18:30.9880321Z remote: Counting objects: 48% (339/706) 2024-12-18T00:18:30.9880687Z remote: Counting objects: 49% (346/706) 2024-12-18T00:18:30.9881102Z remote: Counting objects: 50% (353/706) 2024-12-18T00:18:30.9881462Z remote: Counting objects: 51% (361/706) 2024-12-18T00:18:30.9881819Z remote: Counting objects: 52% (368/706) 2024-12-18T00:18:30.9882316Z remote: Counting objects: 53% (375/706) 2024-12-18T00:18:30.9882681Z remote: Counting objects: 54% (382/706) 2024-12-18T00:18:30.9883057Z remote: Counting objects: 55% (389/706) 2024-12-18T00:18:30.9883424Z remote: Counting objects: 56% (396/706) 2024-12-18T00:18:30.9883842Z remote: Counting objects: 57% (403/706) 2024-12-18T00:18:30.9884218Z remote: Counting objects: 58% (410/706) 2024-12-18T00:18:30.9884584Z remote: Counting objects: 59% (417/706) 2024-12-18T00:18:30.9884958Z remote: Counting objects: 60% (424/706) 2024-12-18T00:18:30.9885326Z remote: Counting objects: 61% (431/706) 2024-12-18T00:18:30.9885697Z remote: Counting objects: 62% (438/706) 2024-12-18T00:18:30.9886089Z remote: Counting objects: 63% (445/706) 2024-12-18T00:18:30.9886451Z remote: Counting objects: 64% (452/706) 2024-12-18T00:18:30.9886824Z remote: Counting objects: 65% (459/706) 2024-12-18T00:18:30.9887192Z remote: Counting objects: 66% (466/706) 2024-12-18T00:18:30.9887565Z remote: Counting objects: 67% (474/706) 2024-12-18T00:18:30.9887939Z remote: Counting objects: 68% (481/706) 2024-12-18T00:18:30.9888308Z remote: Counting objects: 69% (488/706) 2024-12-18T00:18:30.9888681Z remote: Counting objects: 70% (495/706) 2024-12-18T00:18:30.9889048Z remote: Counting objects: 71% (502/706) 2024-12-18T00:18:30.9889414Z remote: Counting objects: 72% (509/706) 2024-12-18T00:18:30.9889883Z remote: Counting objects: 73% (516/706) 2024-12-18T00:18:30.9890254Z remote: Counting objects: 74% (523/706) 2024-12-18T00:18:30.9890614Z remote: Counting objects: 75% (530/706) 2024-12-18T00:18:30.9891014Z remote: Counting objects: 76% (537/706) 2024-12-18T00:18:30.9891405Z remote: Counting objects: 77% (544/706) 2024-12-18T00:18:30.9891769Z remote: Counting objects: 78% (551/706) 2024-12-18T00:18:30.9892138Z remote: Counting objects: 79% (558/706) 2024-12-18T00:18:30.9892509Z remote: Counting objects: 80% (565/706) 2024-12-18T00:18:30.9892871Z remote: Counting objects: 81% (572/706) 2024-12-18T00:18:30.9893236Z remote: Counting objects: 82% (579/706) 2024-12-18T00:18:30.9893600Z remote: Counting objects: 83% (586/706) 2024-12-18T00:18:30.9893969Z remote: Counting objects: 84% (594/706) 2024-12-18T00:18:30.9894328Z remote: Counting objects: 85% (601/706) 2024-12-18T00:18:30.9894705Z remote: Counting objects: 86% (608/706) 2024-12-18T00:18:30.9895075Z remote: Counting objects: 87% (615/706) 2024-12-18T00:18:30.9895447Z remote: Counting objects: 88% (622/706) 2024-12-18T00:18:30.9895811Z remote: Counting objects: 89% (629/706) 2024-12-18T00:18:30.9896169Z remote: Counting objects: 90% (636/706) 2024-12-18T00:18:30.9896534Z remote: Counting objects: 91% (643/706) 2024-12-18T00:18:30.9896899Z remote: Counting objects: 92% (650/706) 2024-12-18T00:18:30.9897273Z remote: Counting objects: 93% (657/706) 2024-12-18T00:18:30.9897645Z remote: Counting objects: 94% (664/706) 2024-12-18T00:18:30.9898010Z remote: Counting objects: 95% (671/706) 2024-12-18T00:18:30.9898370Z remote: Counting objects: 96% (678/706) 2024-12-18T00:18:30.9898736Z remote: Counting objects: 97% (685/706) 2024-12-18T00:18:30.9899491Z remote: Counting objects: 98% (692/706) 2024-12-18T00:18:30.9899867Z remote: Counting objects: 99% (699/706) 2024-12-18T00:18:30.9900231Z remote: Counting objects: 100% (706/706) 2024-12-18T00:18:30.9900616Z remote: Counting objects: 100% (706/706), done. 2024-12-18T00:18:30.9956488Z remote: Compressing objects: 0% (1/300) 2024-12-18T00:18:31.0039634Z remote: Compressing objects: 1% (3/300) 2024-12-18T00:18:31.0088006Z remote: Compressing objects: 2% (6/300) 2024-12-18T00:18:31.0137387Z remote: Compressing objects: 3% (9/300) 2024-12-18T00:18:31.0205195Z remote: Compressing objects: 4% (12/300) 2024-12-18T00:18:31.0594822Z remote: Compressing objects: 5% (15/300) 2024-12-18T00:18:31.1046969Z remote: Compressing objects: 6% (18/300) 2024-12-18T00:18:31.1622185Z remote: Compressing objects: 7% (21/300) 2024-12-18T00:18:31.2321867Z remote: Compressing objects: 8% (24/300) 2024-12-18T00:18:31.2827464Z remote: Compressing objects: 9% (27/300) 2024-12-18T00:18:31.3145364Z remote: Compressing objects: 10% (30/300) 2024-12-18T00:18:31.3307495Z remote: Compressing objects: 11% (33/300) 2024-12-18T00:18:31.3383266Z remote: Compressing objects: 12% (36/300) 2024-12-18T00:18:31.3383730Z remote: Compressing objects: 13% (39/300) 2024-12-18T00:18:31.3387066Z remote: Compressing objects: 14% (42/300) 2024-12-18T00:18:31.3389940Z remote: Compressing objects: 15% (45/300) 2024-12-18T00:18:31.3390444Z remote: Compressing objects: 16% (48/300) 2024-12-18T00:18:31.3395625Z remote: Compressing objects: 17% (51/300) 2024-12-18T00:18:31.3398101Z remote: Compressing objects: 18% (54/300) 2024-12-18T00:18:31.3401091Z remote: Compressing objects: 19% (57/300) 2024-12-18T00:18:31.3401603Z remote: Compressing objects: 20% (60/300) 2024-12-18T00:18:31.3403419Z remote: Compressing objects: 21% (63/300) 2024-12-18T00:18:31.3405756Z remote: Compressing objects: 22% (66/300) 2024-12-18T00:18:31.3408723Z remote: Compressing objects: 23% (69/300) 2024-12-18T00:18:31.3413045Z remote: Compressing objects: 24% (72/300) 2024-12-18T00:18:31.3418019Z remote: Compressing objects: 25% (75/300) 2024-12-18T00:18:31.3420613Z remote: Compressing objects: 26% (78/300) 2024-12-18T00:18:31.3423080Z remote: Compressing objects: 27% (81/300) 2024-12-18T00:18:31.3425649Z remote: Compressing objects: 28% (84/300) 2024-12-18T00:18:31.3428184Z remote: Compressing objects: 29% (87/300) 2024-12-18T00:18:31.3435154Z remote: Compressing objects: 30% (90/300) 2024-12-18T00:18:31.3437609Z remote: Compressing objects: 31% (93/300) 2024-12-18T00:18:31.3440180Z remote: Compressing objects: 32% (96/300) 2024-12-18T00:18:31.3442681Z remote: Compressing objects: 33% (99/300) 2024-12-18T00:18:31.3446440Z remote: Compressing objects: 34% (102/300) 2024-12-18T00:18:31.3452729Z remote: Compressing objects: 35% (105/300) 2024-12-18T00:18:31.3456883Z remote: Compressing objects: 36% (108/300) 2024-12-18T00:18:31.3459422Z remote: Compressing objects: 37% (111/300) 2024-12-18T00:18:31.3462649Z remote: Compressing objects: 38% (114/300) 2024-12-18T00:18:31.3465619Z remote: Compressing objects: 39% (117/300) 2024-12-18T00:18:31.3468310Z remote: Compressing objects: 40% (120/300) 2024-12-18T00:18:31.3470979Z remote: Compressing objects: 41% (123/300) 2024-12-18T00:18:31.3473197Z remote: Compressing objects: 42% (126/300) 2024-12-18T00:18:31.3477574Z remote: Compressing objects: 43% (129/300) 2024-12-18T00:18:31.3480253Z remote: Compressing objects: 44% (132/300) 2024-12-18T00:18:31.3482670Z remote: Compressing objects: 45% (135/300) 2024-12-18T00:18:31.3485692Z remote: Compressing objects: 46% (138/300) 2024-12-18T00:18:31.3490743Z remote: Compressing objects: 47% (141/300) 2024-12-18T00:18:31.3493117Z remote: Compressing objects: 48% (144/300) 2024-12-18T00:18:31.3495743Z remote: Compressing objects: 49% (147/300) 2024-12-18T00:18:31.3498542Z remote: Compressing objects: 50% (150/300) 2024-12-18T00:18:31.3502030Z remote: Compressing objects: 51% (153/300) 2024-12-18T00:18:31.3504842Z remote: Compressing objects: 52% (156/300) 2024-12-18T00:18:31.3507486Z remote: Compressing objects: 53% (159/300) 2024-12-18T00:18:31.3510222Z remote: Compressing objects: 54% (162/300) 2024-12-18T00:18:31.3512621Z remote: Compressing objects: 55% (165/300) 2024-12-18T00:18:31.3515034Z remote: Compressing objects: 56% (168/300) 2024-12-18T00:18:31.3517674Z remote: Compressing objects: 57% (171/300) 2024-12-18T00:18:31.3520249Z remote: Compressing objects: 58% (174/300) 2024-12-18T00:18:31.3523335Z remote: Compressing objects: 59% (177/300) 2024-12-18T00:18:31.3526096Z remote: Compressing objects: 60% (180/300) 2024-12-18T00:18:31.3529241Z remote: Compressing objects: 61% (183/300) 2024-12-18T00:18:31.3531719Z remote: Compressing objects: 62% (186/300) 2024-12-18T00:18:31.3534279Z remote: Compressing objects: 63% (189/300) 2024-12-18T00:18:31.3537279Z remote: Compressing objects: 64% (192/300) 2024-12-18T00:18:31.3539969Z remote: Compressing objects: 65% (195/300) 2024-12-18T00:18:31.3542711Z remote: Compressing objects: 66% (198/300) 2024-12-18T00:18:31.3543149Z remote: Compressing objects: 67% (201/300) 2024-12-18T00:18:31.3549741Z remote: Compressing objects: 68% (204/300) 2024-12-18T00:18:31.3554750Z remote: Compressing objects: 69% (207/300) 2024-12-18T00:18:31.3558062Z remote: Compressing objects: 70% (210/300) 2024-12-18T00:18:31.3560867Z remote: Compressing objects: 71% (213/300) 2024-12-18T00:18:31.3563478Z remote: Compressing objects: 72% (216/300) 2024-12-18T00:18:31.3566645Z remote: Compressing objects: 73% (219/300) 2024-12-18T00:18:31.3569356Z remote: Compressing objects: 74% (222/300) 2024-12-18T00:18:31.3572100Z remote: Compressing objects: 75% (225/300) 2024-12-18T00:18:31.3574631Z remote: Compressing objects: 76% (228/300) 2024-12-18T00:18:31.3577484Z remote: Compressing objects: 77% (231/300) 2024-12-18T00:18:31.3577987Z remote: Compressing objects: 78% (234/300) 2024-12-18T00:18:31.3578374Z remote: Compressing objects: 79% (237/300) 2024-12-18T00:18:31.3580244Z remote: Compressing objects: 80% (240/300) 2024-12-18T00:18:31.3580668Z remote: Compressing objects: 81% (243/300) 2024-12-18T00:18:31.3582688Z remote: Compressing objects: 82% (246/300) 2024-12-18T00:18:31.3585161Z remote: Compressing objects: 83% (249/300) 2024-12-18T00:18:31.3587820Z remote: Compressing objects: 84% (252/300) 2024-12-18T00:18:31.3588298Z remote: Compressing objects: 85% (255/300) 2024-12-18T00:18:31.3590377Z remote: Compressing objects: 86% (258/300) 2024-12-18T00:18:31.3592926Z remote: Compressing objects: 87% (261/300) 2024-12-18T00:18:31.3593388Z remote: Compressing objects: 88% (264/300) 2024-12-18T00:18:31.3595735Z remote: Compressing objects: 89% (267/300) 2024-12-18T00:18:31.3596154Z remote: Compressing objects: 90% (270/300) 2024-12-18T00:18:31.3598163Z remote: Compressing objects: 91% (273/300) 2024-12-18T00:18:31.3598587Z remote: Compressing objects: 92% (276/300) 2024-12-18T00:18:31.3604274Z remote: Compressing objects: 93% (279/300) 2024-12-18T00:18:31.3604781Z remote: Compressing objects: 94% (282/300) 2024-12-18T00:18:31.3605170Z remote: Compressing objects: 95% (285/300) 2024-12-18T00:18:31.3605550Z remote: Compressing objects: 96% (288/300) 2024-12-18T00:18:31.3605924Z remote: Compressing objects: 97% (291/300) 2024-12-18T00:18:31.3606349Z remote: Compressing objects: 98% (294/300) 2024-12-18T00:18:31.3606838Z remote: Compressing objects: 99% (297/300) 2024-12-18T00:18:31.3607229Z remote: Compressing objects: 100% (300/300) 2024-12-18T00:18:31.3607644Z remote: Compressing objects: 100% (300/300), done. 2024-12-18T00:18:54.9928437Z remote: Total 1057277 (delta 567), reused 410 (delta 406), pack-reused 1056571 (from 3) 2024-12-18T00:19:17.0771967Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-18T00:19:17.0799706Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:19:17.0804763Z ##[endgroup] 2024-12-18T00:19:17.0805180Z ##[group]Determining the checkout info 2024-12-18T00:19:17.0806259Z ##[endgroup] 2024-12-18T00:19:17.0806659Z ##[group]Checking out the ref 2024-12-18T00:19:17.0809824Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:19:18.8395139Z ##[endgroup] 2024-12-18T00:19:18.8395793Z ##[group]Setting up auth for fetching submodules 2024-12-18T00:19:18.8399693Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T00:19:18.8442076Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-18T00:19:18.8471807Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-18T00:19:18.8503283Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-18T00:19:18.8532800Z ##[endgroup] 2024-12-18T00:19:18.8533402Z ##[group]Fetching submodules 2024-12-18T00:19:18.8535665Z [command]/usr/bin/git submodule sync --recursive 2024-12-18T00:19:18.8905447Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-18T00:19:18.9254689Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-18T00:19:18.9259018Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-18T00:19:18.9261435Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-18T00:19:18.9264675Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-18T00:19:18.9267330Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-18T00:19:18.9271127Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:18.9274303Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-18T00:19:18.9277772Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-18T00:19:18.9281493Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-18T00:19:18.9285117Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-18T00:19:18.9288697Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-18T00:19:18.9292584Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-18T00:19:18.9296351Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-18T00:19:18.9300573Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-18T00:19:18.9304663Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-18T00:19:18.9310012Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-18T00:19:18.9317251Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-18T00:19:18.9321614Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:18.9325689Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-18T00:19:18.9330279Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-18T00:19:18.9334588Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-18T00:19:18.9339094Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-18T00:19:18.9343650Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-18T00:19:18.9348305Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-18T00:19:18.9353025Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-18T00:19:18.9357717Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-18T00:19:18.9362403Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-18T00:19:18.9367652Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-18T00:19:18.9372396Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-18T00:19:18.9377739Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-18T00:19:18.9382544Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-18T00:19:18.9387754Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-18T00:19:18.9392942Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-18T00:19:18.9401494Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-18T00:19:18.9407229Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-18T00:19:18.9412738Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-18T00:19:18.9447294Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-18T00:19:19.2810174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-18T00:19:19.4773193Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-18T00:19:19.6861174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-18T00:19:19.9454338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-18T00:19:20.3130127Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-18T00:19:22.3282021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-18T00:19:33.3078607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-18T00:19:33.7681231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-18T00:19:35.9384808Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-18T00:19:36.4834059Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-18T00:19:37.1432937Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-18T00:19:38.4106197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-18T00:19:40.5396982Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-18T00:19:46.0974584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-18T00:19:47.7043330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-18T00:19:48.9349181Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-18T00:19:50.2658900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-18T00:19:50.7342549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-18T00:19:51.1285540Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-18T00:19:52.2583014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-18T00:19:52.6436389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-18T00:19:52.9565238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-18T00:19:54.5607276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-18T00:19:55.4332368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-18T00:19:55.8528590Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-18T00:20:03.0062049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-18T00:20:05.1173732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-18T00:20:10.0215134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-18T00:20:10.2884929Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T00:20:19.7667424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T00:20:20.0094803Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T00:20:20.2166180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T00:20:24.1503919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T00:20:24.4677966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T00:20:25.1224643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T00:20:25.5997665Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T00:20:25.6146350Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T00:20:25.6284759Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T00:20:25.6585083Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T00:20:25.7025952Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T00:20:25.7478492Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T00:20:26.6638689Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T00:20:26.6923567Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T00:20:26.9900357Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T00:20:27.0463002Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T00:20:27.1600437Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T00:20:27.2006790Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T00:20:27.8204501Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T00:20:28.1051455Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T00:20:28.2129201Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T00:20:28.2154418Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:20:28.2158445Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:20:28.2162378Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:20:28.2166572Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:20:28.2170545Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:20:28.2202395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T00:20:29.0508715Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T00:20:29.6610749Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T00:20:31.7833818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T00:20:32.7963817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T00:20:33.1746801Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T00:20:33.2840141Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T00:20:33.7879207Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T00:20:33.8567761Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T00:20:33.8717141Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T00:20:34.0295860Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T00:20:34.0735109Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T00:20:34.1193626Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T00:20:34.1524686Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T00:20:34.2034249Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T00:20:34.2199297Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T00:20:34.2220917Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T00:20:34.2251113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T00:20:48.5428841Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T00:20:48.5641898Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T00:20:48.6611688Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T00:20:48.6635195Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:20:48.6638241Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:20:48.6641779Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:20:48.6678266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T00:20:49.5384787Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T00:20:50.7611324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T00:20:51.9656380Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T00:20:51.9675585Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:20:51.9679329Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:20:51.9682640Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:20:51.9686708Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:20:51.9690421Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:20:51.9694382Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:20:51.9698235Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:20:51.9702576Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:20:51.9734306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T00:20:52.8786694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T00:20:53.2651021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T00:20:54.4972662Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T00:20:54.7837768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T00:20:55.2936633Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T00:20:56.3514901Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T00:21:03.4547598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T00:21:03.9040311Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T00:21:03.9266806Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T00:21:03.9685523Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T00:21:03.9853976Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T00:21:03.9873388Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:03.9904960Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T00:21:04.3148704Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T00:21:04.3375660Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T00:21:04.3846510Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T00:21:04.5045051Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T00:21:04.5247275Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T00:21:04.5679122Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T00:21:04.6346376Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T00:21:04.6792145Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T00:21:04.7153403Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T00:21:04.8470195Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T00:21:05.3622521Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T00:21:05.3663323Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:05.3693632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T00:21:06.6006346Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T00:21:06.6901719Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T00:21:06.6925597Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:06.6929102Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:06.6932505Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:06.6936006Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:06.6939832Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:06.6943482Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:06.6947182Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:06.6950930Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:06.6981807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T00:21:07.2893426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T00:21:08.3992542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T00:21:08.7544647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T00:21:15.9315124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T00:21:16.2235149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T00:21:16.4530019Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T00:21:16.7826595Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T00:21:23.7498160Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T00:21:23.7960723Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T00:21:23.8149294Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T00:21:23.9466762Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T00:21:23.9633567Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T00:21:23.9819602Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T00:21:24.0019853Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T00:21:24.0038698Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:24.0042084Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:24.0073337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T00:21:25.8790069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T00:21:27.2044439Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T00:21:27.2580402Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T00:21:27.9086538Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T00:21:27.9230821Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T00:21:28.2511383Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T00:21:28.2537344Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:28.2540680Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:28.2572675Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T00:21:28.7189163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T00:21:29.7582982Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T00:21:29.8381519Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T00:21:29.8497221Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T00:21:29.8647981Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T00:21:29.9073151Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T00:21:29.9412242Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T00:21:29.9908518Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T00:21:30.0253729Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T00:21:30.0275028Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:30.0278253Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:30.0281670Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:30.0285229Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:30.0317155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T00:21:31.0330439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T00:21:31.2939417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T00:21:33.6649062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T00:21:34.8399424Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T00:21:34.8591877Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T00:21:34.9310078Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T00:21:34.9653843Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T00:21:34.9672477Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:34.9704037Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T00:21:35.2012490Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T00:21:35.2058304Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T00:21:35.2411634Z Entering 'android/libs/fbjni' 2024-12-18T00:21:35.2463831Z Entering 'third_party/FP16' 2024-12-18T00:21:35.2521295Z Entering 'third_party/FXdiv' 2024-12-18T00:21:35.2572515Z Entering 'third_party/NNPACK' 2024-12-18T00:21:35.2630748Z Entering 'third_party/NVTX' 2024-12-18T00:21:35.2682017Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:21:35.2739350Z Entering 'third_party/XNNPACK' 2024-12-18T00:21:35.2808383Z Entering 'third_party/benchmark' 2024-12-18T00:21:35.2857942Z Entering 'third_party/composable_kernel' 2024-12-18T00:21:35.2917050Z Entering 'third_party/cpp-httplib' 2024-12-18T00:21:35.2968452Z Entering 'third_party/cpuinfo' 2024-12-18T00:21:35.3021426Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:21:35.3077146Z Entering 'third_party/cutlass' 2024-12-18T00:21:35.3137782Z Entering 'third_party/eigen' 2024-12-18T00:21:35.3192223Z Entering 'third_party/fbgemm' 2024-12-18T00:21:35.3247502Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:21:35.3298075Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:21:35.3348604Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:21:35.3406799Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:21:35.3458519Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:21:35.3510012Z Entering 'third_party/flatbuffers' 2024-12-18T00:21:35.3563915Z Entering 'third_party/fmt' 2024-12-18T00:21:35.3615434Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:21:35.3666880Z Entering 'third_party/gloo' 2024-12-18T00:21:35.3717724Z Entering 'third_party/googletest' 2024-12-18T00:21:35.3768326Z Entering 'third_party/ideep' 2024-12-18T00:21:35.3819408Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:21:35.3878407Z Entering 'third_party/ittapi' 2024-12-18T00:21:35.3930298Z Entering 'third_party/kineto' 2024-12-18T00:21:35.3985703Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:21:35.4037643Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:21:35.4092708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:21:35.4146489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:21:35.4197683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:21:35.4249765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:35.4305088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:21:35.4360064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:21:35.4412752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:21:35.4465036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:21:35.4519614Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:21:35.4569294Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:21:35.4623222Z Entering 'third_party/mimalloc' 2024-12-18T00:21:35.4676532Z Entering 'third_party/nccl/nccl' 2024-12-18T00:21:35.4729619Z Entering 'third_party/nlohmann' 2024-12-18T00:21:35.4782518Z Entering 'third_party/onnx' 2024-12-18T00:21:35.4849858Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:35.4905593Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:21:35.4957832Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:35.5009079Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:35.5059062Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:35.5109911Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:35.5161120Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:35.5211004Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:35.5260809Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:35.5309642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:35.5362081Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:35.5416258Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:35.5487908Z Entering 'third_party/pocketfft' 2024-12-18T00:21:35.5539505Z Entering 'third_party/protobuf' 2024-12-18T00:21:35.5593617Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:35.5646969Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:35.5700956Z Entering 'third_party/psimd' 2024-12-18T00:21:35.5757779Z Entering 'third_party/pthreadpool' 2024-12-18T00:21:35.5809931Z Entering 'third_party/pybind11' 2024-12-18T00:21:35.5862481Z Entering 'third_party/python-peachpy' 2024-12-18T00:21:35.5918189Z Entering 'third_party/sleef' 2024-12-18T00:21:35.5968957Z Entering 'third_party/tensorpipe' 2024-12-18T00:21:35.6021563Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:35.6071150Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:35.6120318Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:35.6169166Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:35.6216637Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:35.6292341Z ##[endgroup] 2024-12-18T00:21:35.6295772Z ##[group]Persisting credentials for submodules 2024-12-18T00:21:35.6299370Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T00:21:35.6656710Z Entering 'android/libs/fbjni' 2024-12-18T00:21:35.6729209Z Entering 'third_party/FP16' 2024-12-18T00:21:35.6800270Z Entering 'third_party/FXdiv' 2024-12-18T00:21:35.6869724Z Entering 'third_party/NNPACK' 2024-12-18T00:21:35.6938930Z Entering 'third_party/NVTX' 2024-12-18T00:21:35.7010099Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:21:35.7078420Z Entering 'third_party/XNNPACK' 2024-12-18T00:21:35.7161633Z Entering 'third_party/benchmark' 2024-12-18T00:21:35.7230759Z Entering 'third_party/composable_kernel' 2024-12-18T00:21:35.7305037Z Entering 'third_party/cpp-httplib' 2024-12-18T00:21:35.7372834Z Entering 'third_party/cpuinfo' 2024-12-18T00:21:35.7441531Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:21:35.7511204Z Entering 'third_party/cutlass' 2024-12-18T00:21:35.7586198Z Entering 'third_party/eigen' 2024-12-18T00:21:35.7655899Z Entering 'third_party/fbgemm' 2024-12-18T00:21:35.7722768Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:21:35.7789275Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:21:35.7856937Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:21:35.7930384Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:21:35.7995531Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:21:35.8064520Z Entering 'third_party/flatbuffers' 2024-12-18T00:21:35.8137538Z Entering 'third_party/fmt' 2024-12-18T00:21:35.8206931Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:21:35.8276099Z Entering 'third_party/gloo' 2024-12-18T00:21:35.8345490Z Entering 'third_party/googletest' 2024-12-18T00:21:35.8414983Z Entering 'third_party/ideep' 2024-12-18T00:21:35.8481770Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:21:35.8557015Z Entering 'third_party/ittapi' 2024-12-18T00:21:35.8627024Z Entering 'third_party/kineto' 2024-12-18T00:21:35.8693298Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:21:35.8764544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:21:35.8834280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:21:35.8902476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:21:35.8971460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:21:35.9037057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:35.9108819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:21:35.9180511Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:21:35.9248253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:21:35.9316388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:21:35.9386562Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:21:35.9452031Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:21:35.9521749Z Entering 'third_party/mimalloc' 2024-12-18T00:21:35.9591435Z Entering 'third_party/nccl/nccl' 2024-12-18T00:21:35.9661525Z Entering 'third_party/nlohmann' 2024-12-18T00:21:35.9732275Z Entering 'third_party/onnx' 2024-12-18T00:21:35.9815021Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:35.9886357Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:21:35.9956506Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:36.0022052Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:36.0087610Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:36.0155180Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:36.0224354Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:36.0294055Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:36.0360389Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:36.0423703Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:36.0491427Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:36.0562268Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:36.0653298Z Entering 'third_party/pocketfft' 2024-12-18T00:21:36.0722486Z Entering 'third_party/protobuf' 2024-12-18T00:21:36.0791600Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:36.0858369Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:36.0929535Z Entering 'third_party/psimd' 2024-12-18T00:21:36.0998659Z Entering 'third_party/pthreadpool' 2024-12-18T00:21:36.1068272Z Entering 'third_party/pybind11' 2024-12-18T00:21:36.1137884Z Entering 'third_party/python-peachpy' 2024-12-18T00:21:36.1208323Z Entering 'third_party/sleef' 2024-12-18T00:21:36.1277404Z Entering 'third_party/tensorpipe' 2024-12-18T00:21:36.1345317Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:36.1412294Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:36.1477964Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:36.1544043Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:36.1616279Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:36.1709074Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T00:21:36.2064806Z Entering 'android/libs/fbjni' 2024-12-18T00:21:36.2129662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T00:21:36.2150691Z Entering 'third_party/FP16' 2024-12-18T00:21:36.2211551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T00:21:36.2232683Z Entering 'third_party/FXdiv' 2024-12-18T00:21:36.2292922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T00:21:36.2314244Z Entering 'third_party/NNPACK' 2024-12-18T00:21:36.2374837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T00:21:36.2395931Z Entering 'third_party/NVTX' 2024-12-18T00:21:36.2456669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T00:21:36.2478391Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:21:36.2540264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T00:21:36.2561282Z Entering 'third_party/XNNPACK' 2024-12-18T00:21:36.2622921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T00:21:36.2659256Z Entering 'third_party/benchmark' 2024-12-18T00:21:36.2720982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:21:36.2741986Z Entering 'third_party/composable_kernel' 2024-12-18T00:21:36.2802809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T00:21:36.2829810Z Entering 'third_party/cpp-httplib' 2024-12-18T00:21:36.2890271Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T00:21:36.2911814Z Entering 'third_party/cpuinfo' 2024-12-18T00:21:36.2972360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:21:36.2993729Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:21:36.3054550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T00:21:36.3077327Z Entering 'third_party/cutlass' 2024-12-18T00:21:36.3139615Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:21:36.3169361Z Entering 'third_party/eigen' 2024-12-18T00:21:36.3231496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T00:21:36.3255679Z Entering 'third_party/fbgemm' 2024-12-18T00:21:36.3317369Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T00:21:36.3337103Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:21:36.3402067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T00:21:36.3423331Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:21:36.3486104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:21:36.3507759Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:21:36.3568070Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:21:36.3594923Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:21:36.3656823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.3677000Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:21:36.3737721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T00:21:36.3761054Z Entering 'third_party/flatbuffers' 2024-12-18T00:21:36.3822228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T00:21:36.3846767Z Entering 'third_party/fmt' 2024-12-18T00:21:36.3908518Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:21:36.3930259Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:21:36.3992499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T00:21:36.4014599Z Entering 'third_party/gloo' 2024-12-18T00:21:36.4075670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T00:21:36.4097585Z Entering 'third_party/googletest' 2024-12-18T00:21:36.4158950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.4180218Z Entering 'third_party/ideep' 2024-12-18T00:21:36.4243117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T00:21:36.4262307Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:21:36.4323440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T00:21:36.4353327Z Entering 'third_party/ittapi' 2024-12-18T00:21:36.4415864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T00:21:36.4437222Z Entering 'third_party/kineto' 2024-12-18T00:21:36.4498657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T00:21:36.4518661Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:21:36.4580688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T00:21:36.4600314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:21:36.4663575Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T00:21:36.4685833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:21:36.4748541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T00:21:36.4767968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:21:36.4829536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:21:36.4850530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:21:36.4911709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T00:21:36.4930355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:36.4993436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T00:21:36.5016795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:21:36.5077817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T00:21:36.5098555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:21:36.5162429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.5183850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:21:36.5246178Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T00:21:36.5268549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:21:36.5330734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T00:21:36.5355263Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:21:36.5417948Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T00:21:36.5439431Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:21:36.5501291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.5524614Z Entering 'third_party/mimalloc' 2024-12-18T00:21:36.5586681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T00:21:36.5608979Z Entering 'third_party/nccl/nccl' 2024-12-18T00:21:36.5670554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T00:21:36.5692870Z Entering 'third_party/nlohmann' 2024-12-18T00:21:36.5754946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T00:21:36.5778004Z Entering 'third_party/onnx' 2024-12-18T00:21:36.5840942Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T00:21:36.5875783Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:36.5937029Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:21:36.5963719Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:21:36.6026734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T00:21:36.6048306Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:36.6110157Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:21:36.6131439Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:36.6197201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.6218247Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:36.6278235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T00:21:36.6299578Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:36.6359261Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T00:21:36.6381576Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:36.6443148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T00:21:36.6463643Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:36.6524908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T00:21:36.6545140Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:36.6610747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T00:21:36.6629508Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:36.6691278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T00:21:36.6714777Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:36.6776872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T00:21:36.6803546Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:36.6866635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T00:21:36.6911027Z Entering 'third_party/pocketfft' 2024-12-18T00:21:36.6972807Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T00:21:36.6994421Z Entering 'third_party/protobuf' 2024-12-18T00:21:36.7058066Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T00:21:36.7080351Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:36.7142478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:21:36.7168703Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:36.7229337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.7254191Z Entering 'third_party/psimd' 2024-12-18T00:21:36.7315431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T00:21:36.7336714Z Entering 'third_party/pthreadpool' 2024-12-18T00:21:36.7397420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T00:21:36.7420215Z Entering 'third_party/pybind11' 2024-12-18T00:21:36.7481529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:21:36.7504402Z Entering 'third_party/python-peachpy' 2024-12-18T00:21:36.7565809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T00:21:36.7587752Z Entering 'third_party/sleef' 2024-12-18T00:21:36.7649632Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T00:21:36.7671742Z Entering 'third_party/tensorpipe' 2024-12-18T00:21:36.7734534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T00:21:36.7754740Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:36.7816765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:21:36.7838511Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:36.7898485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T00:21:36.7918654Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:36.7977716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T00:21:36.7998559Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:36.8060375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:21:36.8078900Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:36.8141704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T00:21:36.8810427Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T00:21:36.9168114Z Entering 'android/libs/fbjni' 2024-12-18T00:21:36.9220191Z Entering 'third_party/FP16' 2024-12-18T00:21:36.9272699Z Entering 'third_party/FXdiv' 2024-12-18T00:21:36.9326880Z Entering 'third_party/NNPACK' 2024-12-18T00:21:36.9379704Z Entering 'third_party/NVTX' 2024-12-18T00:21:36.9433690Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:21:36.9486873Z Entering 'third_party/XNNPACK' 2024-12-18T00:21:36.9554580Z Entering 'third_party/benchmark' 2024-12-18T00:21:36.9606952Z Entering 'third_party/composable_kernel' 2024-12-18T00:21:36.9665544Z Entering 'third_party/cpp-httplib' 2024-12-18T00:21:36.9718253Z Entering 'third_party/cpuinfo' 2024-12-18T00:21:36.9771191Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:21:36.9824415Z Entering 'third_party/cutlass' 2024-12-18T00:21:36.9882611Z Entering 'third_party/eigen' 2024-12-18T00:21:36.9936670Z Entering 'third_party/fbgemm' 2024-12-18T00:21:36.9986946Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:21:37.0040024Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:21:37.0091263Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:21:37.0153354Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:21:37.0203785Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:21:37.0254960Z Entering 'third_party/flatbuffers' 2024-12-18T00:21:37.0310976Z Entering 'third_party/fmt' 2024-12-18T00:21:37.0362858Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:21:37.0416251Z Entering 'third_party/gloo' 2024-12-18T00:21:37.0473055Z Entering 'third_party/googletest' 2024-12-18T00:21:37.0525909Z Entering 'third_party/ideep' 2024-12-18T00:21:37.0577188Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:21:37.0638445Z Entering 'third_party/ittapi' 2024-12-18T00:21:37.0690759Z Entering 'third_party/kineto' 2024-12-18T00:21:37.0742667Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:21:37.0795634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:21:37.0848534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:21:37.0901337Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:21:37.0958885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:21:37.1003252Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:37.1059353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:21:37.1112145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:21:37.1166808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:21:37.1219705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:21:37.1271491Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:21:37.1323323Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:21:37.1374869Z Entering 'third_party/mimalloc' 2024-12-18T00:21:37.1429198Z Entering 'third_party/nccl/nccl' 2024-12-18T00:21:37.1481224Z Entering 'third_party/nlohmann' 2024-12-18T00:21:37.1533755Z Entering 'third_party/onnx' 2024-12-18T00:21:37.1598355Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:37.1655710Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:21:37.1708521Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:37.1760503Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:37.1811560Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:37.1860521Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:37.1910665Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:37.1961690Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:37.2011745Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:37.2060098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:37.2113625Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:37.2169884Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:37.2239708Z Entering 'third_party/pocketfft' 2024-12-18T00:21:37.2292236Z Entering 'third_party/protobuf' 2024-12-18T00:21:37.2350385Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:37.2400357Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:37.2455124Z Entering 'third_party/psimd' 2024-12-18T00:21:37.2507821Z Entering 'third_party/pthreadpool' 2024-12-18T00:21:37.2560491Z Entering 'third_party/pybind11' 2024-12-18T00:21:37.2613994Z Entering 'third_party/python-peachpy' 2024-12-18T00:21:37.2666559Z Entering 'third_party/sleef' 2024-12-18T00:21:37.2721577Z Entering 'third_party/tensorpipe' 2024-12-18T00:21:37.2771717Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:37.2821247Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:37.2872445Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:37.2924319Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:37.2973393Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:37.3047893Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T00:21:37.3401921Z Entering 'android/libs/fbjni' 2024-12-18T00:21:37.3453825Z Entering 'third_party/FP16' 2024-12-18T00:21:37.3506304Z Entering 'third_party/FXdiv' 2024-12-18T00:21:37.3557890Z Entering 'third_party/NNPACK' 2024-12-18T00:21:37.3610287Z Entering 'third_party/NVTX' 2024-12-18T00:21:37.3662474Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:21:37.3715111Z Entering 'third_party/XNNPACK' 2024-12-18T00:21:37.3781430Z Entering 'third_party/benchmark' 2024-12-18T00:21:37.3833773Z Entering 'third_party/composable_kernel' 2024-12-18T00:21:37.3892513Z Entering 'third_party/cpp-httplib' 2024-12-18T00:21:37.3947553Z Entering 'third_party/cpuinfo' 2024-12-18T00:21:37.4000486Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:21:37.4051990Z Entering 'third_party/cutlass' 2024-12-18T00:21:37.4110969Z Entering 'third_party/eigen' 2024-12-18T00:21:37.4164341Z Entering 'third_party/fbgemm' 2024-12-18T00:21:37.4219081Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:21:37.4269830Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:21:37.4322131Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:21:37.4377061Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:21:37.4426994Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:21:37.4478889Z Entering 'third_party/flatbuffers' 2024-12-18T00:21:37.4532780Z Entering 'third_party/fmt' 2024-12-18T00:21:37.4583586Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:21:37.4635435Z Entering 'third_party/gloo' 2024-12-18T00:21:37.4686296Z Entering 'third_party/googletest' 2024-12-18T00:21:37.4739740Z Entering 'third_party/ideep' 2024-12-18T00:21:37.4789058Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:21:37.4850119Z Entering 'third_party/ittapi' 2024-12-18T00:21:37.4901350Z Entering 'third_party/kineto' 2024-12-18T00:21:37.4951009Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:21:37.5004146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:21:37.5057003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:21:37.5107572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:21:37.5158053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:21:37.5207345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:21:37.5262299Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:21:37.5313466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:21:37.5364911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:21:37.5417064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:21:37.5470207Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:21:37.5521982Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:21:37.5574756Z Entering 'third_party/mimalloc' 2024-12-18T00:21:37.5626591Z Entering 'third_party/nccl/nccl' 2024-12-18T00:21:37.5677996Z Entering 'third_party/nlohmann' 2024-12-18T00:21:37.5731938Z Entering 'third_party/onnx' 2024-12-18T00:21:37.5797994Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:21:37.5852409Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:21:37.5905575Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:21:37.5956442Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:21:37.6007696Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:21:37.6058109Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:21:37.6107984Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:21:37.6157732Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:21:37.6208623Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:21:37.6257902Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:21:37.6311080Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:21:37.6364809Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:21:37.6434675Z Entering 'third_party/pocketfft' 2024-12-18T00:21:37.6487472Z Entering 'third_party/protobuf' 2024-12-18T00:21:37.6541625Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:21:37.6593758Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:21:37.6649849Z Entering 'third_party/psimd' 2024-12-18T00:21:37.6703639Z Entering 'third_party/pthreadpool' 2024-12-18T00:21:37.6756155Z Entering 'third_party/pybind11' 2024-12-18T00:21:37.6808900Z Entering 'third_party/python-peachpy' 2024-12-18T00:21:37.6860739Z Entering 'third_party/sleef' 2024-12-18T00:21:37.6913527Z Entering 'third_party/tensorpipe' 2024-12-18T00:21:37.6965644Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:21:37.7015259Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:21:37.7065754Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:21:37.7121055Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:21:37.7169151Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:21:37.7243466Z ##[endgroup] 2024-12-18T00:21:37.7280422Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T00:21:37.7305265Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T00:21:37.7503497Z Prepare all required actions 2024-12-18T00:21:37.7503983Z Getting action download info 2024-12-18T00:21:37.9114731Z ##[group]Run ./.github/actions/setup-linux 2024-12-18T00:21:37.9115062Z env: 2024-12-18T00:21:37.9115285Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:37.9115587Z ##[endgroup] 2024-12-18T00:21:37.9161488Z ##[group]Run set -euo pipefail 2024-12-18T00:21:37.9161825Z set -euo pipefail 2024-12-18T00:21:37.9162133Z function get_ec2_metadata() { 2024-12-18T00:21:37.9162520Z  # Pulled from instance metadata endpoint for EC2 2024-12-18T00:21:37.9163148Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-12-18T00:21:37.9163716Z  category=$1 2024-12-18T00:21:37.9164084Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-12-18T00:21:37.9164520Z  runner_name_str=i-096ab043c4c0d1de7 2024-12-18T00:21:37.9164900Z  if [[ -f /.inarc ]]; then 2024-12-18T00:21:37.9165254Z  echo "ARC Runner, no info on ec2 metadata" 2024-12-18T00:21:37.9165640Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-12-18T00:21:37.9166106Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-12-18T00:21:37.9166529Z  else 2024-12-18T00:21:37.9167350Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-12-18T00:21:37.9168214Z  fi 2024-12-18T00:21:37.9168438Z } 2024-12-18T00:21:37.9168708Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-12-18T00:21:37.9169140Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-12-18T00:21:37.9169617Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-12-18T00:21:37.9170034Z echo "system info $(uname -a)" 2024-12-18T00:21:37.9182180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:37.9182552Z env: 2024-12-18T00:21:37.9182766Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:37.9183034Z ##[endgroup] 2024-12-18T00:21:37.9345486Z ami-id: ami-0fff1b9a61dec8a5f 2024-12-18T00:21:37.9456567Z instance-id: i-096ab043c4c0d1de7 2024-12-18T00:21:37.9572184Z instance-type: g5.4xlarge 2024-12-18T00:21:37.9585168Z system info Linux ip-10-0-22-90.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-12-18T00:21:37.9619944Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:21:37.9620918Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:21:37.9630996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:37.9631396Z env: 2024-12-18T00:21:37.9631621Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:37.9631896Z ##[endgroup] 2024-12-18T00:21:37.9703572Z ##[group]Run if systemctl is-active --quiet docker; then 2024-12-18T00:21:37.9704026Z if systemctl is-active --quiet docker; then 2024-12-18T00:21:37.9704418Z  echo "Docker daemon is running..."; 2024-12-18T00:21:37.9704745Z else 2024-12-18T00:21:37.9705106Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-12-18T00:21:37.9705535Z fi 2024-12-18T00:21:37.9714351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:37.9714898Z env: 2024-12-18T00:21:37.9715118Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:37.9715379Z ##[endgroup] 2024-12-18T00:21:37.9805374Z Docker daemon is running... 2024-12-18T00:21:37.9855895Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:21:37.9856199Z with: 2024-12-18T00:21:37.9856407Z shell: bash 2024-12-18T00:21:37.9856835Z timeout_minutes: 5 2024-12-18T00:21:37.9857089Z max_attempts: 3 2024-12-18T00:21:37.9857332Z retry_wait_seconds: 30 2024-12-18T00:21:37.9859486Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-12-18T00:21:37.9861655Z polling_interval_seconds: 1 2024-12-18T00:21:37.9861951Z warning_on_retry: true 2024-12-18T00:21:37.9862218Z continue_on_error: false 2024-12-18T00:21:37.9862485Z env: 2024-12-18T00:21:37.9862712Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:37.9862985Z AWS_RETRY_MODE: standard 2024-12-18T00:21:37.9863256Z AWS_MAX_ATTEMPTS: 5 2024-12-18T00:21:37.9863533Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T00:21:37.9863816Z ##[endgroup] 2024-12-18T00:21:39.2870989Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:21:39.2871606Z Configure a credential helper to remove this warning. See 2024-12-18T00:21:39.2872197Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:21:39.2872581Z 2024-12-18T00:21:39.2872684Z Login Succeeded 2024-12-18T00:21:40.1801846Z Command completed after 1 attempt(s). 2024-12-18T00:21:40.1884138Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:21:40.1884680Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:21:40.1885154Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:21:40.1895751Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:40.1896126Z env: 2024-12-18T00:21:40.1896356Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.1896630Z ##[endgroup] 2024-12-18T00:21:40.1996292Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:21:40.1996863Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:21:40.1997291Z # shellcheck disable=SC2046 2024-12-18T00:21:40.1997631Z docker stop $(docker ps -q) || true 2024-12-18T00:21:40.1997981Z # Prune all of the docker images 2024-12-18T00:21:40.1998312Z docker system prune -af 2024-12-18T00:21:40.2007334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:40.2007760Z env: 2024-12-18T00:21:40.2007988Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.2008261Z ##[endgroup] 2024-12-18T00:21:40.2320180Z "docker stop" requires at least 1 argument. 2024-12-18T00:21:40.2320674Z See 'docker stop --help'. 2024-12-18T00:21:40.2320905Z 2024-12-18T00:21:40.2321108Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T00:21:40.2321386Z 2024-12-18T00:21:40.2321500Z Stop one or more running containers 2024-12-18T00:21:40.2821316Z Total reclaimed space: 0B 2024-12-18T00:21:40.2875107Z ##[group]Run set +e 2024-12-18T00:21:40.2875405Z set +e 2024-12-18T00:21:40.2875665Z set -x 2024-12-18T00:21:40.2875921Z  2024-12-18T00:21:40.2876172Z PT_DOMAIN=download.pytorch.org 2024-12-18T00:21:40.2876749Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-12-18T00:21:40.2877494Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-12-18T00:21:40.2878203Z # one is returned at random 2024-12-18T00:21:40.2878608Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-12-18T00:21:40.2878988Z  2024-12-18T00:21:40.2879413Z if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:21:40.2879843Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-12-18T00:21:40.2880365Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-12-18T00:21:40.2880754Z  2024-12-18T00:21:40.2881010Z  if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:21:40.2881407Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-12-18T00:21:40.2881772Z  exit 1 2024-12-18T00:21:40.2882018Z  fi 2024-12-18T00:21:40.2882243Z fi 2024-12-18T00:21:40.2882462Z  2024-12-18T00:21:40.2882736Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-12-18T00:21:40.2883113Z  # Clean up any old records first 2024-12-18T00:21:40.2883483Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-12-18T00:21:40.2883821Z fi 2024-12-18T00:21:40.2884047Z  2024-12-18T00:21:40.2884458Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-12-18T00:21:40.2884931Z cat /etc/hosts 2024-12-18T00:21:40.2913293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:40.2913671Z env: 2024-12-18T00:21:40.2913890Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.2914157Z ##[endgroup] 2024-12-18T00:21:40.2942639Z + PT_DOMAIN=download.pytorch.org 2024-12-18T00:21:40.2948897Z ++ dig -4 +short download.pytorch.org 2024-12-18T00:21:40.2949889Z ++ tail -n1 2024-12-18T00:21:40.3403705Z + RESOLVED_IP=18.160.18.56 2024-12-18T00:21:40.3403999Z + '[' -z 18.160.18.56 ']' 2024-12-18T00:21:40.3404297Z + grep -r download.pytorch.org /etc/hosts 2024-12-18T00:21:40.3421796Z + echo '18.160.18.56 download.pytorch.org' 2024-12-18T00:21:40.3422557Z + sudo tee -a /etc/hosts 2024-12-18T00:21:40.5705436Z 18.160.18.56 download.pytorch.org 2024-12-18T00:21:40.5727817Z + cat /etc/hosts 2024-12-18T00:21:40.5737594Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-12-18T00:21:40.5743675Z ::1 localhost6 localhost6.localdomain6 2024-12-18T00:21:40.5744196Z 18.160.18.56 download.pytorch.org 2024-12-18T00:21:40.5897554Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T00:21:40.5898052Z with: 2024-12-18T00:21:40.5899225Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.5900176Z docker-build-dir: .ci/docker 2024-12-18T00:21:40.5900474Z working-directory: . 2024-12-18T00:21:40.5900829Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:40.5901248Z force-push: false 2024-12-18T00:21:40.5901491Z env: 2024-12-18T00:21:40.5901716Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.5901995Z ##[endgroup] 2024-12-18T00:21:40.5932613Z ##[group]Run set -ex 2024-12-18T00:21:40.5932913Z set -ex 2024-12-18T00:21:40.5933151Z  2024-12-18T00:21:40.5933548Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T00:21:40.5934244Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T00:21:40.5934823Z # job could then download the pre-built image as usual 2024-12-18T00:21:40.5935336Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T00:21:40.5935813Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5936274Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5936684Z  2024-12-18T00:21:40.5937213Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T00:21:40.5937653Z  exit 0 2024-12-18T00:21:40.5937896Z else 2024-12-18T00:21:40.5938178Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5938517Z fi 2024-12-18T00:21:40.5938740Z  2024-12-18T00:21:40.5939097Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T00:21:40.5939688Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T00:21:40.5940230Z  # use it as it is, but first let's extract the tag 2024-12-18T00:21:40.5940713Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T00:21:40.5941230Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5941716Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5942132Z else 2024-12-18T00:21:40.5942448Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T00:21:40.5942912Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5943541Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.5944097Z fi 2024-12-18T00:21:40.5954102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:40.5954480Z env: 2024-12-18T00:21:40.5954696Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.5954971Z REPO_NAME: pytorch 2024-12-18T00:21:40.5955846Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.5956746Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:21:40.5957127Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:40.5957523Z ##[endgroup] 2024-12-18T00:21:40.5988371Z + [[ ! -d .ci/docker ]] 2024-12-18T00:21:40.5988676Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T00:21:40.5988963Z + echo skip=false 2024-12-18T00:21:40.5990475Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T00:21:40.5997063Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.5997932Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T00:21:40.6024241Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.6024666Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.6025647Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.6064262Z ##[group]Run set +e 2024-12-18T00:21:40.6064590Z set +e 2024-12-18T00:21:40.6064830Z set -x 2024-12-18T00:21:40.6065080Z  2024-12-18T00:21:40.6065299Z login() { 2024-12-18T00:21:40.6065800Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:21:40.6066316Z } 2024-12-18T00:21:40.6066541Z  2024-12-18T00:21:40.6066786Z retry () { 2024-12-18T00:21:40.6067076Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:21:40.6067396Z } 2024-12-18T00:21:40.6067616Z  2024-12-18T00:21:40.6067859Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:21:40.6068177Z  2024-12-18T00:21:40.6068490Z START_TIME=$(date +%s) 2024-12-18T00:21:40.6068796Z # Wait up to 90 minutes 2024-12-18T00:21:40.6069347Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T00:21:40.6069841Z  # Check if image already exists, if it does then skip building it 2024-12-18T00:21:40.6070341Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T00:21:40.6070724Z  exit 0 2024-12-18T00:21:40.6070970Z  fi 2024-12-18T00:21:40.6071192Z  2024-12-18T00:21:40.6071592Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T00:21:40.6072241Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T00:21:40.6072893Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T00:21:40.6073409Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T00:21:40.6073814Z  # It's a Docker build job, let's build the image 2024-12-18T00:21:40.6074168Z  break 2024-12-18T00:21:40.6074415Z  else 2024-12-18T00:21:40.6074755Z  # It's a regular build job, wait for the image to become available 2024-12-18T00:21:40.6075168Z  sleep 300 2024-12-18T00:21:40.6075420Z  fi 2024-12-18T00:21:40.6075667Z done 2024-12-18T00:21:40.6075921Z  2024-12-18T00:21:40.6076273Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T00:21:40.6076844Z # be empty. The default action would be to continue rebuild the image 2024-12-18T00:21:40.6077363Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T00:21:40.6077815Z  # if we're on the base branch then use the parent commit 2024-12-18T00:21:40.6078225Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T00:21:40.6078549Z else 2024-12-18T00:21:40.6078877Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T00:21:40.6079353Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T00:21:40.6079724Z fi 2024-12-18T00:21:40.6079948Z  2024-12-18T00:21:40.6080192Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T00:21:40.6080545Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.6080880Z  2024-12-18T00:21:40.6081517Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T00:21:40.6082062Z  exit 0 2024-12-18T00:21:40.6082296Z fi 2024-12-18T00:21:40.6082507Z  2024-12-18T00:21:40.6082827Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T00:21:40.6083507Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T00:21:40.6084080Z  exit 1 2024-12-18T00:21:40.6084318Z fi 2024-12-18T00:21:40.6084545Z  2024-12-18T00:21:40.6084912Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T00:21:40.6085567Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T00:21:40.6086157Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T00:21:40.6086836Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T00:21:40.6087592Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T00:21:40.6088048Z fi 2024-12-18T00:21:40.6088266Z  2024-12-18T00:21:40.6088539Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:21:40.6097460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:40.6097841Z env: 2024-12-18T00:21:40.6098066Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:40.6098354Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:21:40.6099085Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:21:40.6100154Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.6101066Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:40.6101517Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:40.6101910Z DOCKER_PUSH: 2024-12-18T00:21:40.6102144Z ##[endgroup] 2024-12-18T00:21:40.6129061Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:40.6129507Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:40.6132100Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:21:40.6133204Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:41.1636478Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:21:41.1637145Z Configure a credential helper to remove this warning. See 2024-12-18T00:21:41.1637761Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:21:41.1638189Z 2024-12-18T00:21:41.1638365Z Login Succeeded 2024-12-18T00:21:41.1665660Z ++ date +%s 2024-12-18T00:21:41.1678229Z + START_TIME=1734481301 2024-12-18T00:21:41.1681384Z ++ date +%s 2024-12-18T00:21:41.1691734Z + [[ 1734475901 -lt 1734481301 ]] 2024-12-18T00:21:41.1692677Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:41.4120432Z { 2024-12-18T00:21:41.4121116Z "schemaVersion": 2, 2024-12-18T00:21:41.4121711Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T00:21:41.4122291Z "config": { 2024-12-18T00:21:41.4122644Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T00:21:41.4123072Z "size": 52680, 2024-12-18T00:21:41.4123635Z "digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b" 2024-12-18T00:21:41.4124257Z }, 2024-12-18T00:21:41.4124525Z "layers": [ 2024-12-18T00:21:41.4124821Z { 2024-12-18T00:21:41.4125180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4125645Z "size": 28584506, 2024-12-18T00:21:41.4126546Z "digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733" 2024-12-18T00:21:41.4127139Z }, 2024-12-18T00:21:41.4127332Z { 2024-12-18T00:21:41.4127668Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4128083Z "size": 7944698, 2024-12-18T00:21:41.4128511Z "digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0" 2024-12-18T00:21:41.4128982Z }, 2024-12-18T00:21:41.4129174Z { 2024-12-18T00:21:41.4129500Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4129927Z "size": 57593527, 2024-12-18T00:21:41.4130363Z "digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5" 2024-12-18T00:21:41.4131016Z }, 2024-12-18T00:21:41.4131250Z { 2024-12-18T00:21:41.4131609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4132167Z "size": 185, 2024-12-18T00:21:41.4132735Z "digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc" 2024-12-18T00:21:41.4133324Z }, 2024-12-18T00:21:41.4133516Z { 2024-12-18T00:21:41.4133872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4134461Z "size": 6886, 2024-12-18T00:21:41.4134871Z "digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609" 2024-12-18T00:21:41.4135321Z }, 2024-12-18T00:21:41.4135522Z { 2024-12-18T00:21:41.4135870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4136291Z "size": 1374287670, 2024-12-18T00:21:41.4136911Z "digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd" 2024-12-18T00:21:41.4137365Z }, 2024-12-18T00:21:41.4137564Z { 2024-12-18T00:21:41.4137902Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4138319Z "size": 62641, 2024-12-18T00:21:41.4138738Z "digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5" 2024-12-18T00:21:41.4139205Z }, 2024-12-18T00:21:41.4139399Z { 2024-12-18T00:21:41.4139733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4140148Z "size": 1685, 2024-12-18T00:21:41.4140562Z "digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38" 2024-12-18T00:21:41.4141020Z }, 2024-12-18T00:21:41.4141219Z { 2024-12-18T00:21:41.4141555Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4141969Z "size": 1523, 2024-12-18T00:21:41.4142382Z "digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441" 2024-12-18T00:21:41.4142840Z }, 2024-12-18T00:21:41.4143033Z { 2024-12-18T00:21:41.4143363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4143779Z "size": 2626741021, 2024-12-18T00:21:41.4144201Z "digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc" 2024-12-18T00:21:41.4144662Z }, 2024-12-18T00:21:41.4144858Z { 2024-12-18T00:21:41.4145187Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4145601Z "size": 86545, 2024-12-18T00:21:41.4146002Z "digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4" 2024-12-18T00:21:41.4146497Z }, 2024-12-18T00:21:41.4146689Z { 2024-12-18T00:21:41.4147016Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4147428Z "size": 1824, 2024-12-18T00:21:41.4147846Z "digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb" 2024-12-18T00:21:41.4148480Z }, 2024-12-18T00:21:41.4148676Z { 2024-12-18T00:21:41.4149012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4149519Z "size": 246420139, 2024-12-18T00:21:41.4149958Z "digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653" 2024-12-18T00:21:41.4150416Z }, 2024-12-18T00:21:41.4150614Z { 2024-12-18T00:21:41.4151056Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4151476Z "size": 703, 2024-12-18T00:21:41.4151878Z "digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d" 2024-12-18T00:21:41.4152326Z }, 2024-12-18T00:21:41.4152523Z { 2024-12-18T00:21:41.4152857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4153278Z "size": 1256, 2024-12-18T00:21:41.4153699Z "digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70" 2024-12-18T00:21:41.4154181Z }, 2024-12-18T00:21:41.4154382Z { 2024-12-18T00:21:41.4154723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4155143Z "size": 484, 2024-12-18T00:21:41.4155558Z "digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a" 2024-12-18T00:21:41.4156030Z }, 2024-12-18T00:21:41.4156227Z { 2024-12-18T00:21:41.4156572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4156989Z "size": 91715265, 2024-12-18T00:21:41.4157407Z "digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069" 2024-12-18T00:21:41.4157877Z }, 2024-12-18T00:21:41.4158083Z { 2024-12-18T00:21:41.4158414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4158827Z "size": 3706, 2024-12-18T00:21:41.4159239Z "digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b" 2024-12-18T00:21:41.4159704Z }, 2024-12-18T00:21:41.4159989Z { 2024-12-18T00:21:41.4160321Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4160737Z "size": 1860, 2024-12-18T00:21:41.4161153Z "digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6" 2024-12-18T00:21:41.4161628Z }, 2024-12-18T00:21:41.4161829Z { 2024-12-18T00:21:41.4162164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4162577Z "size": 701, 2024-12-18T00:21:41.4162999Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:21:41.4163476Z }, 2024-12-18T00:21:41.4163671Z { 2024-12-18T00:21:41.4163998Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4164418Z "size": 478, 2024-12-18T00:21:41.4164811Z "digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66" 2024-12-18T00:21:41.4165267Z }, 2024-12-18T00:21:41.4165468Z { 2024-12-18T00:21:41.4165810Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4166229Z "size": 2941869535, 2024-12-18T00:21:41.4166647Z "digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9" 2024-12-18T00:21:41.4167106Z }, 2024-12-18T00:21:41.4167300Z { 2024-12-18T00:21:41.4167635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4168053Z "size": 380, 2024-12-18T00:21:41.4168459Z "digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e" 2024-12-18T00:21:41.4168919Z }, 2024-12-18T00:21:41.4169112Z { 2024-12-18T00:21:41.4169456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4169870Z "size": 12901, 2024-12-18T00:21:41.4170289Z "digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e" 2024-12-18T00:21:41.4170771Z }, 2024-12-18T00:21:41.4170973Z { 2024-12-18T00:21:41.4171312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4171734Z "size": 863, 2024-12-18T00:21:41.4172151Z "digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1" 2024-12-18T00:21:41.4172626Z }, 2024-12-18T00:21:41.4172831Z { 2024-12-18T00:21:41.4173174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4173594Z "size": 106, 2024-12-18T00:21:41.4174123Z "digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a" 2024-12-18T00:21:41.4174615Z }, 2024-12-18T00:21:41.4174823Z { 2024-12-18T00:21:41.4175173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4175607Z "size": 504, 2024-12-18T00:21:41.4176033Z "digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732" 2024-12-18T00:21:41.4176521Z }, 2024-12-18T00:21:41.4176729Z { 2024-12-18T00:21:41.4177074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4177498Z "size": 121477314, 2024-12-18T00:21:41.4177928Z "digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15" 2024-12-18T00:21:41.4178421Z }, 2024-12-18T00:21:41.4178629Z { 2024-12-18T00:21:41.4178972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4179401Z "size": 109, 2024-12-18T00:21:41.4179844Z "digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263" 2024-12-18T00:21:41.4180336Z }, 2024-12-18T00:21:41.4180547Z { 2024-12-18T00:21:41.4180900Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4181339Z "size": 489, 2024-12-18T00:21:41.4181756Z "digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1" 2024-12-18T00:21:41.4182237Z }, 2024-12-18T00:21:41.4182443Z { 2024-12-18T00:21:41.4182787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4183213Z "size": 296, 2024-12-18T00:21:41.4183713Z "digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8" 2024-12-18T00:21:41.4184196Z }, 2024-12-18T00:21:41.4184410Z { 2024-12-18T00:21:41.4184759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4185187Z "size": 103, 2024-12-18T00:21:41.4185608Z "digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486" 2024-12-18T00:21:41.4186094Z }, 2024-12-18T00:21:41.4186305Z { 2024-12-18T00:21:41.4186650Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4187077Z "size": 1473, 2024-12-18T00:21:41.4187502Z "digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f" 2024-12-18T00:21:41.4187984Z }, 2024-12-18T00:21:41.4188277Z { 2024-12-18T00:21:41.4188623Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4189049Z "size": 430258758, 2024-12-18T00:21:41.4189483Z "digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d" 2024-12-18T00:21:41.4189981Z }, 2024-12-18T00:21:41.4190184Z { 2024-12-18T00:21:41.4190520Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4190948Z "size": 164, 2024-12-18T00:21:41.4191352Z "digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b" 2024-12-18T00:21:41.4191828Z }, 2024-12-18T00:21:41.4192036Z { 2024-12-18T00:21:41.4192376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4192797Z "size": 564, 2024-12-18T00:21:41.4193202Z "digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8" 2024-12-18T00:21:41.4193679Z }, 2024-12-18T00:21:41.4193879Z { 2024-12-18T00:21:41.4194212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4194629Z "size": 35865656, 2024-12-18T00:21:41.4195047Z "digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639" 2024-12-18T00:21:41.4195513Z }, 2024-12-18T00:21:41.4195710Z { 2024-12-18T00:21:41.4196050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4196460Z "size": 104, 2024-12-18T00:21:41.4196865Z "digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9" 2024-12-18T00:21:41.4197328Z }, 2024-12-18T00:21:41.4197532Z { 2024-12-18T00:21:41.4197969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4198384Z "size": 425, 2024-12-18T00:21:41.4198785Z "digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa" 2024-12-18T00:21:41.4199590Z }, 2024-12-18T00:21:41.4199802Z { 2024-12-18T00:21:41.4200161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4200587Z "size": 20262160, 2024-12-18T00:21:41.4201033Z "digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1" 2024-12-18T00:21:41.4201529Z }, 2024-12-18T00:21:41.4201742Z { 2024-12-18T00:21:41.4202085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4202499Z "size": 643, 2024-12-18T00:21:41.4202935Z "digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba" 2024-12-18T00:21:41.4203426Z }, 2024-12-18T00:21:41.4203644Z { 2024-12-18T00:21:41.4204024Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4204485Z "size": 701, 2024-12-18T00:21:41.4204925Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:21:41.4205414Z }, 2024-12-18T00:21:41.4205630Z { 2024-12-18T00:21:41.4206006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4206450Z "size": 142, 2024-12-18T00:21:41.4206868Z "digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d" 2024-12-18T00:21:41.4207348Z }, 2024-12-18T00:21:41.4207720Z { 2024-12-18T00:21:41.4208068Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4208481Z "size": 135, 2024-12-18T00:21:41.4208902Z "digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69" 2024-12-18T00:21:41.4209375Z }, 2024-12-18T00:21:41.4209585Z { 2024-12-18T00:21:41.4209933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4210359Z "size": 4995482131, 2024-12-18T00:21:41.4210807Z "digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2" 2024-12-18T00:21:41.4211291Z }, 2024-12-18T00:21:41.4211493Z { 2024-12-18T00:21:41.4211948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4212418Z "size": 196, 2024-12-18T00:21:41.4213132Z "digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348" 2024-12-18T00:21:41.4213684Z }, 2024-12-18T00:21:41.4214045Z { 2024-12-18T00:21:41.4214482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4214994Z "size": 564, 2024-12-18T00:21:41.4215583Z "digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1" 2024-12-18T00:21:41.4216156Z }, 2024-12-18T00:21:41.4216446Z { 2024-12-18T00:21:41.4216926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4217446Z "size": 43164612, 2024-12-18T00:21:41.4236543Z "digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f" 2024-12-18T00:21:41.4237019Z }, 2024-12-18T00:21:41.4237217Z { 2024-12-18T00:21:41.4237554Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4237976Z "size": 106, 2024-12-18T00:21:41.4238400Z "digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84" 2024-12-18T00:21:41.4238865Z }, 2024-12-18T00:21:41.4239059Z { 2024-12-18T00:21:41.4239391Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4239810Z "size": 1402, 2024-12-18T00:21:41.4240250Z "digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa" 2024-12-18T00:21:41.4240702Z }, 2024-12-18T00:21:41.4240898Z { 2024-12-18T00:21:41.4241226Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4241632Z "size": 701, 2024-12-18T00:21:41.4242223Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:21:41.4242690Z }, 2024-12-18T00:21:41.4242880Z { 2024-12-18T00:21:41.4243205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4243616Z "size": 137, 2024-12-18T00:21:41.4244012Z "digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391" 2024-12-18T00:21:41.4244457Z }, 2024-12-18T00:21:41.4244647Z { 2024-12-18T00:21:41.4244973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4245387Z "size": 120, 2024-12-18T00:21:41.4245791Z "digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575" 2024-12-18T00:21:41.4246240Z }, 2024-12-18T00:21:41.4246434Z { 2024-12-18T00:21:41.4246760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4247168Z "size": 4886226234, 2024-12-18T00:21:41.4247599Z "digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc" 2024-12-18T00:21:41.4248050Z }, 2024-12-18T00:21:41.4248242Z { 2024-12-18T00:21:41.4248572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4248985Z "size": 174, 2024-12-18T00:21:41.4249396Z "digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef" 2024-12-18T00:21:41.4249855Z }, 2024-12-18T00:21:41.4250050Z { 2024-12-18T00:21:41.4250379Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4250795Z "size": 908, 2024-12-18T00:21:41.4251294Z "digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474" 2024-12-18T00:21:41.4251746Z }, 2024-12-18T00:21:41.4251943Z { 2024-12-18T00:21:41.4252274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4252694Z "size": 701, 2024-12-18T00:21:41.4253118Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:21:41.4253583Z }, 2024-12-18T00:21:41.4253773Z { 2024-12-18T00:21:41.4254106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4254521Z "size": 135, 2024-12-18T00:21:41.4254924Z "digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084" 2024-12-18T00:21:41.4255367Z }, 2024-12-18T00:21:41.4255559Z { 2024-12-18T00:21:41.4255896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4256307Z "size": 32, 2024-12-18T00:21:41.4256717Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:21:41.4257180Z }, 2024-12-18T00:21:41.4257373Z { 2024-12-18T00:21:41.4257706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4258110Z "size": 159, 2024-12-18T00:21:41.4258511Z "digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6" 2024-12-18T00:21:41.4258957Z }, 2024-12-18T00:21:41.4259156Z { 2024-12-18T00:21:41.4259486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4259891Z "size": 1932, 2024-12-18T00:21:41.4260284Z "digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492" 2024-12-18T00:21:41.4260720Z }, 2024-12-18T00:21:41.4260918Z { 2024-12-18T00:21:41.4261248Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4261657Z "size": 196810750, 2024-12-18T00:21:41.4262074Z "digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e" 2024-12-18T00:21:41.4262531Z }, 2024-12-18T00:21:41.4262727Z { 2024-12-18T00:21:41.4263058Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4263461Z "size": 165, 2024-12-18T00:21:41.4263856Z "digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06" 2024-12-18T00:21:41.4264307Z }, 2024-12-18T00:21:41.4264499Z { 2024-12-18T00:21:41.4264913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4265322Z "size": 7943, 2024-12-18T00:21:41.4265719Z "digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0" 2024-12-18T00:21:41.4266188Z }, 2024-12-18T00:21:41.4266406Z { 2024-12-18T00:21:41.4266736Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4267147Z "size": 8070, 2024-12-18T00:21:41.4267554Z "digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39" 2024-12-18T00:21:41.4268023Z }, 2024-12-18T00:21:41.4268296Z { 2024-12-18T00:21:41.4268643Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4269051Z "size": 303, 2024-12-18T00:21:41.4269471Z "digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d" 2024-12-18T00:21:41.4269942Z }, 2024-12-18T00:21:41.4270133Z { 2024-12-18T00:21:41.4270461Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4270873Z "size": 7633763, 2024-12-18T00:21:41.4271285Z "digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2" 2024-12-18T00:21:41.4271751Z }, 2024-12-18T00:21:41.4271948Z { 2024-12-18T00:21:41.4272272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4272697Z "size": 108, 2024-12-18T00:21:41.4273102Z "digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598" 2024-12-18T00:21:41.4273571Z }, 2024-12-18T00:21:41.4273760Z { 2024-12-18T00:21:41.4274173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4274583Z "size": 54145665, 2024-12-18T00:21:41.4274990Z "digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362" 2024-12-18T00:21:41.4275440Z }, 2024-12-18T00:21:41.4275632Z { 2024-12-18T00:21:41.4275959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4276374Z "size": 484, 2024-12-18T00:21:41.4276782Z "digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9" 2024-12-18T00:21:41.4277238Z }, 2024-12-18T00:21:41.4277427Z { 2024-12-18T00:21:41.4277757Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4278167Z "size": 1374859271, 2024-12-18T00:21:41.4278582Z "digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2" 2024-12-18T00:21:41.4279037Z }, 2024-12-18T00:21:41.4279230Z { 2024-12-18T00:21:41.4279559Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4279976Z "size": 106, 2024-12-18T00:21:41.4280382Z "digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4" 2024-12-18T00:21:41.4280844Z }, 2024-12-18T00:21:41.4281035Z { 2024-12-18T00:21:41.4281363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4281773Z "size": 568, 2024-12-18T00:21:41.4282177Z "digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e" 2024-12-18T00:21:41.4282632Z }, 2024-12-18T00:21:41.4282823Z { 2024-12-18T00:21:41.4283161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4283611Z "size": 303706071, 2024-12-18T00:21:41.4284109Z "digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c" 2024-12-18T00:21:41.4284569Z }, 2024-12-18T00:21:41.4284759Z { 2024-12-18T00:21:41.4285093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4285511Z "size": 111, 2024-12-18T00:21:41.4285918Z "digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6" 2024-12-18T00:21:41.4286380Z }, 2024-12-18T00:21:41.4286576Z { 2024-12-18T00:21:41.4286909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4287323Z "size": 529, 2024-12-18T00:21:41.4287826Z "digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3" 2024-12-18T00:21:41.4288296Z }, 2024-12-18T00:21:41.4288485Z { 2024-12-18T00:21:41.4288818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4289236Z "size": 48289052, 2024-12-18T00:21:41.4289657Z "digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab" 2024-12-18T00:21:41.4290107Z }, 2024-12-18T00:21:41.4290301Z { 2024-12-18T00:21:41.4290637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4291053Z "size": 106, 2024-12-18T00:21:41.4291458Z "digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676" 2024-12-18T00:21:41.4291920Z }, 2024-12-18T00:21:41.4292115Z { 2024-12-18T00:21:41.4292439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4292852Z "size": 32, 2024-12-18T00:21:41.4293277Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:21:41.4293732Z }, 2024-12-18T00:21:41.4294147Z { 2024-12-18T00:21:41.4294480Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4294888Z "size": 32, 2024-12-18T00:21:41.4295295Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:21:41.4295761Z }, 2024-12-18T00:21:41.4295951Z { 2024-12-18T00:21:41.4296281Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4296689Z "size": 32, 2024-12-18T00:21:41.4297186Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:21:41.4297648Z }, 2024-12-18T00:21:41.4297842Z { 2024-12-18T00:21:41.4298164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:21:41.4298573Z "size": 32, 2024-12-18T00:21:41.4299175Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:21:41.4299647Z } 2024-12-18T00:21:41.4299845Z ] 2024-12-18T00:21:41.4300034Z } 2024-12-18T00:21:41.4300238Z + exit 0 2024-12-18T00:21:41.4339962Z ##[group]Run set -eux 2024-12-18T00:21:41.4340258Z set -eux 2024-12-18T00:21:41.4341096Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T00:21:41.4350772Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:41.4351159Z env: 2024-12-18T00:21:41.4351390Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:41.4351660Z ##[endgroup] 2024-12-18T00:21:41.4383346Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T00:21:41.4384402Z + jq --raw-output .SecretString 2024-12-18T00:21:41.4386355Z + jq -r .docker_hub_readonly_token 2024-12-18T00:21:41.4386860Z + docker login --username pytorchbot --password-stdin 2024-12-18T00:21:42.0564452Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:21:42.0565061Z Configure a credential helper to remove this warning. See 2024-12-18T00:21:42.0565635Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:21:42.0566028Z 2024-12-18T00:21:42.0566185Z Login Succeeded 2024-12-18T00:21:42.0662221Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:21:42.0662597Z tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:21:42.0663012Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-12-18T00:21:42.0671971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:42.0672350Z env: 2024-12-18T00:21:42.0672574Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:42.0673431Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.0674294Z ##[endgroup] 2024-12-18T00:21:42.0705067Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.0760339Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T00:21:42.0760808Z with: 2024-12-18T00:21:42.0761606Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.0762567Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:42.0762971Z env: 2024-12-18T00:21:42.0763196Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:42.0763470Z ##[endgroup] 2024-12-18T00:21:42.0789467Z ##[group]Run set -x 2024-12-18T00:21:42.0789747Z set -x 2024-12-18T00:21:42.0789985Z set +e 2024-12-18T00:21:42.0790206Z  2024-12-18T00:21:42.0790424Z login() { 2024-12-18T00:21:42.0790907Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:21:42.0791417Z } 2024-12-18T00:21:42.0791633Z  2024-12-18T00:21:42.0791876Z retry () { 2024-12-18T00:21:42.0792153Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:21:42.0792461Z } 2024-12-18T00:21:42.0792674Z  2024-12-18T00:21:42.0792920Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:21:42.0793229Z  2024-12-18T00:21:42.0793447Z set -e 2024-12-18T00:21:42.0793785Z # ignore output since only exit code is used for conditional 2024-12-18T00:21:42.0794448Z # only pull docker image if it's not available locally 2024-12-18T00:21:42.0795012Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T00:21:42.0795510Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T00:21:42.0795836Z fi 2024-12-18T00:21:42.0804976Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:21:42.0805339Z env: 2024-12-18T00:21:42.0805561Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:21:42.0806415Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.0807371Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:42.0807756Z ##[endgroup] 2024-12-18T00:21:42.0837618Z + set +e 2024-12-18T00:21:42.0837945Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:42.0838391Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:42.0841155Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:21:42.0843076Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:21:42.6392783Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:21:42.6393479Z Configure a credential helper to remove this warning. See 2024-12-18T00:21:42.6394119Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:21:42.6394571Z 2024-12-18T00:21:42.6394701Z Login Succeeded 2024-12-18T00:21:42.6416110Z + set -e 2024-12-18T00:21:42.6417072Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.6578660Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.6580133Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:21:42.9198280Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks 2024-12-18T00:21:42.9199578Z 80888bc6716f: Pulling fs layer 2024-12-18T00:21:42.9199986Z fbcd35dc5bc3: Pulling fs layer 2024-12-18T00:21:42.9200713Z c7232af9ae05: Pulling fs layer 2024-12-18T00:21:42.9201131Z db6cdef1932a: Pulling fs layer 2024-12-18T00:21:42.9201546Z 56dc85502937: Pulling fs layer 2024-12-18T00:21:42.9201939Z 30c0ea6140d0: Pulling fs layer 2024-12-18T00:21:42.9202343Z 71bdb1a72c2d: Pulling fs layer 2024-12-18T00:21:42.9202743Z 4829486be7c3: Pulling fs layer 2024-12-18T00:21:42.9203148Z 1f4e68d7b5e4: Pulling fs layer 2024-12-18T00:21:42.9203592Z 7c373e2d9b7e: Pulling fs layer 2024-12-18T00:21:42.9203974Z 622381141745: Pulling fs layer 2024-12-18T00:21:42.9204367Z f003d065e31e: Pulling fs layer 2024-12-18T00:21:42.9204762Z db6cdef1932a: Waiting 2024-12-18T00:21:42.9205140Z cd6ce7116815: Pulling fs layer 2024-12-18T00:21:42.9205506Z 56dc85502937: Waiting 2024-12-18T00:21:42.9205846Z e843066a8f68: Pulling fs layer 2024-12-18T00:21:42.9206265Z bac50ca2930b: Pulling fs layer 2024-12-18T00:21:42.9206669Z 57e52c6751f4: Pulling fs layer 2024-12-18T00:21:42.9207065Z 30c0ea6140d0: Waiting 2024-12-18T00:21:42.9207442Z a9c5c14d77c0: Pulling fs layer 2024-12-18T00:21:42.9207844Z b86099b8619a: Pulling fs layer 2024-12-18T00:21:42.9208255Z 3c8ea8a600ff: Pulling fs layer 2024-12-18T00:21:42.9208658Z 71bdb1a72c2d: Waiting 2024-12-18T00:21:42.9209062Z bebd539a1d59: Pulling fs layer 2024-12-18T00:21:42.9209472Z 292b18ec2dab: Pulling fs layer 2024-12-18T00:21:42.9209872Z 76e3817b4b4c: Pulling fs layer 2024-12-18T00:21:42.9210242Z 50b1f245e9f5: Pulling fs layer 2024-12-18T00:21:42.9210888Z ad93cf5205f2: Pulling fs layer 2024-12-18T00:21:42.9211285Z 1f4e68d7b5e4: Waiting 2024-12-18T00:21:42.9211649Z 8d17c59eaf7d: Pulling fs layer 2024-12-18T00:21:42.9212060Z b36a8ccf5c8c: Pulling fs layer 2024-12-18T00:21:42.9212470Z aa0a3b87e9e3: Pulling fs layer 2024-12-18T00:21:42.9212880Z 72a632052c5b: Pulling fs layer 2024-12-18T00:21:42.9213272Z e843066a8f68: Waiting 2024-12-18T00:21:42.9213624Z 7c373e2d9b7e: Waiting 2024-12-18T00:21:42.9213968Z f003d065e31e: Waiting 2024-12-18T00:21:42.9214323Z ceed6316bc24: Pulling fs layer 2024-12-18T00:21:42.9214738Z d0ca0f491f80: Pulling fs layer 2024-12-18T00:21:42.9215056Z cd6ce7116815: Waiting 2024-12-18T00:21:42.9215317Z cf02c706cd56: Pulling fs layer 2024-12-18T00:21:42.9215599Z 16073d3f107d: Pulling fs layer 2024-12-18T00:21:42.9215882Z d234faf2f1bf: Pulling fs layer 2024-12-18T00:21:42.9216180Z 3f0ade86af1a: Pulling fs layer 2024-12-18T00:21:42.9216484Z bac50ca2930b: Waiting 2024-12-18T00:21:42.9216765Z 034b6a03f5f7: Pulling fs layer 2024-12-18T00:21:42.9217053Z 72a632052c5b: Waiting 2024-12-18T00:21:42.9217294Z 50b1f245e9f5: Waiting 2024-12-18T00:21:42.9217544Z 21f8f066852d: Pulling fs layer 2024-12-18T00:21:42.9217827Z 9dc1f315a979: Pulling fs layer 2024-12-18T00:21:42.9218105Z ad93cf5205f2: Waiting 2024-12-18T00:21:42.9218351Z cd6a6063178b: Pulling fs layer 2024-12-18T00:21:42.9218630Z 2116738f4ea9: Pulling fs layer 2024-12-18T00:21:42.9218905Z cf02c706cd56: Waiting 2024-12-18T00:21:42.9219157Z d7fa134316d9: Pulling fs layer 2024-12-18T00:21:42.9219440Z af03dc3fd15e: Pulling fs layer 2024-12-18T00:21:42.9219724Z fd9de3dc4258: Pulling fs layer 2024-12-18T00:21:42.9220004Z ceed6316bc24: Waiting 2024-12-18T00:21:42.9220256Z b85093661f77: Pulling fs layer 2024-12-18T00:21:42.9220530Z 76e3817b4b4c: Waiting 2024-12-18T00:21:42.9220775Z d234faf2f1bf: Waiting 2024-12-18T00:21:42.9221030Z ab6ef6c2ac4d: Pulling fs layer 2024-12-18T00:21:42.9221314Z 56c896077120: Pulling fs layer 2024-12-18T00:21:42.9221596Z d88e9c5e7c70: Pulling fs layer 2024-12-18T00:21:42.9221877Z 3f0ade86af1a: Waiting 2024-12-18T00:21:42.9222120Z aa0a3b87e9e3: Waiting 2024-12-18T00:21:42.9222370Z 53617ee83302: Pulling fs layer 2024-12-18T00:21:42.9222641Z b36a8ccf5c8c: Waiting 2024-12-18T00:21:42.9222896Z 3edbde10d404: Pulling fs layer 2024-12-18T00:21:42.9223179Z f4038f7524c3: Pulling fs layer 2024-12-18T00:21:42.9223444Z 2116738f4ea9: Waiting 2024-12-18T00:21:42.9223682Z af03dc3fd15e: Waiting 2024-12-18T00:21:42.9223940Z 4a51173f7757: Pulling fs layer 2024-12-18T00:21:42.9224223Z ed6a71d41274: Pulling fs layer 2024-12-18T00:21:42.9224642Z 389dc939e2a4: Pulling fs layer 2024-12-18T00:21:42.9224916Z 21f8f066852d: Waiting 2024-12-18T00:21:42.9225163Z 0341eaeed33d: Pulling fs layer 2024-12-18T00:21:42.9225447Z 537cdb2d3168: Pulling fs layer 2024-12-18T00:21:42.9225724Z bebd539a1d59: Waiting 2024-12-18T00:21:42.9225977Z b85093661f77: Waiting 2024-12-18T00:21:42.9226217Z 3edbde10d404: Waiting 2024-12-18T00:21:42.9226481Z 76d4a0702497: Pulling fs layer 2024-12-18T00:21:42.9226778Z ab6ef6c2ac4d: Waiting 2024-12-18T00:21:42.9227048Z 4f4fb700ef54: Pulling fs layer 2024-12-18T00:21:42.9227338Z 3287a539d61e: Pulling fs layer 2024-12-18T00:21:42.9227617Z 9b763e24a7d4: Pulling fs layer 2024-12-18T00:21:42.9227891Z f4038f7524c3: Waiting 2024-12-18T00:21:42.9228155Z f6ef3d4c814e: Pulling fs layer 2024-12-18T00:21:42.9228551Z 56c896077120: Waiting 2024-12-18T00:21:42.9228806Z 06914b00eb49: Pulling fs layer 2024-12-18T00:21:42.9229089Z 724553524224: Pulling fs layer 2024-12-18T00:21:42.9229370Z 47d9fb6dae26: Pulling fs layer 2024-12-18T00:21:42.9229647Z d88e9c5e7c70: Waiting 2024-12-18T00:21:42.9229913Z cd6a6063178b: Waiting 2024-12-18T00:21:42.9230156Z 53617ee83302: Waiting 2024-12-18T00:21:42.9230397Z 16073d3f107d: Waiting 2024-12-18T00:21:42.9230635Z d7fa134316d9: Waiting 2024-12-18T00:21:42.9230877Z a9c5c14d77c0: Waiting 2024-12-18T00:21:42.9231137Z acce7b690dcb: Pulling fs layer 2024-12-18T00:21:42.9231415Z 034b6a03f5f7: Waiting 2024-12-18T00:21:42.9231668Z 58dc9f64def7: Pulling fs layer 2024-12-18T00:21:42.9232021Z 4a51173f7757: Waiting 2024-12-18T00:21:42.9232280Z 94dfe5ef1b7e: Pulling fs layer 2024-12-18T00:21:42.9232570Z 40a2c9135b06: Pulling fs layer 2024-12-18T00:21:42.9232851Z 8839d6b0c821: Pulling fs layer 2024-12-18T00:21:42.9233118Z 537cdb2d3168: Waiting 2024-12-18T00:21:42.9233375Z 1c588f0a9e97: Pulling fs layer 2024-12-18T00:21:42.9233658Z 6727e7e27936: Pulling fs layer 2024-12-18T00:21:42.9233929Z f6ef3d4c814e: Waiting 2024-12-18T00:21:42.9234193Z 3a55100d1957: Pulling fs layer 2024-12-18T00:21:42.9234468Z 3d6f160d3351: Pulling fs layer 2024-12-18T00:21:42.9234762Z b2a8bb947a87: Pulling fs layer 2024-12-18T00:21:42.9235115Z acce7b690dcb: Waiting 2024-12-18T00:21:42.9235703Z 9dc1f315a979: Waiting 2024-12-18T00:21:42.9236033Z 69a74b2360f4: Pulling fs layer 2024-12-18T00:21:42.9236446Z 724553524224: Waiting 2024-12-18T00:21:42.9236880Z f9e12a253d89: Pulling fs layer 2024-12-18T00:21:42.9237257Z ed6a71d41274: Waiting 2024-12-18T00:21:42.9237613Z 0341eaeed33d: Waiting 2024-12-18T00:21:42.9237992Z 8fdb06d51b96: Pulling fs layer 2024-12-18T00:21:42.9255646Z 6727e7e27936: Waiting 2024-12-18T00:21:42.9255912Z 1c588f0a9e97: Waiting 2024-12-18T00:21:42.9256170Z 58dc9f64def7: Waiting 2024-12-18T00:21:42.9256411Z 06914b00eb49: Waiting 2024-12-18T00:21:42.9256661Z 8fdb06d51b96: Waiting 2024-12-18T00:21:42.9256914Z 69a74b2360f4: Waiting 2024-12-18T00:21:42.9257161Z 3d6f160d3351: Waiting 2024-12-18T00:21:42.9257409Z 8839d6b0c821: Waiting 2024-12-18T00:21:42.9257646Z 40a2c9135b06: Waiting 2024-12-18T00:21:42.9257890Z 47d9fb6dae26: Waiting 2024-12-18T00:21:42.9258145Z 3287a539d61e: Waiting 2024-12-18T00:21:42.9258386Z 622381141745: Waiting 2024-12-18T00:21:43.0470331Z fbcd35dc5bc3: Verifying Checksum 2024-12-18T00:21:43.0470695Z fbcd35dc5bc3: Download complete 2024-12-18T00:21:43.2052153Z 56dc85502937: Verifying Checksum 2024-12-18T00:21:43.2052479Z 56dc85502937: Download complete 2024-12-18T00:21:43.2778134Z 80888bc6716f: Download complete 2024-12-18T00:21:43.3709564Z 71bdb1a72c2d: Download complete 2024-12-18T00:21:43.4548484Z 4829486be7c3: Download complete 2024-12-18T00:21:43.5340833Z 1f4e68d7b5e4: Verifying Checksum 2024-12-18T00:21:43.5341299Z 1f4e68d7b5e4: Download complete 2024-12-18T00:21:43.5631831Z c7232af9ae05: Verifying Checksum 2024-12-18T00:21:43.5632294Z c7232af9ae05: Download complete 2024-12-18T00:21:43.6406264Z 622381141745: Verifying Checksum 2024-12-18T00:21:43.6406720Z 622381141745: Download complete 2024-12-18T00:21:43.7113867Z f003d065e31e: Verifying Checksum 2024-12-18T00:21:43.7114226Z f003d065e31e: Download complete 2024-12-18T00:21:44.3778910Z 80888bc6716f: Pull complete 2024-12-18T00:21:44.6577989Z fbcd35dc5bc3: Pull complete 2024-12-18T00:21:45.4040171Z c7232af9ae05: Pull complete 2024-12-18T00:21:45.4286477Z db6cdef1932a: Pull complete 2024-12-18T00:21:45.4504199Z 56dc85502937: Pull complete 2024-12-18T00:21:46.2434983Z cd6ce7116815: Verifying Checksum 2024-12-18T00:21:46.2435347Z cd6ce7116815: Download complete 2024-12-18T00:21:46.3526575Z e843066a8f68: Verifying Checksum 2024-12-18T00:21:46.3526931Z e843066a8f68: Download complete 2024-12-18T00:21:46.4495698Z bac50ca2930b: Verifying Checksum 2024-12-18T00:21:46.4496043Z bac50ca2930b: Download complete 2024-12-18T00:21:46.5332899Z 57e52c6751f4: Verifying Checksum 2024-12-18T00:21:46.5333245Z 57e52c6751f4: Download complete 2024-12-18T00:21:47.5196779Z a9c5c14d77c0: Verifying Checksum 2024-12-18T00:21:47.5197153Z a9c5c14d77c0: Download complete 2024-12-18T00:21:47.7553678Z b86099b8619a: Verifying Checksum 2024-12-18T00:21:47.7554175Z b86099b8619a: Download complete 2024-12-18T00:21:47.8598084Z 3c8ea8a600ff: Verifying Checksum 2024-12-18T00:21:47.8599057Z 3c8ea8a600ff: Download complete 2024-12-18T00:21:47.9488907Z bebd539a1d59: Verifying Checksum 2024-12-18T00:21:48.2101998Z 292b18ec2dab: Verifying Checksum 2024-12-18T00:21:48.2102461Z 292b18ec2dab: Download complete 2024-12-18T00:21:56.9967225Z 30c0ea6140d0: Verifying Checksum 2024-12-18T00:21:56.9967591Z 30c0ea6140d0: Download complete 2024-12-18T00:21:57.0740712Z 50b1f245e9f5: Verifying Checksum 2024-12-18T00:21:57.0741517Z 50b1f245e9f5: Download complete 2024-12-18T00:21:57.1599768Z ad93cf5205f2: Verifying Checksum 2024-12-18T00:21:57.1600138Z ad93cf5205f2: Download complete 2024-12-18T00:21:57.3018253Z 8d17c59eaf7d: Download complete 2024-12-18T00:21:57.4287391Z b36a8ccf5c8c: Verifying Checksum 2024-12-18T00:21:57.4287747Z b36a8ccf5c8c: Download complete 2024-12-18T00:21:57.5231319Z aa0a3b87e9e3: Verifying Checksum 2024-12-18T00:21:57.5231696Z aa0a3b87e9e3: Download complete 2024-12-18T00:21:58.7896089Z 72a632052c5b: Verifying Checksum 2024-12-18T00:21:58.7896627Z 72a632052c5b: Download complete 2024-12-18T00:21:58.8844708Z ceed6316bc24: Verifying Checksum 2024-12-18T00:21:58.8845198Z ceed6316bc24: Download complete 2024-12-18T00:21:58.9766653Z d0ca0f491f80: Verifying Checksum 2024-12-18T00:21:58.9767145Z d0ca0f491f80: Download complete 2024-12-18T00:21:59.0564483Z cf02c706cd56: Verifying Checksum 2024-12-18T00:21:59.0565829Z cf02c706cd56: Download complete 2024-12-18T00:21:59.1555405Z 16073d3f107d: Verifying Checksum 2024-12-18T00:21:59.1555879Z 16073d3f107d: Download complete 2024-12-18T00:21:59.3258001Z d234faf2f1bf: Download complete 2024-12-18T00:22:03.6891555Z 3f0ade86af1a: Verifying Checksum 2024-12-18T00:22:03.6892030Z 3f0ade86af1a: Download complete 2024-12-18T00:22:03.8497422Z 034b6a03f5f7: Verifying Checksum 2024-12-18T00:22:03.8497894Z 034b6a03f5f7: Download complete 2024-12-18T00:22:03.9656734Z 21f8f066852d: Verifying Checksum 2024-12-18T00:22:03.9657213Z 21f8f066852d: Download complete 2024-12-18T00:22:04.4130817Z 9dc1f315a979: Verifying Checksum 2024-12-18T00:22:04.4854574Z 9dc1f315a979: Download complete 2024-12-18T00:22:04.4854920Z cd6a6063178b: Verifying Checksum 2024-12-18T00:22:04.4855225Z cd6a6063178b: Download complete 2024-12-18T00:22:04.5785738Z 2116738f4ea9: Verifying Checksum 2024-12-18T00:22:04.5786098Z 2116738f4ea9: Download complete 2024-12-18T00:22:04.8374473Z d7fa134316d9: Verifying Checksum 2024-12-18T00:22:04.8374861Z d7fa134316d9: Download complete 2024-12-18T00:22:04.9281650Z af03dc3fd15e: Verifying Checksum 2024-12-18T00:22:04.9282053Z af03dc3fd15e: Download complete 2024-12-18T00:22:05.0061801Z fd9de3dc4258: Verifying Checksum 2024-12-18T00:22:05.0062155Z fd9de3dc4258: Download complete 2024-12-18T00:22:05.0946584Z b85093661f77: Verifying Checksum 2024-12-18T00:22:05.0946983Z b85093661f77: Download complete 2024-12-18T00:22:09.2484113Z 30c0ea6140d0: Pull complete 2024-12-18T00:22:09.4717496Z 71bdb1a72c2d: Pull complete 2024-12-18T00:22:09.7010581Z 4829486be7c3: Pull complete 2024-12-18T00:22:09.8517758Z 7c373e2d9b7e: Verifying Checksum 2024-12-18T00:22:09.8518380Z 7c373e2d9b7e: Download complete 2024-12-18T00:22:09.9382431Z 1f4e68d7b5e4: Pull complete 2024-12-18T00:22:09.9693171Z 56c896077120: Verifying Checksum 2024-12-18T00:22:09.9693627Z 56c896077120: Download complete 2024-12-18T00:22:10.0635966Z d88e9c5e7c70: Verifying Checksum 2024-12-18T00:22:10.0636453Z d88e9c5e7c70: Download complete 2024-12-18T00:22:10.5490032Z 53617ee83302: Verifying Checksum 2024-12-18T00:22:10.5490509Z 53617ee83302: Download complete 2024-12-18T00:22:10.6618910Z 3edbde10d404: Verifying Checksum 2024-12-18T00:22:10.6619385Z 3edbde10d404: Download complete 2024-12-18T00:22:10.7585597Z f4038f7524c3: Verifying Checksum 2024-12-18T00:22:10.7589645Z f4038f7524c3: Download complete 2024-12-18T00:22:10.8575661Z 4a51173f7757: Verifying Checksum 2024-12-18T00:22:10.8578085Z 4a51173f7757: Download complete 2024-12-18T00:22:10.9323542Z ed6a71d41274: Download complete 2024-12-18T00:22:17.6998483Z 76e3817b4b4c: Verifying Checksum 2024-12-18T00:22:17.6999249Z 76e3817b4b4c: Download complete 2024-12-18T00:22:17.7763218Z 0341eaeed33d: Verifying Checksum 2024-12-18T00:22:17.7765584Z 0341eaeed33d: Download complete 2024-12-18T00:22:17.8564001Z 537cdb2d3168: Verifying Checksum 2024-12-18T00:22:17.8564487Z 537cdb2d3168: Download complete 2024-12-18T00:22:17.9480917Z 76d4a0702497: Download complete 2024-12-18T00:22:17.9559960Z 4f4fb700ef54: Verifying Checksum 2024-12-18T00:22:17.9560413Z 4f4fb700ef54: Download complete 2024-12-18T00:22:18.0469148Z 3287a539d61e: Verifying Checksum 2024-12-18T00:22:18.0469542Z 3287a539d61e: Download complete 2024-12-18T00:22:18.1269983Z 9b763e24a7d4: Verifying Checksum 2024-12-18T00:22:18.1270445Z 9b763e24a7d4: Download complete 2024-12-18T00:22:20.2484028Z f6ef3d4c814e: Verifying Checksum 2024-12-18T00:22:20.2484513Z f6ef3d4c814e: Download complete 2024-12-18T00:22:20.3465297Z 06914b00eb49: Download complete 2024-12-18T00:22:20.4472250Z 724553524224: Verifying Checksum 2024-12-18T00:22:20.4472807Z 724553524224: Download complete 2024-12-18T00:22:20.5347961Z 47d9fb6dae26: Verifying Checksum 2024-12-18T00:22:20.5348479Z 47d9fb6dae26: Download complete 2024-12-18T00:22:20.6038867Z acce7b690dcb: Download complete 2024-12-18T00:22:20.7590523Z 58dc9f64def7: Verifying Checksum 2024-12-18T00:22:20.7590976Z 58dc9f64def7: Download complete 2024-12-18T00:22:20.8537716Z 94dfe5ef1b7e: Verifying Checksum 2024-12-18T00:22:20.8538149Z 94dfe5ef1b7e: Download complete 2024-12-18T00:22:21.4440684Z 40a2c9135b06: Verifying Checksum 2024-12-18T00:22:21.4441108Z 40a2c9135b06: Download complete 2024-12-18T00:22:21.5231657Z 8839d6b0c821: Verifying Checksum 2024-12-18T00:22:21.5232078Z 8839d6b0c821: Download complete 2024-12-18T00:22:58.2521980Z 1c588f0a9e97: Verifying Checksum 2024-12-18T00:22:58.2522338Z 1c588f0a9e97: Download complete 2024-12-18T00:22:58.3382391Z 6727e7e27936: Download complete 2024-12-18T00:22:58.4393892Z 3a55100d1957: Verifying Checksum 2024-12-18T00:22:58.4394222Z 3a55100d1957: Download complete 2024-12-18T00:23:07.1722112Z 3d6f160d3351: Verifying Checksum 2024-12-18T00:23:07.1722601Z 3d6f160d3351: Download complete 2024-12-18T00:23:07.2668446Z b2a8bb947a87: Download complete 2024-12-18T00:23:07.4051708Z 69a74b2360f4: Verifying Checksum 2024-12-18T00:23:07.4052046Z 69a74b2360f4: Download complete 2024-12-18T00:23:08.9804748Z f9e12a253d89: Verifying Checksum 2024-12-18T00:23:08.9805133Z f9e12a253d89: Download complete 2024-12-18T00:23:09.0573508Z 8fdb06d51b96: Download complete 2024-12-18T00:23:35.8308774Z ab6ef6c2ac4d: Verifying Checksum 2024-12-18T00:23:35.8309150Z ab6ef6c2ac4d: Download complete 2024-12-18T00:23:40.5001034Z 389dc939e2a4: Verifying Checksum 2024-12-18T00:23:40.5001405Z 389dc939e2a4: Download complete 2024-12-18T00:23:51.9501400Z 7c373e2d9b7e: Pull complete 2024-12-18T00:23:52.1853125Z 622381141745: Pull complete 2024-12-18T00:23:52.4169510Z f003d065e31e: Pull complete 2024-12-18T00:24:00.1100343Z cd6ce7116815: Pull complete 2024-12-18T00:24:00.3411353Z e843066a8f68: Pull complete 2024-12-18T00:24:00.5604019Z bac50ca2930b: Pull complete 2024-12-18T00:24:00.5976416Z 57e52c6751f4: Pull complete 2024-12-18T00:24:02.8636765Z a9c5c14d77c0: Pull complete 2024-12-18T00:24:03.0566642Z b86099b8619a: Pull complete 2024-12-18T00:24:03.2784155Z 3c8ea8a600ff: Pull complete 2024-12-18T00:24:03.4330459Z bebd539a1d59: Pull complete 2024-12-18T00:24:03.6234063Z 292b18ec2dab: Pull complete 2024-12-18T00:24:56.4860891Z 76e3817b4b4c: Pull complete 2024-12-18T00:24:56.7150191Z 50b1f245e9f5: Pull complete 2024-12-18T00:24:56.9664798Z ad93cf5205f2: Pull complete 2024-12-18T00:24:57.2055805Z 8d17c59eaf7d: Pull complete 2024-12-18T00:24:57.4406189Z b36a8ccf5c8c: Pull complete 2024-12-18T00:24:57.6678774Z aa0a3b87e9e3: Pull complete 2024-12-18T00:25:00.1467537Z 72a632052c5b: Pull complete 2024-12-18T00:25:00.2664868Z ceed6316bc24: Pull complete 2024-12-18T00:25:00.4417831Z d0ca0f491f80: Pull complete 2024-12-18T00:25:00.6734361Z cf02c706cd56: Pull complete 2024-12-18T00:25:00.8063236Z 16073d3f107d: Pull complete 2024-12-18T00:25:01.0259633Z d234faf2f1bf: Pull complete 2024-12-18T00:25:09.5486198Z 3f0ade86af1a: Pull complete 2024-12-18T00:25:09.7648262Z 034b6a03f5f7: Pull complete 2024-12-18T00:25:09.9881095Z 21f8f066852d: Pull complete 2024-12-18T00:25:10.7700403Z 9dc1f315a979: Pull complete 2024-12-18T00:25:10.9616399Z cd6a6063178b: Pull complete 2024-12-18T00:25:11.1392473Z 2116738f4ea9: Pull complete 2024-12-18T00:25:11.5123156Z d7fa134316d9: Pull complete 2024-12-18T00:25:11.6513756Z af03dc3fd15e: Pull complete 2024-12-18T00:25:11.9872583Z fd9de3dc4258: Pull complete 2024-12-18T00:25:12.1802604Z b85093661f77: Pull complete 2024-12-18T00:25:43.7312579Z ab6ef6c2ac4d: Pull complete 2024-12-18T00:25:43.8281180Z 56c896077120: Pull complete 2024-12-18T00:25:43.9663859Z d88e9c5e7c70: Pull complete 2024-12-18T00:25:46.5263754Z 53617ee83302: Pull complete 2024-12-18T00:25:46.7510074Z 3edbde10d404: Pull complete 2024-12-18T00:25:46.9810493Z f4038f7524c3: Pull complete 2024-12-18T00:25:47.4008030Z 4a51173f7757: Pull complete 2024-12-18T00:25:47.6352094Z ed6a71d41274: Pull complete 2024-12-18T00:27:17.0026535Z 389dc939e2a4: Pull complete 2024-12-18T00:27:17.2358510Z 0341eaeed33d: Pull complete 2024-12-18T00:27:17.4731585Z 537cdb2d3168: Pull complete 2024-12-18T00:27:17.9193004Z 76d4a0702497: Pull complete 2024-12-18T00:27:18.1484335Z 4f4fb700ef54: Pull complete 2024-12-18T00:27:18.3777723Z 3287a539d61e: Pull complete 2024-12-18T00:27:18.6190411Z 9b763e24a7d4: Pull complete 2024-12-18T00:27:25.4083019Z f6ef3d4c814e: Pull complete 2024-12-18T00:27:25.6455632Z 06914b00eb49: Pull complete 2024-12-18T00:27:25.8744884Z 724553524224: Pull complete 2024-12-18T00:27:26.1060281Z 47d9fb6dae26: Pull complete 2024-12-18T00:27:26.3330448Z acce7b690dcb: Pull complete 2024-12-18T00:27:27.7172969Z 58dc9f64def7: Pull complete 2024-12-18T00:27:27.9404175Z 94dfe5ef1b7e: Pull complete 2024-12-18T00:27:29.6432559Z 40a2c9135b06: Pull complete 2024-12-18T00:27:29.8766081Z 8839d6b0c821: Pull complete 2024-12-18T00:27:42.2084334Z 1c588f0a9e97: Pull complete 2024-12-18T00:27:42.2750572Z 6727e7e27936: Pull complete 2024-12-18T00:27:42.3613357Z 3a55100d1957: Pull complete 2024-12-18T00:27:45.1837940Z 3d6f160d3351: Pull complete 2024-12-18T00:27:45.2782198Z b2a8bb947a87: Pull complete 2024-12-18T00:27:45.4651754Z 69a74b2360f4: Pull complete 2024-12-18T00:27:46.1052458Z f9e12a253d89: Pull complete 2024-12-18T00:27:46.3012173Z 8fdb06d51b96: Pull complete 2024-12-18T00:27:47.2118857Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb 2024-12-18T00:27:47.2599933Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:27:47.2845239Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:27:47.2907617Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:27:47.2908685Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:27:47.2920672Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:47.2921050Z env: 2024-12-18T00:27:47.2921273Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:47.2921544Z ##[endgroup] 2024-12-18T00:27:47.3158042Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6 2024-12-18T00:27:47.3158497Z with: 2024-12-18T00:27:47.3158725Z driver-version: 550.54.15 2024-12-18T00:27:47.3158991Z env: 2024-12-18T00:27:47.3159210Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:47.3159470Z ##[endgroup] 2024-12-18T00:27:47.3283341Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-12-18T00:27:47.3283748Z with: 2024-12-18T00:27:47.3283970Z timeout_minutes: 10 2024-12-18T00:27:47.3284219Z max_attempts: 3 2024-12-18T00:27:47.3308839Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-12-18T00:27:47.3333268Z retry_wait_seconds: 10 2024-12-18T00:27:47.3333553Z polling_interval_seconds: 1 2024-12-18T00:27:47.3333839Z warning_on_retry: true 2024-12-18T00:27:47.3334109Z continue_on_error: false 2024-12-18T00:27:47.3334369Z env: 2024-12-18T00:27:47.3334589Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:47.3334866Z DRIVER_VERSION: 550.54.15 2024-12-18T00:27:47.3335131Z ##[endgroup] 2024-12-18T00:27:47.4228603Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-12-18T00:27:47.4229415Z + pre_install_nvidia_driver_amzn2 2024-12-18T00:27:47.4231666Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-12-18T00:27:47.8870944Z No match for argument: nvidia-driver-latest-dkms 2024-12-18T00:27:47.8871706Z No packages marked for removal. 2024-12-18T00:27:47.8934126Z Dependencies resolved. 2024-12-18T00:27:47.8944150Z Nothing to do. 2024-12-18T00:27:47.8944745Z Complete! 2024-12-18T00:27:47.9727612Z + install_nvidia_driver_common 2024-12-18T00:27:47.9728227Z + echo 'Before installing NVIDIA driver' 2024-12-18T00:27:47.9730368Z Before installing NVIDIA driver 2024-12-18T00:27:47.9732129Z + lspci 2024-12-18T00:27:48.0782622Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:27:48.0783480Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:27:48.0784099Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:27:48.0784653Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:27:48.0785175Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:27:48.0785757Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:27:48.0786284Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:27:48.0786811Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:27:48.0787578Z + lsmod 2024-12-18T00:27:48.0825883Z Module Size Used by 2024-12-18T00:27:48.0826459Z xt_conntrack 16384 1 2024-12-18T00:27:48.0827012Z nft_chain_nat 16384 3 2024-12-18T00:27:48.0827537Z xt_MASQUERADE 20480 1 2024-12-18T00:27:48.0828138Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:27:48.0828922Z nf_conntrack_netlink 57344 0 2024-12-18T00:27:48.0829695Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:27:48.0830555Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:27:48.0831176Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:27:48.0831750Z xfrm_user 57344 1 2024-12-18T00:27:48.0832272Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:27:48.0832826Z xt_addrtype 16384 2 2024-12-18T00:27:48.0833338Z nft_compat 20480 4 2024-12-18T00:27:48.0833768Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:27:48.0834219Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:27:48.0834811Z br_netfilter 36864 0 2024-12-18T00:27:48.0835108Z bridge 323584 1 br_netfilter 2024-12-18T00:27:48.0835426Z stp 16384 1 bridge 2024-12-18T00:27:48.0835742Z llc 16384 2 bridge,stp 2024-12-18T00:27:48.0836052Z overlay 167936 0 2024-12-18T00:27:48.0836328Z tls 114688 0 2024-12-18T00:27:48.0836590Z nls_ascii 16384 1 2024-12-18T00:27:48.0836868Z sunrpc 692224 1 2024-12-18T00:27:48.0837148Z nls_cp437 20480 1 2024-12-18T00:27:48.0837421Z vfat 24576 1 2024-12-18T00:27:48.0837694Z fat 86016 1 vfat 2024-12-18T00:27:48.0837975Z ena 167936 0 2024-12-18T00:27:48.0838242Z i8042 45056 0 2024-12-18T00:27:48.0838521Z serio 28672 3 i8042 2024-12-18T00:27:48.0838823Z button 24576 0 2024-12-18T00:27:48.0839113Z ghash_clmulni_intel 16384 0 2024-12-18T00:27:48.0839395Z sch_fq_codel 20480 17 2024-12-18T00:27:48.0839674Z dm_mod 188416 0 2024-12-18T00:27:48.0839943Z fuse 163840 1 2024-12-18T00:27:48.0840217Z dax 45056 1 dm_mod 2024-12-18T00:27:48.0840512Z configfs 57344 1 2024-12-18T00:27:48.0840780Z loop 36864 0 2024-12-18T00:27:48.0841050Z dmi_sysfs 20480 0 2024-12-18T00:27:48.0841327Z crc32_pclmul 16384 0 2024-12-18T00:27:48.0841611Z crc32c_intel 24576 0 2024-12-18T00:27:48.0841894Z efivarfs 24576 1 2024-12-18T00:27:48.0842160Z + modinfo nvidia 2024-12-18T00:27:48.0846330Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:27:48.0847258Z alias: char-major-195-* 2024-12-18T00:27:48.0847740Z version: 550.54.15 2024-12-18T00:27:48.0848040Z supported: external 2024-12-18T00:27:48.0848335Z license: NVIDIA 2024-12-18T00:27:48.0848725Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:27:48.0849119Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:27:48.0849599Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:27:48.0850067Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:27:48.0850559Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:27:48.0850932Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:27:48.0851269Z depends: i2c-core,drm 2024-12-18T00:27:48.0851589Z retpoline: Y 2024-12-18T00:27:48.0851908Z name: nvidia 2024-12-18T00:27:48.0852303Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:27:48.0852802Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:27:48.0853282Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:27:48.0853935Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:27:48.0854354Z parm: NVreg_RmLogonRC:int 2024-12-18T00:27:48.0854780Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:27:48.0855117Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:27:48.0855446Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:27:48.0855784Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:27:48.0856169Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:27:48.0856576Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:27:48.0856930Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:27:48.0857248Z parm: NVreg_EnableMSI:int 2024-12-18T00:27:48.0857653Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:27:48.0857999Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:27:48.0858382Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:27:48.0858805Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:27:48.0859219Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:27:48.0859773Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:27:48.0860208Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:27:48.0860660Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:27:48.0861098Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:27:48.0861461Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:27:48.0861858Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:27:48.0862266Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:27:48.0862634Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:27:48.0862974Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:27:48.0863328Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:27:48.0863675Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:27:48.0864006Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:27:48.0864384Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:27:48.0864774Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:27:48.0865124Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:27:48.0865482Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:27:48.0865857Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:27:48.0866217Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:27:48.0866579Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:27:48.0866929Z parm: NVreg_RmMsg:charp 2024-12-18T00:27:48.0867238Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:27:48.0867587Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:27:48.0867937Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:27:48.0868270Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:27:48.0868743Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:27:48.0869098Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:27:48.0869436Z parm: rm_firmware_active:charp 2024-12-18T00:27:48.0869767Z + HAS_NVIDIA_DRIVER=0 2024-12-18T00:27:48.0870020Z ++ command -v nvidia-smi 2024-12-18T00:27:48.0870297Z + '[' -x /usr/bin/nvidia-smi ']' 2024-12-18T00:27:48.0870575Z + set +e 2024-12-18T00:27:48.0870905Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-12-18T00:27:50.3257817Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-12-18T00:27:50.3258179Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:27:50.3258612Z + '[' 0 -ne 0 ']' 2024-12-18T00:27:50.3258889Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-12-18T00:27:50.3259272Z + HAS_NVIDIA_DRIVER=1 2024-12-18T00:27:50.3259889Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-12-18T00:27:50.3260566Z + set -e 2024-12-18T00:27:50.3260856Z + '[' 1 -eq 0 ']' 2024-12-18T00:27:50.3261411Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-12-18T00:27:50.3262456Z + post_install_nvidia_driver_common 2024-12-18T00:27:50.3265258Z + sudo modprobe nvidia 2024-12-18T00:27:50.4397583Z + echo 'After installing NVIDIA driver' 2024-12-18T00:27:50.4397939Z + lspci 2024-12-18T00:27:50.4398178Z After installing NVIDIA driver 2024-12-18T00:27:50.4514780Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:27:50.4515318Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:27:50.4515908Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:27:50.4516471Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:27:50.4516986Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:27:50.4517556Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:27:50.4518080Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:27:50.4518620Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:27:50.4519303Z + lsmod 2024-12-18T00:27:50.4549141Z Module Size Used by 2024-12-18T00:27:50.4549479Z nvidia_uvm 4706304 0 2024-12-18T00:27:50.4549864Z nvidia 54071296 1 nvidia_uvm 2024-12-18T00:27:50.4550174Z drm 602112 1 nvidia 2024-12-18T00:27:50.4550497Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:27:50.4550828Z backlight 24576 1 drm 2024-12-18T00:27:50.4551136Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:27:50.4551449Z xt_conntrack 16384 1 2024-12-18T00:27:50.4551729Z nft_chain_nat 16384 3 2024-12-18T00:27:50.4552002Z xt_MASQUERADE 20480 1 2024-12-18T00:27:50.4552329Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:27:50.4552688Z nf_conntrack_netlink 57344 0 2024-12-18T00:27:50.4553121Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:27:50.4553594Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:27:50.4553932Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:27:50.4554247Z xfrm_user 57344 1 2024-12-18T00:27:50.4554536Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:27:50.4554845Z xt_addrtype 16384 2 2024-12-18T00:27:50.4555124Z nft_compat 20480 4 2024-12-18T00:27:50.4555451Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:27:50.4555894Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:27:50.4556302Z br_netfilter 36864 0 2024-12-18T00:27:50.4556603Z bridge 323584 1 br_netfilter 2024-12-18T00:27:50.4556922Z stp 16384 1 bridge 2024-12-18T00:27:50.4557222Z llc 16384 2 bridge,stp 2024-12-18T00:27:50.4557525Z overlay 167936 0 2024-12-18T00:27:50.4557796Z tls 114688 0 2024-12-18T00:27:50.4558071Z nls_ascii 16384 1 2024-12-18T00:27:50.4558345Z sunrpc 692224 1 2024-12-18T00:27:50.4558609Z nls_cp437 20480 1 2024-12-18T00:27:50.4558874Z vfat 24576 1 2024-12-18T00:27:50.4559142Z fat 86016 1 vfat 2024-12-18T00:27:50.4559431Z ena 167936 0 2024-12-18T00:27:50.4559695Z i8042 45056 0 2024-12-18T00:27:50.4559959Z serio 28672 3 i8042 2024-12-18T00:27:50.4560249Z button 24576 0 2024-12-18T00:27:50.4560528Z ghash_clmulni_intel 16384 0 2024-12-18T00:27:50.4560818Z sch_fq_codel 20480 17 2024-12-18T00:27:50.4561093Z dm_mod 188416 0 2024-12-18T00:27:50.4561352Z fuse 163840 1 2024-12-18T00:27:50.4561626Z dax 45056 1 dm_mod 2024-12-18T00:27:50.4561921Z configfs 57344 1 2024-12-18T00:27:50.4562195Z loop 36864 0 2024-12-18T00:27:50.4562465Z dmi_sysfs 20480 0 2024-12-18T00:27:50.4562908Z crc32_pclmul 16384 0 2024-12-18T00:27:50.4563188Z crc32c_intel 24576 0 2024-12-18T00:27:50.4563469Z efivarfs 24576 1 2024-12-18T00:27:50.4563737Z + modinfo nvidia 2024-12-18T00:27:50.4568990Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:27:50.4569486Z alias: char-major-195-* 2024-12-18T00:27:50.4569772Z version: 550.54.15 2024-12-18T00:27:50.4570038Z supported: external 2024-12-18T00:27:50.4570299Z license: NVIDIA 2024-12-18T00:27:50.4570593Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:27:50.4570957Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:27:50.4571292Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:27:50.4571634Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:27:50.4572002Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:27:50.4572362Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:27:50.4572702Z depends: i2c-core,drm 2024-12-18T00:27:50.4573126Z retpoline: Y 2024-12-18T00:27:50.4573364Z name: nvidia 2024-12-18T00:27:50.4573750Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:27:50.4574257Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:27:50.4574737Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:27:50.4575180Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:27:50.4575521Z parm: NVreg_RmLogonRC:int 2024-12-18T00:27:50.4575849Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:27:50.4576191Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:27:50.4576519Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:27:50.4576843Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:27:50.4577241Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:27:50.4577663Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:27:50.4578026Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:27:50.4578348Z parm: NVreg_EnableMSI:int 2024-12-18T00:27:50.4578659Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:27:50.4579005Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:27:50.4579396Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:27:50.4579823Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:27:50.4580236Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:27:50.4580683Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:27:50.4581122Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:27:50.4581567Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:27:50.4582013Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:27:50.4582374Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:27:50.4582784Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:27:50.4583191Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:27:50.4583556Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:27:50.4583904Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:27:50.4584265Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:27:50.4584618Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:27:50.4584957Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:27:50.4585328Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:27:50.4585711Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:27:50.4586063Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:27:50.4586426Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:27:50.4586796Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:27:50.4587160Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:27:50.4587520Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:27:50.4589270Z parm: NVreg_RmMsg:charp 2024-12-18T00:27:50.4589619Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:27:50.4589975Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:27:50.4590328Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:27:50.4590665Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:27:50.4591020Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:27:50.4591375Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:27:50.4591715Z parm: rm_firmware_active:charp 2024-12-18T00:27:50.4592021Z + set +e 2024-12-18T00:27:50.4592225Z + nvidia-smi 2024-12-18T00:27:52.0315447Z Wed Dec 18 00:27:52 2024 2024-12-18T00:27:52.0315876Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:27:52.0316414Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:27:52.0316957Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:27:52.0317769Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:27:52.0318330Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:27:52.0318791Z | | | MIG M. | 2024-12-18T00:27:52.0319139Z |=========================================+========================+======================| 2024-12-18T00:27:52.0414223Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:27:52.0414720Z | 0% 25C P0 55W / 300W | 0MiB / 23028MiB | 5% Default | 2024-12-18T00:27:52.0415135Z | | | N/A | 2024-12-18T00:27:52.0415568Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:27:52.0416215Z 2024-12-18T00:27:52.0416665Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:27:52.0417125Z | Processes: | 2024-12-18T00:27:52.0417604Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:27:52.0418051Z | ID ID Usage | 2024-12-18T00:27:52.0418406Z |=========================================================================================| 2024-12-18T00:27:52.0419545Z | No running processes found | 2024-12-18T00:27:52.0420044Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:27:52.6607393Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-12-18T00:27:54.2292068Z NVIDIA A10G 2024-12-18T00:27:54.6838466Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:27:54.6838785Z + '[' 0 -eq 0 ']' 2024-12-18T00:27:54.6839052Z + echo 'INFO: Ignoring allowed status 0' 2024-12-18T00:27:54.6839367Z + set -e 2024-12-18T00:27:54.6839590Z INFO: Ignoring allowed status 0 2024-12-18T00:27:54.6845854Z == Installing nvidia container toolkit for amzn2023 == 2024-12-18T00:27:54.6850021Z + sudo yum install -y yum-utils 2024-12-18T00:27:55.0865207Z Last metadata expiration check: 0:31:21 ago on Tue Dec 17 23:56:34 2024. 2024-12-18T00:27:55.1087767Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-12-18T00:27:55.1419004Z Dependencies resolved. 2024-12-18T00:27:55.1556528Z Nothing to do. 2024-12-18T00:27:55.1557437Z Complete! 2024-12-18T00:27:55.2413637Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-12-18T00:27:55.2414758Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:27:55.2415976Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:27:55.5143407Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:27:55.5840411Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2024-12-18T00:27:56.0863729Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-12-18T00:27:56.1103704Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-12-18T00:27:56.1442138Z Dependencies resolved. 2024-12-18T00:27:56.1577739Z ================================================================================ 2024-12-18T00:27:56.1578188Z Package Arch Version Repository Size 2024-12-18T00:27:56.1578617Z ================================================================================ 2024-12-18T00:27:56.1578934Z Downgrading: 2024-12-18T00:27:56.1579354Z nvidia-container-toolkit x86_64 1.16.2-1 nvidia-container-toolkit 1.2 M 2024-12-18T00:27:56.1580243Z nvidia-container-toolkit-base x86_64 1.16.2-1 nvidia-container-toolkit 5.6 M 2024-12-18T00:27:56.1580620Z 2024-12-18T00:27:56.1580728Z Transaction Summary 2024-12-18T00:27:56.1581002Z ================================================================================ 2024-12-18T00:27:56.1581323Z Downgrade 2 Packages 2024-12-18T00:27:56.1581493Z 2024-12-18T00:27:56.1581605Z Total download size: 6.8 M 2024-12-18T00:27:56.1582471Z Downloading Packages: 2024-12-18T00:27:56.3096584Z (1/2): nvidia-container-toolkit-base-1.16.2-1.x 37 MB/s | 5.6 MB 00:00 2024-12-18T00:27:56.3629704Z (2/2): nvidia-container-toolkit-1.16.2-1.x86_64 6.1 MB/s | 1.2 MB 00:00 2024-12-18T00:27:56.3640863Z -------------------------------------------------------------------------------- 2024-12-18T00:27:56.3643814Z Total 33 MB/s | 6.8 MB 00:00 2024-12-18T00:27:56.3646229Z Running transaction check 2024-12-18T00:27:56.3745040Z Transaction check succeeded. 2024-12-18T00:27:56.3745440Z Running transaction test 2024-12-18T00:27:56.4041402Z Transaction test succeeded. 2024-12-18T00:27:56.4044190Z Running transaction 2024-12-18T00:27:57.1039787Z Preparing : 1/1 2024-12-18T00:27:57.2484666Z Downgrading : nvidia-container-toolkit-base-1.16.2-1.x86_64 1/4 2024-12-18T00:27:57.2762056Z Downgrading : nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:27:57.2910427Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 2/4 2024-12-18T00:27:57.2911788Z Cleanup : nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:27:57.3260272Z Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64 3/4 2024-12-18T00:27:57.3530501Z Cleanup : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4 2024-12-18T00:28:49.3351893Z Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64 4/4 2024-12-18T00:28:49.3352737Z Verifying : nvidia-container-toolkit-1.16.2-1.x86_64 1/4 2024-12-18T00:28:49.3353459Z Verifying : nvidia-container-toolkit-1.17.3-1.x86_64 2/4 2024-12-18T00:28:49.3354058Z Verifying : nvidia-container-toolkit-base-1.16.2-1.x86_64 3/4 2024-12-18T00:28:49.4669400Z Verifying : nvidia-container-toolkit-base-1.17.3-1.x86_64 4/4================================================================================ 2024-12-18T00:28:49.4670119Z WARNING: 2024-12-18T00:28:49.4670387Z A newer release of "Amazon Linux" is available. 2024-12-18T00:28:49.4670738Z 2024-12-18T00:28:49.4670900Z Available Versions: 2024-12-18T00:28:49.4671126Z 2024-12-18T00:28:49.4671257Z Version 2023.6.20241010: 2024-12-18T00:28:49.4671719Z Run the following command to upgrade to 2023.6.20241010: 2024-12-18T00:28:49.4672113Z 2024-12-18T00:28:49.4672521Z dnf upgrade --releasever=2023.6.20241010 2024-12-18T00:28:49.4672772Z 2024-12-18T00:28:49.4672866Z Release notes: 2024-12-18T00:28:49.4673311Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html 2024-12-18T00:28:49.4673703Z 2024-12-18T00:28:49.4673807Z Version 2023.6.20241028: 2024-12-18T00:28:49.4674137Z Run the following command to upgrade to 2023.6.20241028: 2024-12-18T00:28:49.4674410Z 2024-12-18T00:28:49.4674549Z dnf upgrade --releasever=2023.6.20241028 2024-12-18T00:28:49.4674780Z 2024-12-18T00:28:49.4674874Z Release notes: 2024-12-18T00:28:49.4675301Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html 2024-12-18T00:28:49.4675691Z 2024-12-18T00:28:49.4675787Z Version 2023.6.20241031: 2024-12-18T00:28:49.4676121Z Run the following command to upgrade to 2023.6.20241031: 2024-12-18T00:28:49.4676386Z 2024-12-18T00:28:49.4676517Z dnf upgrade --releasever=2023.6.20241031 2024-12-18T00:28:49.4676745Z 2024-12-18T00:28:49.4676996Z Release notes: 2024-12-18T00:28:49.4677425Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html 2024-12-18T00:28:49.4677814Z 2024-12-18T00:28:49.4677915Z Version 2023.6.20241111: 2024-12-18T00:28:49.4678255Z Run the following command to upgrade to 2023.6.20241111: 2024-12-18T00:28:49.4678519Z 2024-12-18T00:28:49.4678650Z dnf upgrade --releasever=2023.6.20241111 2024-12-18T00:28:49.4678873Z 2024-12-18T00:28:49.4678973Z Release notes: 2024-12-18T00:28:49.4679391Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html 2024-12-18T00:28:49.4679781Z 2024-12-18T00:28:49.4679878Z Version 2023.6.20241121: 2024-12-18T00:28:49.4680215Z Run the following command to upgrade to 2023.6.20241121: 2024-12-18T00:28:49.4680487Z 2024-12-18T00:28:49.4680611Z dnf upgrade --releasever=2023.6.20241121 2024-12-18T00:28:49.4680834Z 2024-12-18T00:28:49.4680936Z Release notes: 2024-12-18T00:28:49.4681363Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html 2024-12-18T00:28:49.4681754Z 2024-12-18T00:28:49.4681850Z Version 2023.6.20241212: 2024-12-18T00:28:49.4682187Z Run the following command to upgrade to 2023.6.20241212: 2024-12-18T00:28:49.4682461Z 2024-12-18T00:28:49.4682588Z dnf upgrade --releasever=2023.6.20241212 2024-12-18T00:28:49.4682818Z 2024-12-18T00:28:49.4682911Z Release notes: 2024-12-18T00:28:49.4683335Z https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html 2024-12-18T00:28:49.4683718Z 2024-12-18T00:28:49.4683841Z ================================================================================ 2024-12-18T00:28:49.4936785Z 2024-12-18T00:28:49.4937141Z 2024-12-18T00:28:49.4937376Z Downgraded: 2024-12-18T00:28:49.4938093Z nvidia-container-toolkit-1.16.2-1.x86_64 2024-12-18T00:28:49.4939215Z nvidia-container-toolkit-base-1.16.2-1.x86_64 2024-12-18T00:28:49.4939877Z 2024-12-18T00:28:49.4940053Z Complete! 2024-12-18T00:28:49.5369434Z + sudo systemctl restart docker 2024-12-18T00:28:54.8889464Z Wed Dec 18 00:28:54 2024 2024-12-18T00:28:54.8889904Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:28:54.8890442Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:28:54.8891131Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:28:54.8891675Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:28:54.8892247Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:28:54.8892709Z | | | MIG M. | 2024-12-18T00:28:54.8893516Z |=========================================+========================+======================| 2024-12-18T00:28:54.9009100Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:28:54.9009638Z | 0% 25C P0 55W / 300W | 0MiB / 23028MiB | 5% Default | 2024-12-18T00:28:54.9010174Z | | | N/A | 2024-12-18T00:28:54.9010817Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:28:54.9011267Z 2024-12-18T00:28:54.9011690Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:28:54.9012156Z | Processes: | 2024-12-18T00:28:54.9012642Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:28:54.9013114Z | ID ID Usage | 2024-12-18T00:28:54.9013876Z |=========================================================================================| 2024-12-18T00:28:54.9014503Z | No running processes found | 2024-12-18T00:28:54.9015011Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:28:55.4627221Z Command completed after 1 attempt(s). 2024-12-18T00:28:55.4718220Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:28:55.4718799Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:28:55.4719309Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-12-18T00:28:55.4719792Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:28:55.4733595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:28:55.4733975Z env: 2024-12-18T00:28:55.4734214Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:28:55.4734556Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:28:55.4734916Z ##[endgroup] 2024-12-18T00:28:55.7534049Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:28:56.1721089Z Collecting psutil==5.9.1 2024-12-18T00:28:56.2104120Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-12-18T00:28:56.2581501Z Collecting nvidia-ml-py==11.525.84 2024-12-18T00:28:56.2622120Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-12-18T00:28:56.3436597Z Installing collected packages: psutil, nvidia-ml-py 2024-12-18T00:28:56.5033185Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-12-18T00:28:56.7051903Z Prepare all required actions 2024-12-18T00:28:56.7053098Z Getting action download info 2024-12-18T00:28:56.8390491Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T00:28:57.1265975Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T00:28:57.4972181Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T00:28:57.4972552Z with: 2024-12-18T00:28:57.4972814Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:28:57.4973175Z s3-bucket: gha-artifacts 2024-12-18T00:28:57.4973443Z env: 2024-12-18T00:28:57.4973656Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:28:57.4974001Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:28:57.4974362Z ##[endgroup] 2024-12-18T00:28:57.5009422Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:28:57.5009772Z with: 2024-12-18T00:28:57.5010064Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:28:57.5010413Z s3-bucket: gha-artifacts 2024-12-18T00:28:57.5010684Z region: us-east-1 2024-12-18T00:28:57.5010924Z env: 2024-12-18T00:28:57.5011151Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:28:57.5011510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:28:57.5011873Z ##[endgroup] 2024-12-18T00:28:57.9640924Z (node:53459) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:28:57.9641445Z 2024-12-18T00:28:57.9641642Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:28:57.9642182Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:28:57.9642731Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:28:58.1687761Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-12-18T00:28:58.1688548Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:29:07.7751665Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:29:07.7756813Z Artifact download has finished successfully 2024-12-18T00:29:07.8129609Z ##[group]Run unzip -o artifacts.zip 2024-12-18T00:29:07.8129962Z unzip -o artifacts.zip 2024-12-18T00:29:07.8139402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:07.8139779Z env: 2024-12-18T00:29:07.8140007Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:07.8140360Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:07.8140733Z ##[endgroup] 2024-12-18T00:29:07.8208079Z Archive: artifacts.zip 2024-12-18T00:29:07.8209264Z creating: dist/ 2024-12-18T00:29:10.0054447Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:29:10.0055437Z creating: build/custom_test_artifacts/ 2024-12-18T00:29:10.0056239Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T00:29:10.0057115Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T00:29:10.0057739Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:29:10.0063539Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:29:10.0064205Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T00:29:10.0064857Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:29:10.0065552Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:29:10.0066229Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:29:10.0068415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:29:10.0070162Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:29:10.0070884Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:29:10.0071583Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:29:10.0074084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:29:10.0075277Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:29:10.0077552Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:29:10.0078486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:29:10.0079921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:29:10.0081348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:29:10.0082182Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:29:10.0082986Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:29:10.0124621Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:29:10.0165720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:29:10.0166823Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:29:10.0213652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:29:10.0214675Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:29:10.0215705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:29:10.0216941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:29:10.0217958Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:29:10.0218941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:29:10.0219942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:29:10.0220929Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:29:10.0221904Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:29:10.0222823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:29:10.0223747Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:29:10.0224846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:29:10.0225920Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:29:10.0227096Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:29:10.0228737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:29:10.0302936Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:29:10.0376250Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:29:10.0377105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:29:10.0378526Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:29:10.0379744Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:29:10.0381010Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T00:29:10.0382322Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T00:29:10.0383768Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T00:29:10.0385410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T00:29:10.0387002Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T00:29:10.0387786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T00:29:10.0388612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T00:29:10.0389383Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T00:29:10.0390151Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T00:29:10.0390924Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T00:29:10.0391682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T00:29:10.0408227Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T00:29:10.0552428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T00:29:10.0553157Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T00:29:10.0554097Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T00:29:10.0554980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T00:29:10.0555823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T00:29:10.0556609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T00:29:10.0557418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T00:29:10.0558235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T00:29:10.0559045Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T00:29:10.0559867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T00:29:10.0560676Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T00:29:10.0581545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T00:29:10.0665850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T00:29:10.0667247Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:29:10.0668033Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:29:10.0668964Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T00:29:10.0669630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T00:29:10.0670276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T00:29:10.0670932Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-12-18T00:29:10.0673494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T00:29:10.0674121Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T00:29:10.0675107Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T00:29:10.0796130Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T00:29:10.0859549Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T00:29:10.0860077Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T00:29:10.0860568Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T00:29:10.0861148Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:29:10.0868322Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:29:10.0868995Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T00:29:10.0869658Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:29:10.0870357Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:29:10.0871043Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:29:10.0873132Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:29:10.0874450Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:29:10.0875164Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:29:10.0875862Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:29:10.0878401Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:29:10.0879728Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:29:10.0881892Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:29:10.0882710Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:29:10.0884530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:29:10.0885724Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:29:10.0886471Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:29:10.0887184Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:29:10.0928826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:29:10.0970056Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:29:10.0971084Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:29:10.1018148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:29:10.1019158Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:29:10.1020365Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:29:10.1021417Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:29:10.1022434Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:29:10.1023425Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:29:10.1024424Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:29:10.1025408Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:29:10.1026380Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:29:10.1027312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:29:10.1028273Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:29:10.1029156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:29:10.1030035Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:29:10.1031573Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:29:10.1032856Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:29:10.1106591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:29:10.1180299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:29:10.1181254Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:29:10.1182137Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:29:10.1182756Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:29:10.1183389Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T00:29:10.1184065Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T00:29:10.1184825Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T00:29:10.1185685Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T00:29:10.1186514Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T00:29:10.1187277Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T00:29:10.1188082Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T00:29:10.1188965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T00:29:10.1189762Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T00:29:10.1190569Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T00:29:10.1191364Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T00:29:10.1212618Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T00:29:10.1276697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T00:29:10.1277552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:29:10.1278347Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:29:10.1279056Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T00:29:10.1279726Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T00:29:10.1281088Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T00:29:10.1282389Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-12-18T00:29:10.1285600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T00:29:10.1286486Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T00:29:10.1287288Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T00:29:10.1338137Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T00:29:10.1339340Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T00:29:10.1340392Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T00:29:10.1341637Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:29:10.1346854Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:29:10.1347668Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T00:29:10.1348484Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:29:10.1349234Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:29:10.1349985Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:29:10.1351010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:29:10.1352517Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:29:10.1353356Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:29:10.1354163Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:29:10.1356402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:29:10.1357722Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:29:10.1360036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:29:10.1360987Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:29:10.1362292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:29:10.1363724Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:29:10.1364564Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:29:10.1365515Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:29:10.1407660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:29:10.1449014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:29:10.1450394Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:29:10.1496932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:29:10.1498194Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:29:10.1499793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:29:10.1501117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:29:10.1502404Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:29:10.1503602Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:29:10.1504797Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:29:10.1506037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:29:10.1507295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:29:10.1508579Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:29:10.1509657Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:29:10.1510657Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:29:10.1511614Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:29:10.1512783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:29:10.1513882Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:29:10.1586291Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:29:10.1660313Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:29:10.1661445Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:29:10.1662434Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:29:10.1663213Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:29:10.1664086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T00:29:10.1664953Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T00:29:10.1665938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T00:29:10.1667029Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T00:29:10.1668038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T00:29:10.1669092Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T00:29:10.1670174Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T00:29:10.1671344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T00:29:10.1672223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T00:29:10.1673096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T00:29:10.1673936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T00:29:10.1675426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T00:29:10.1797558Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T00:29:10.1798586Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T00:29:10.1799812Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T00:29:10.1800980Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T00:29:10.1802024Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T00:29:10.1803097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T00:29:10.1804144Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T00:29:10.1805221Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T00:29:10.1806142Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T00:29:10.1807045Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T00:29:10.1807943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T00:29:10.1827530Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T00:29:10.1882339Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T00:29:10.1883521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:29:10.1884499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:29:10.1885436Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T00:29:10.1886195Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T00:29:10.1887130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T00:29:10.1888266Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-12-18T00:29:10.1891363Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T00:29:10.1892294Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T00:29:10.1893128Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T00:29:10.1994537Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T00:29:10.2037732Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T00:29:10.2038317Z creating: build/lib/ 2024-12-18T00:29:10.2127343Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T00:29:10.2583114Z inflating: build/lib/libprotobuf.a 2024-12-18T00:29:10.2592110Z inflating: build/lib/libpthreadpool.a 2024-12-18T00:29:10.2601038Z inflating: build/lib/libcpuinfo.a 2024-12-18T00:29:10.2609016Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T00:29:10.2609706Z inflating: build/lib/libclog.a 2024-12-18T00:29:10.2628124Z inflating: build/lib/libnnpack.a 2024-12-18T00:29:10.2631339Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T00:29:10.2817847Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T00:29:10.2886083Z inflating: build/lib/libgtest.a 2024-12-18T00:29:10.2960784Z inflating: build/lib/libbenchmark.a 2024-12-18T00:29:10.3024480Z inflating: build/lib/libasmjit.a 2024-12-18T00:29:10.3032287Z inflating: build/lib/libittnotify.a 2024-12-18T00:29:10.3059919Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T00:29:10.3187075Z inflating: build/lib/libgloo.a 2024-12-18T00:29:10.3208115Z inflating: build/lib/libfmt.a 2024-12-18T00:29:10.3307444Z inflating: build/lib/libc10.so 2024-12-18T00:29:10.3309350Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T00:29:10.3310516Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T00:29:10.3813975Z inflating: build/lib/libprotoc.a 2024-12-18T00:29:10.3833349Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T00:29:10.3850747Z inflating: build/lib/libgmock.a 2024-12-18T00:29:10.3851531Z inflating: build/lib/libgtest_main.a 2024-12-18T00:29:10.3852207Z inflating: build/lib/libbenchmark_main.a 2024-12-18T00:29:10.4425789Z inflating: build/lib/libtensorpipe.a 2024-12-18T00:29:10.4814392Z inflating: build/lib/libgloo_cuda.a 2024-12-18T00:29:11.5100876Z inflating: build/lib/libdnnl.a 2024-12-18T00:29:11.5161057Z inflating: build/lib/libc10_cuda.so 2024-12-18T00:29:11.5161623Z inflating: build/lib/libgmock_main.a 2024-12-18T00:29:11.6418911Z inflating: build/lib/libfbgemm.a 2024-12-18T00:29:11.6674551Z inflating: build/lib/libtensorpipe_cuda.a 2024-12-18T00:29:11.7181752Z inflating: build/lib/libkineto.a 2024-12-18T00:29:11.7270784Z inflating: build/lib/libXNNPACK.a 2024-12-18T00:29:11.7313484Z inflating: build/lib/libonnx_proto.a 2024-12-18T00:29:11.8043962Z inflating: build/lib/libonnx.a 2024-12-18T00:29:11.8911218Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T00:29:14.4133055Z inflating: build/lib/libtorch_cpu.so 2024-12-18T00:29:14.4137951Z inflating: build/lib/libunbox_lib.a 2024-12-18T00:29:14.4141542Z inflating: build/lib/libshm.so 2024-12-18T00:29:16.5022599Z inflating: build/lib/libtorch_cuda.so 2024-12-18T00:29:16.5024210Z inflating: build/lib/libtorch.so 2024-12-18T00:29:16.5026368Z inflating: build/lib/libc10d_cuda_test.so 2024-12-18T00:29:17.3791452Z inflating: build/lib/libtorch_cuda_linalg.so 2024-12-18T00:29:17.5871751Z inflating: build/lib/libtorch_python.so 2024-12-18T00:29:17.5941708Z inflating: build/lib/libtorchbind_test.so 2024-12-18T00:29:17.5962367Z inflating: build/lib/libjitbackend_test.so 2024-12-18T00:29:17.5986832Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T00:29:17.6012109Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T00:29:17.6046642Z inflating: build/lib/libnnapi_backend.so 2024-12-18T00:29:17.6047272Z creating: build/bin/ 2024-12-18T00:29:17.6101128Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T00:29:17.6156024Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T00:29:17.6211295Z inflating: build/bin/c10_Device_test 2024-12-18T00:29:17.6273697Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T00:29:17.6331145Z inflating: build/bin/c10_Scalar_test 2024-12-18T00:29:17.6383254Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T00:29:17.6438196Z inflating: build/bin/c10_SymInt_test 2024-12-18T00:29:17.6495588Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T00:29:17.6554751Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T00:29:17.6614512Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T00:29:17.6688903Z inflating: build/bin/c10_cow_test 2024-12-18T00:29:17.6742291Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T00:29:17.6799139Z inflating: build/bin/c10_Bitset_test 2024-12-18T00:29:17.6850576Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T00:29:17.6904338Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T00:29:17.6958241Z inflating: build/bin/c10_Half_test 2024-12-18T00:29:17.7017956Z inflating: build/bin/c10_LeftRight_test 2024-12-18T00:29:17.7076235Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T00:29:17.7133715Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T00:29:17.7186280Z inflating: build/bin/c10_Synchronized_test 2024-12-18T00:29:17.7245270Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T00:29:17.7301169Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T00:29:17.7355709Z inflating: build/bin/c10_TypeList_test 2024-12-18T00:29:17.7408053Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T00:29:17.7463769Z inflating: build/bin/c10_accumulate_test 2024-12-18T00:29:17.7522618Z inflating: build/bin/c10_bfloat16_test 2024-12-18T00:29:17.7576996Z inflating: build/bin/c10_bit_cast_test 2024-12-18T00:29:17.7636712Z inflating: build/bin/c10_complex_math_test 2024-12-18T00:29:17.7695657Z inflating: build/bin/c10_complex_test 2024-12-18T00:29:17.7748698Z inflating: build/bin/c10_error_test 2024-12-18T00:29:17.7804511Z inflating: build/bin/c10_exception_test 2024-12-18T00:29:17.7859094Z inflating: build/bin/c10_flags_test 2024-12-18T00:29:17.7912431Z inflating: build/bin/c10_generic_math_test 2024-12-18T00:29:17.8083619Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T00:29:17.8138772Z inflating: build/bin/c10_irange_test 2024-12-18T00:29:17.8194889Z inflating: build/bin/c10_lazy_test 2024-12-18T00:29:17.8255708Z inflating: build/bin/c10_logging_test 2024-12-18T00:29:17.8335656Z inflating: build/bin/c10_optional_test 2024-12-18T00:29:17.8401468Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T00:29:17.8458482Z inflating: build/bin/c10_registry_test 2024-12-18T00:29:17.8618597Z inflating: build/bin/c10_small_vector_test 2024-12-18T00:29:17.8674083Z inflating: build/bin/c10_ssize_test 2024-12-18T00:29:17.8729141Z inflating: build/bin/c10_string_util_test 2024-12-18T00:29:17.8792513Z inflating: build/bin/c10_string_view_test 2024-12-18T00:29:17.8846253Z inflating: build/bin/c10_tempfile_test 2024-12-18T00:29:17.8906235Z inflating: build/bin/c10_typeid_test 2024-12-18T00:29:17.8954421Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T00:29:17.9402633Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T00:29:17.9849905Z inflating: build/bin/protoc 2024-12-18T00:29:17.9906372Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-12-18T00:29:17.9962603Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-12-18T00:29:18.0018750Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-12-18T00:29:18.0074296Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-12-18T00:29:18.0131352Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T00:29:18.0187760Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T00:29:18.0243116Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-12-18T00:29:18.0295968Z inflating: build/bin/c10_cuda_CUDATest 2024-12-18T00:29:18.0697478Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T00:29:18.1108402Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T00:29:18.1535270Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T00:29:18.1590449Z inflating: build/bin/BackoffTest 2024-12-18T00:29:18.1647037Z inflating: build/bin/FileStoreTest 2024-12-18T00:29:18.1705922Z inflating: build/bin/TCPStoreTest 2024-12-18T00:29:18.1762763Z inflating: build/bin/HashStoreTest 2024-12-18T00:29:18.1776160Z inflating: build/bin/ProcessGroupMPITest 2024-12-18T00:29:18.1833958Z inflating: build/bin/test_edge_op_registration 2024-12-18T00:29:18.1838388Z inflating: build/bin/torch_shm_manager 2024-12-18T00:29:18.1840844Z inflating: build/bin/example_allreduce 2024-12-18T00:29:18.1899700Z inflating: build/bin/test_dist_autograd 2024-12-18T00:29:18.1972837Z inflating: build/bin/test_cpp_rpc 2024-12-18T00:29:18.1976081Z inflating: build/bin/parallel_benchmark 2024-12-18T00:29:18.2046119Z inflating: build/bin/test_mobile_nnc 2024-12-18T00:29:18.2055311Z inflating: build/bin/aot_model_compiler_test 2024-12-18T00:29:18.2408526Z inflating: build/bin/test_lazy 2024-12-18T00:29:18.3596977Z inflating: build/bin/test_api 2024-12-18T00:29:18.3675168Z inflating: build/bin/Dict_test 2024-12-18T00:29:18.3731277Z inflating: build/bin/Dimname_test 2024-12-18T00:29:18.3800352Z inflating: build/bin/MaybeOwned_test 2024-12-18T00:29:18.3860996Z inflating: build/bin/NamedTensor_test 2024-12-18T00:29:18.3923634Z inflating: build/bin/apply_utils_test 2024-12-18T00:29:18.3986590Z inflating: build/bin/atest 2024-12-18T00:29:18.4054189Z inflating: build/bin/basic 2024-12-18T00:29:18.4112287Z inflating: build/bin/broadcast_test 2024-12-18T00:29:18.4166873Z inflating: build/bin/cpu_allocator_test 2024-12-18T00:29:18.4229150Z inflating: build/bin/cpu_generator_test 2024-12-18T00:29:18.4286486Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T00:29:18.4383125Z inflating: build/bin/cpu_rng_test 2024-12-18T00:29:18.4437543Z inflating: build/bin/dispatch_key_set_test 2024-12-18T00:29:18.4490873Z inflating: build/bin/dlconvertor_test 2024-12-18T00:29:18.4552722Z inflating: build/bin/extension_backend_test 2024-12-18T00:29:18.4611661Z inflating: build/bin/half_test 2024-12-18T00:29:18.4711727Z inflating: build/bin/ivalue_test 2024-12-18T00:29:18.4765062Z inflating: build/bin/lazy_tensor_test 2024-12-18T00:29:18.4822776Z inflating: build/bin/math_kernel_test 2024-12-18T00:29:18.4880110Z inflating: build/bin/memory_format_test 2024-12-18T00:29:18.4937481Z inflating: build/bin/memory_overlapping_test 2024-12-18T00:29:18.4993241Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T00:29:18.5053384Z inflating: build/bin/native_test 2024-12-18T00:29:18.5108216Z inflating: build/bin/operator_name_test 2024-12-18T00:29:18.5162505Z inflating: build/bin/operators_test 2024-12-18T00:29:18.5218080Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T00:29:18.5288708Z inflating: build/bin/pow_test 2024-12-18T00:29:18.5349933Z inflating: build/bin/quantized_test 2024-12-18T00:29:18.5403297Z inflating: build/bin/reduce_ops_test 2024-12-18T00:29:18.5457905Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T00:29:18.5518944Z inflating: build/bin/scalar_tensor_test 2024-12-18T00:29:18.5581958Z inflating: build/bin/scalar_test 2024-12-18T00:29:18.5636618Z inflating: build/bin/StorageUtils_test 2024-12-18T00:29:18.5692476Z inflating: build/bin/stride_properties_test 2024-12-18T00:29:18.5776591Z inflating: build/bin/tensor_iterator_test 2024-12-18T00:29:18.5834067Z inflating: build/bin/test_parallel 2024-12-18T00:29:18.5837298Z inflating: build/bin/thread_init_test 2024-12-18T00:29:18.5896596Z inflating: build/bin/type_ptr_test 2024-12-18T00:29:18.5959752Z inflating: build/bin/type_test 2024-12-18T00:29:18.6015434Z inflating: build/bin/undefined_tensor_test 2024-12-18T00:29:18.6016877Z inflating: build/bin/verify_api_visibility 2024-12-18T00:29:18.6091711Z inflating: build/bin/legacy_vmap_test 2024-12-18T00:29:18.6146289Z inflating: build/bin/weakref_test 2024-12-18T00:29:18.6201461Z inflating: build/bin/wrapdim_test 2024-12-18T00:29:18.6256288Z inflating: build/bin/xla_tensor_test 2024-12-18T00:29:18.6320455Z inflating: build/bin/IListRef_test 2024-12-18T00:29:18.6431942Z inflating: build/bin/List_test 2024-12-18T00:29:18.6502513Z inflating: build/bin/KernelFunction_test 2024-12-18T00:29:18.6627497Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T00:29:18.6728217Z inflating: build/bin/kernel_function_test 2024-12-18T00:29:18.6861035Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T00:29:18.6968631Z inflating: build/bin/kernel_lambda_test 2024-12-18T00:29:18.7033722Z inflating: build/bin/kernel_stackbased_test 2024-12-18T00:29:18.7134082Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T00:29:18.7188909Z inflating: build/bin/CppSignature_test 2024-12-18T00:29:18.7248015Z inflating: build/bin/backend_fallback_test 2024-12-18T00:29:18.7300465Z inflating: build/bin/op_allowlist_test 2024-12-18T00:29:18.7607856Z inflating: build/bin/op_registration_test 2024-12-18T00:29:18.7675437Z inflating: build/bin/inline_container_test 2024-12-18T00:29:18.7732034Z inflating: build/bin/cuda_apply_test 2024-12-18T00:29:18.7788172Z inflating: build/bin/cuda_allocator_test 2024-12-18T00:29:18.7845762Z inflating: build/bin/cuda_caching_host_allocator_test 2024-12-18T00:29:18.7909126Z inflating: build/bin/cuda_atomic_ops_test 2024-12-18T00:29:18.7983820Z inflating: build/bin/cuda_complex_math_test 2024-12-18T00:29:18.8045763Z inflating: build/bin/cuda_complex_test 2024-12-18T00:29:18.8099506Z inflating: build/bin/cuda_device_test 2024-12-18T00:29:18.8161579Z inflating: build/bin/cuda_cub_test 2024-12-18T00:29:18.8215952Z inflating: build/bin/cuda_dlconvertor_test 2024-12-18T00:29:18.8284823Z inflating: build/bin/cuda_distributions_test 2024-12-18T00:29:18.8345521Z inflating: build/bin/cuda_generator_test 2024-12-18T00:29:18.8399108Z inflating: build/bin/cuda_half_test 2024-12-18T00:29:18.8453347Z inflating: build/bin/cuda_integer_divider_test 2024-12-18T00:29:18.8506657Z inflating: build/bin/cuda_optional_test 2024-12-18T00:29:18.8561903Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-12-18T00:29:18.8619851Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-12-18T00:29:18.8673140Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-12-18T00:29:18.8738483Z inflating: build/bin/cuda_stream_test 2024-12-18T00:29:18.8791068Z inflating: build/bin/cuda_cudnn_test 2024-12-18T00:29:18.8847057Z inflating: build/bin/cuda_vectorized_test 2024-12-18T00:29:18.8861616Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T00:29:18.8931059Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T00:29:18.8992049Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-12-18T00:29:18.9059846Z inflating: build/bin/ProcessGroupNCCLTest 2024-12-18T00:29:18.9125307Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-12-18T00:29:18.9970991Z inflating: build/bin/test_tensorexpr 2024-12-18T00:29:19.0556661Z inflating: build/bin/test_jit 2024-12-18T00:29:19.0557142Z creating: .additional_ci_files/ 2024-12-18T00:29:19.0647765Z inflating: .additional_ci_files/test-times.json 2024-12-18T00:29:19.1007901Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T00:29:19.1040891Z ##[group]Run rm artifacts.zip 2024-12-18T00:29:19.1041211Z rm artifacts.zip 2024-12-18T00:29:19.1050564Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:19.1050936Z env: 2024-12-18T00:29:19.1051160Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:19.1051512Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:19.1051881Z ##[endgroup] 2024-12-18T00:29:19.3189797Z ##[group]Run df -H 2024-12-18T00:29:19.3190061Z df -H 2024-12-18T00:29:19.3198715Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:19.3199468Z env: 2024-12-18T00:29:19.3199713Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:19.3200066Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:19.3200436Z ##[endgroup] 2024-12-18T00:29:19.3249185Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T00:29:19.3249610Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-12-18T00:29:19.3249946Z tmpfs 34G 0 34G 0% /dev/shm 2024-12-18T00:29:19.3250294Z tmpfs 14G 553k 14G 1% /run 2024-12-18T00:29:19.3250627Z /dev/nvme0n1p1 161G 59G 103G 37% / 2024-12-18T00:29:19.3250955Z tmpfs 34G 13k 34G 1% /tmp 2024-12-18T00:29:19.3251304Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-12-18T00:29:19.3251670Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-12-18T00:29:19.3283681Z Prepare all required actions 2024-12-18T00:29:19.3284062Z Getting action download info 2024-12-18T00:29:19.5026990Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T00:29:19.5027345Z with: 2024-12-18T00:29:19.5027551Z env: 2024-12-18T00:29:19.5027772Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:19.5028118Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:19.5028553Z ##[endgroup] 2024-12-18T00:29:19.5227513Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:29:19.5227852Z with: 2024-12-18T00:29:19.5228066Z name: td_results 2024-12-18T00:29:19.5228420Z s3-bucket: gha-artifacts 2024-12-18T00:29:19.5228685Z region: us-east-1 2024-12-18T00:29:19.5228918Z env: 2024-12-18T00:29:19.5229138Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:19.5229478Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:19.5229843Z ##[endgroup] 2024-12-18T00:29:19.9698429Z (node:53481) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:29:19.9699302Z 2024-12-18T00:29:19.9699540Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:29:19.9700619Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:29:19.9701717Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:29:20.0695442Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/ 2024-12-18T00:29:20.0702696Z Artifact download has finished successfully 2024-12-18T00:29:20.1176718Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T00:29:20.1177289Z mkdir -p .additional_ci_files 2024-12-18T00:29:20.1177730Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T00:29:20.1187046Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:20.1187422Z env: 2024-12-18T00:29:20.1187638Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:20.1187993Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:20.1188420Z ##[endgroup] 2024-12-18T00:29:20.1245511Z mv: cannot stat 'td_results.json': No such file or directory 2024-12-18T00:29:20.1784138Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T00:29:20.1784516Z .github/scripts/parse_ref.py 2024-12-18T00:29:20.1793108Z shell: /usr/bin/bash -e {0} 2024-12-18T00:29:20.1793387Z env: 2024-12-18T00:29:20.1793609Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:20.1793962Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:20.1794336Z ##[endgroup] 2024-12-18T00:29:20.2070855Z Prepare all required actions 2024-12-18T00:29:20.2168906Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T00:29:20.2169259Z with: 2024-12-18T00:29:20.2169660Z github-token: *** 2024-12-18T00:29:20.2170117Z env: 2024-12-18T00:29:20.2170425Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:20.2170838Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:20.2171223Z ##[endgroup] 2024-12-18T00:29:20.2369275Z ##[group]Run set -eux 2024-12-18T00:29:20.2369683Z set -eux 2024-12-18T00:29:20.2370443Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T00:29:20.2380806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:20.2393238Z env: 2024-12-18T00:29:20.2393473Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:20.2393875Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:20.2394464Z GITHUB_TOKEN: *** 2024-12-18T00:29:20.2394712Z ##[endgroup] 2024-12-18T00:29:20.2422923Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-096ab043c4c0d1de7 2024-12-18T00:29:21.2758778Z setting job-id=34567152522 2024-12-18T00:29:21.2759889Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:21.2933333Z Prepare all required actions 2024-12-18T00:29:21.2933705Z Getting action download info 2024-12-18T00:29:21.4490475Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T00:29:21.4490858Z with: 2024-12-18T00:29:21.4491252Z github-token: *** 2024-12-18T00:29:21.4492956Z test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-12-18T00:29:21.4494957Z job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:21.4495480Z env: 2024-12-18T00:29:21.4495918Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:21.4496267Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:21.4496625Z ##[endgroup] 2024-12-18T00:29:21.4632471Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:29:21.4632775Z with: 2024-12-18T00:29:21.4632989Z shell: bash 2024-12-18T00:29:21.4633211Z timeout_minutes: 10 2024-12-18T00:29:21.4633459Z max_attempts: 5 2024-12-18T00:29:21.4633704Z retry_wait_seconds: 30 2024-12-18T00:29:21.4634476Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:29:21.4635274Z polling_interval_seconds: 1 2024-12-18T00:29:21.4635742Z warning_on_retry: true 2024-12-18T00:29:21.4636007Z continue_on_error: false 2024-12-18T00:29:21.4636269Z env: 2024-12-18T00:29:21.4636482Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:21.4636831Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:21.4637387Z GITHUB_TOKEN: *** 2024-12-18T00:29:21.4637623Z ##[endgroup] 2024-12-18T00:29:21.5618221Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:29:21.8037459Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:29:21.9907765Z Collecting requests==2.27.1 2024-12-18T00:29:22.0258828Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-12-18T00:29:22.2686407Z Collecting pyyaml==6.0.1 2024-12-18T00:29:22.2729091Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-12-18T00:29:22.4087732Z Collecting certifi>=2017.4.17 2024-12-18T00:29:22.4122804Z Downloading certifi-2024.12.14-py3-none-any.whl (164 kB) 2024-12-18T00:29:22.8158803Z Collecting charset-normalizer~=2.0.0 2024-12-18T00:29:22.8198192Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-12-18T00:29:22.8478118Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-12-18T00:29:22.8481424Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-12-18T00:29:22.9251893Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-12-18T00:29:23.2365423Z Successfully installed certifi-2024.12.14 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-12-18T00:29:23.5389431Z Command completed after 1 attempt(s). 2024-12-18T00:29:23.5512524Z ##[group]Run set -x 2024-12-18T00:29:23.5512791Z set -x 2024-12-18T00:29:23.5513026Z  2024-12-18T00:29:23.5513414Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:29:23.5513907Z # in runner workspace 2024-12-18T00:29:23.5514305Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T00:29:23.5523833Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:23.5524434Z env: 2024-12-18T00:29:23.5524662Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:23.5524997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:23.5525360Z ##[endgroup] 2024-12-18T00:29:23.5554288Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T00:29:23.5870109Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:29:23.5870530Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:29:23.5870887Z echo "Job name: ${JOB_NAME}" 2024-12-18T00:29:23.5871204Z  2024-12-18T00:29:23.5871638Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:29:23.5872122Z # in runner workspace 2024-12-18T00:29:23.5872576Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T00:29:23.5873056Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T00:29:23.5873408Z  --job-name "${JOB_NAME}" \ 2024-12-18T00:29:23.5875225Z  --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-12-18T00:29:23.5877037Z  --selected-test-configs "" \ 2024-12-18T00:29:23.5877388Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T00:29:23.5877891Z  --tag "${TAG}" \ 2024-12-18T00:29:23.5878199Z  --event-name "${EVENT_NAME}" \ 2024-12-18T00:29:23.5878534Z  --schedule "${SCHEDULE}" \ 2024-12-18T00:29:23.5878871Z  --branch "${HEAD_BRANCH}" 2024-12-18T00:29:23.5887749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:23.5888123Z env: 2024-12-18T00:29:23.5888353Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:23.5888704Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:23.5889266Z GITHUB_TOKEN: *** 2024-12-18T00:29:23.5889761Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:23.5890297Z PR_NUMBER: 2024-12-18T00:29:23.5890519Z TAG: 2024-12-18T00:29:23.5890741Z EVENT_NAME: push 2024-12-18T00:29:23.5890985Z SCHEDULE: 2024-12-18T00:29:23.5891212Z HEAD_BRANCH: 2024-12-18T00:29:23.5891452Z ##[endgroup] 2024-12-18T00:29:23.5920108Z Workflow: inductor 2024-12-18T00:29:23.5920618Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:23.8578287Z ##[group]Run echo "Filtered matrix:" 2024-12-18T00:29:23.8578637Z echo "Filtered matrix:" 2024-12-18T00:29:23.8580736Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-12-18T00:29:23.8582811Z  2024-12-18T00:29:23.8583032Z echo 2024-12-18T00:29:23.8583329Z echo "Is the current job unstable? False" 2024-12-18T00:29:23.8583700Z  2024-12-18T00:29:23.8583927Z echo 2024-12-18T00:29:23.8584204Z echo "Is keep-going label set? False" 2024-12-18T00:29:23.8584543Z  2024-12-18T00:29:23.8584755Z echo 2024-12-18T00:29:23.8585205Z echo "Renabled issues? " 2024-12-18T00:29:23.8594102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:23.8594485Z env: 2024-12-18T00:29:23.8594701Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:23.8595053Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:23.8595416Z ##[endgroup] 2024-12-18T00:29:23.8624006Z Filtered matrix: 2024-12-18T00:29:23.8625688Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-12-18T00:29:23.8627333Z 2024-12-18T00:29:23.8627471Z Is the current job unstable? False 2024-12-18T00:29:23.8627690Z 2024-12-18T00:29:23.8627816Z Is keep-going label set? False 2024-12-18T00:29:23.8628017Z 2024-12-18T00:29:23.8628115Z Renabled issues? 2024-12-18T00:29:23.8907545Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:29:23.8908072Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:29:23.8916495Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:29:23.8916965Z env: 2024-12-18T00:29:23.8917413Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:23.8917861Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:23.8918316Z JOB_TIMEOUT: 240 2024-12-18T00:29:23.8918708Z ##[endgroup] 2024-12-18T00:29:23.9119886Z ##[group]Run set -x 2024-12-18T00:29:23.9120201Z set -x 2024-12-18T00:29:23.9120605Z  2024-12-18T00:29:23.9120866Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T00:29:23.9121271Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T00:29:23.9121675Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T00:29:23.9122044Z  TEST_COMMAND=.ci/onnx/test.sh 2024-12-18T00:29:23.9122353Z else 2024-12-18T00:29:23.9122616Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:29:23.9122931Z fi 2024-12-18T00:29:23.9123147Z  2024-12-18T00:29:23.9123484Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T00:29:23.9124032Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T00:29:23.9124502Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T00:29:23.9124923Z # shellcheck disable=SC2086,SC2090 2024-12-18T00:29:23.9125263Z container_name=$(docker run \ 2024-12-18T00:29:23.9125580Z  ${GPU_FLAG:-} \ 2024-12-18T00:29:23.9125898Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-12-18T00:29:23.9126249Z  -e BUILD_ENVIRONMENT \ 2024-12-18T00:29:23.9126560Z  -e PR_NUMBER \ 2024-12-18T00:29:23.9126833Z  -e GITHUB_ACTIONS \ 2024-12-18T00:29:23.9127130Z  -e GITHUB_REPOSITORY \ 2024-12-18T00:29:23.9127436Z  -e GITHUB_WORKFLOW \ 2024-12-18T00:29:23.9127733Z  -e GITHUB_JOB \ 2024-12-18T00:29:23.9128012Z  -e GITHUB_RUN_ID \ 2024-12-18T00:29:23.9128293Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T00:29:23.9128597Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T00:29:23.9128893Z  -e JOB_ID \ 2024-12-18T00:29:23.9129158Z  -e JOB_NAME \ 2024-12-18T00:29:23.9129424Z  -e BASE_SHA \ 2024-12-18T00:29:23.9129678Z  -e BRANCH \ 2024-12-18T00:29:23.9129944Z  -e SHA1 \ 2024-12-18T00:29:23.9130211Z  -e AWS_DEFAULT_REGION \ 2024-12-18T00:29:23.9130523Z  -e IN_WHEEL_TEST \ 2024-12-18T00:29:23.9130805Z  -e SHARD_NUMBER \ 2024-12-18T00:29:23.9131079Z  -e TEST_CONFIG \ 2024-12-18T00:29:23.9131364Z  -e NUM_TEST_SHARDS \ 2024-12-18T00:29:23.9131667Z  -e REENABLED_ISSUES \ 2024-12-18T00:29:23.9131979Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T00:29:23.9132299Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T00:29:23.9132592Z  -e TEST_SHOWLOCALS \ 2024-12-18T00:29:23.9132885Z  -e NO_TEST_TIMEOUT \ 2024-12-18T00:29:23.9133169Z  -e NO_TD \ 2024-12-18T00:29:23.9133433Z  -e TD_DISTRIBUTED \ 2024-12-18T00:29:23.9133720Z  -e PR_LABELS \ 2024-12-18T00:29:23.9134022Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T00:29:23.9134352Z  -e SCCACHE_BUCKET \ 2024-12-18T00:29:23.9134649Z  -e SCCACHE_REGION \ 2024-12-18T00:29:23.9134946Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-12-18T00:29:23.9135256Z  -e XLA_CUDA \ 2024-12-18T00:29:23.9135555Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T00:29:23.9135915Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T00:29:23.9136291Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T00:29:23.9136667Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-12-18T00:29:23.9137018Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-12-18T00:29:23.9137354Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-12-18T00:29:23.9137680Z  -e DASHBOARD_TAG \ 2024-12-18T00:29:23.9137969Z  -e IS_A100_RUNNER \ 2024-12-18T00:29:23.9138268Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-12-18T00:29:23.9138643Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T00:29:23.9139052Z  --security-opt seccomp=unconfined \ 2024-12-18T00:29:23.9139535Z  --cap-add=SYS_PTRACE \ 2024-12-18T00:29:23.9139831Z  --ipc=host \ 2024-12-18T00:29:23.9140112Z  --shm-size="${SHM_SIZE}" \ 2024-12-18T00:29:23.9140499Z  --tty \ 2024-12-18T00:29:23.9140754Z  --detach \ 2024-12-18T00:29:23.9141030Z  --name="${container_name}" \ 2024-12-18T00:29:23.9141343Z  --user jenkins \ 2024-12-18T00:29:23.9141707Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T00:29:23.9142112Z  -w /var/lib/jenkins/workspace \ 2024-12-18T00:29:23.9142432Z  "${DOCKER_IMAGE}" 2024-12-18T00:29:23.9142702Z ) 2024-12-18T00:29:23.9142996Z # Propagate download.pytorch.org IP to container 2024-12-18T00:29:23.9143659Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-12-18T00:29:23.9144364Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-12-18T00:29:23.9145062Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-12-18T00:29:23.9153459Z shell: /usr/bin/bash -e {0} 2024-12-18T00:29:23.9153726Z env: 2024-12-18T00:29:23.9153935Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:29:23.9154273Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:23.9154720Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:29:23.9155096Z PR_NUMBER: 2024-12-18T00:29:23.9155349Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T00:29:23.9155671Z GITHUB_WORKFLOW: inductor 2024-12-18T00:29:23.9155934Z GITHUB_JOB: test 2024-12-18T00:29:23.9156179Z GITHUB_RUN_ID: 12383255690 2024-12-18T00:29:23.9156454Z GITHUB_RUN_NUMBER: 109429 2024-12-18T00:29:23.9156742Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T00:29:23.9156998Z JOB_ID: 34567152522 2024-12-18T00:29:23.9157478Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:23.9158010Z BRANCH: release/2.6 2024-12-18T00:29:23.9158293Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:23.9158689Z BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:23.9159057Z TEST_CONFIG: inductor_huggingface 2024-12-18T00:29:23.9159357Z SHARD_NUMBER: 1 2024-12-18T00:29:23.9159594Z NUM_TEST_SHARDS: 1 2024-12-18T00:29:23.9159842Z REENABLED_ISSUES: 2024-12-18T00:29:23.9160106Z CONTINUE_THROUGH_ERROR: False 2024-12-18T00:29:23.9160399Z VERBOSE_TEST_LOGS: False 2024-12-18T00:29:23.9160687Z TEST_SHOWLOCALS: False 2024-12-18T00:29:23.9160958Z NO_TEST_TIMEOUT: False 2024-12-18T00:29:23.9161215Z NO_TD: False 2024-12-18T00:29:23.9161464Z TD_DISTRIBUTED: False 2024-12-18T00:29:23.9161790Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T00:29:23.9162161Z SCCACHE_REGION: us-east-1 2024-12-18T00:29:23.9162449Z SCCACHE_S3_KEY_PREFIX: inductor 2024-12-18T00:29:23.9162729Z SHM_SIZE: 2g 2024-12-18T00:29:23.9163530Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:29:23.9164379Z XLA_CUDA: 2024-12-18T00:29:23.9164744Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:29:23.9165205Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T00:29:23.9165529Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T00:29:23.9165824Z DASHBOARD_TAG: 2024-12-18T00:29:23.9166243Z HUGGING_FACE_HUB_TOKEN: *** 2024-12-18T00:29:23.9166671Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-12-18T00:29:23.9166966Z IS_A100_RUNNER: 0 2024-12-18T00:29:23.9167445Z ARTIFACTS_FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T00:29:23.9167981Z ##[endgroup] 2024-12-18T00:29:23.9194629Z + [[ inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2024-12-18T00:29:23.9195193Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-12-18T00:29:23.9195865Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:29:23.9203983Z +++ nproc --ignore=2 2024-12-18T00:29:23.9432822Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:29:30.2775022Z + container_name=2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T00:29:30.2781172Z + grep download.pytorch.org /etc/hosts 2024-12-18T00:29:30.2782190Z + docker exec -i 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 sudo bash -c '/bin/cat >> /etc/hosts' 2024-12-18T00:29:30.4156083Z + echo DOCKER_CONTAINER_ID=2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T00:29:30.4160217Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:29:30.4163282Z + docker exec -t 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-12-18T00:29:30.8120977Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1) 2024-12-18T00:29:31.1426325Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T00:29:31.1427900Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T00:29:31.1431180Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T00:29:31.1434261Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T00:29:31.1437618Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T00:29:31.1442457Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T00:29:31.1458703Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T00:29:31.1471797Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0) 2024-12-18T00:29:31.1489404Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4) 2024-12-18T00:29:31.1879839Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T00:29:31.7912269Z Installing collected packages: torch 2024-12-18T00:29:42.4128312Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-12-18T00:29:42.4129355Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-12-18T00:29:42.4129885Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T00:29:42.4942549Z + export TERM=vt100 2024-12-18T00:29:42.4942829Z + TERM=vt100 2024-12-18T00:29:42.4944965Z ++ dirname .ci/pytorch/test.sh 2024-12-18T00:29:42.4957518Z + source .ci/pytorch/common.sh 2024-12-18T00:29:42.4960911Z +++ dirname .ci/pytorch/common.sh 2024-12-18T00:29:42.4972585Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T00:29:42.4974595Z +++ declare -f -t trap_add 2024-12-18T00:29:42.4979886Z ++ set -ex 2024-12-18T00:29:42.4980362Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-12-18T00:29:42.4980819Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T00:29:42.4981659Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-12-18T00:29:42.4982039Z + [[ -d /var/lib/jenkins/workspace ]] 2024-12-18T00:29:42.4984665Z ++ stat -c %u /var/lib/jenkins/workspace 2024-12-18T00:29:42.5004720Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-12-18T00:29:42.5005148Z + trap_add cleanup_workspace EXIT 2024-12-18T00:29:42.5005471Z + trap_add_cmd=cleanup_workspace 2024-12-18T00:29:42.5005762Z + shift 2024-12-18T00:29:42.5005983Z + for trap_add_name in "$@" 2024-12-18T00:29:42.5013465Z +++ trap -p EXIT 2024-12-18T00:29:42.5016996Z ++ eval 'extract_trap_cmd ' 2024-12-18T00:29:42.5017307Z +++ extract_trap_cmd 2024-12-18T00:29:42.5017652Z +++ printf '%s\n' '' 2024-12-18T00:29:42.5017937Z ++ printf '%s\n' cleanup_workspace 2024-12-18T00:29:42.5020756Z + trap -- ' 2024-12-18T00:29:42.5021008Z cleanup_workspace' EXIT 2024-12-18T00:29:42.5021421Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-12-18T00:29:43.2198521Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-12-18T00:29:43.2220976Z + echo 'Environment variables:' 2024-12-18T00:29:43.2221284Z Environment variables: 2024-12-18T00:29:43.2222001Z + env 2024-12-18T00:29:43.2232957Z INSTALLED_DB=yes 2024-12-18T00:29:43.2233347Z NV_LIBCUBLAS_VERSION=12.4.5.8-1 2024-12-18T00:29:43.2233833Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:29:43.2234225Z NV_NVML_DEV_VERSION=12.4.127-1 2024-12-18T00:29:43.2234723Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:29:43.2235176Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:29:43.2235513Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4 2024-12-18T00:29:43.2236008Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:29:43.2236539Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:29:43.2237075Z HOSTNAME=2fd5f2f4acae 2024-12-18T00:29:43.2237760Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2238484Z GITHUB_ACTION=__self 2024-12-18T00:29:43.2238854Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:29:43.2243394Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-12-18T00:29:43.2247852Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1 2024-12-18T00:29:43.2248221Z NV_NVTX_VERSION=12.4.127-1 2024-12-18T00:29:43.2248500Z GITHUB_RUN_NUMBER=109429 2024-12-18T00:29:43.2248772Z TEST_CONFIG=inductor_huggingface 2024-12-18T00:29:43.2249078Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:29:43.2249411Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:29:43.2249720Z IS_A100_RUNNER=0 2024-12-18T00:29:43.2249975Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 2024-12-18T00:29:43.2250284Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1 2024-12-18T00:29:43.2250823Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:29:43.2251123Z NV_LIBNPP_VERSION=12.2.5.30-1 2024-12-18T00:29:43.2251420Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:29:43.2251759Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:29:43.2252107Z GITHUB_REF_TYPE=branch 2024-12-18T00:29:43.2252375Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:29:43.2252656Z NCCL_VERSION=2.21.5-1 2024-12-18T00:29:43.2252952Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2253284Z XLA_CUDA= 2024-12-18T00:29:43.2253608Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:29:43.2255818Z *** 2024-12-18T00:29:43.2256052Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:29:43.2256364Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:29:43.2256646Z GITHUB_ACTIONS=true 2024-12-18T00:29:43.2256904Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:43.2257233Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1 2024-12-18T00:29:43.2257602Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1 2024-12-18T00:29:43.2257962Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2258314Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:29:43.2258681Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2259226Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6 2024-12-18T00:29:43.2259726Z UCC_HOME=/usr 2024-12-18T00:29:43.2259968Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 2024-12-18T00:29:43.2260314Z VERBOSE_TEST_LOGS=False 2024-12-18T00:29:43.2260579Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:29:43.2260890Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-12-18T00:29:43.2261252Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:29:43.2261557Z NV_CUDA_CUDART_VERSION=12.4.127-1 2024-12-18T00:29:43.2261843Z SHARD_NUMBER=1 2024-12-18T00:29:43.2262084Z GITHUB_REF_PROTECTED=true 2024-12-18T00:29:43.2262345Z HOME=/var/lib/jenkins 2024-12-18T00:29:43.2262633Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:29:43.2262964Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:29:43.2263318Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:29:43.2263674Z SCCACHE_S3_KEY_PREFIX=inductor 2024-12-18T00:29:43.2263954Z CUDA_VERSION=12.4.1 2024-12-18T00:29:43.2264232Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1 2024-12-18T00:29:43.2264556Z NUM_TEST_SHARDS=1 2024-12-18T00:29:43.2264827Z UCX_HOME=/usr 2024-12-18T00:29:43.2265178Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1 2024-12-18T00:29:43.2265929Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2266765Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:43.2267589Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2268507Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:29:43.2269121Z GITHUB_EVENT_NAME=push 2024-12-18T00:29:43.2269384Z DASHBOARD_TAG= 2024-12-18T00:29:43.2269626Z GITHUB_RUN_ID=12383255690 2024-12-18T00:29:43.2270014Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1 2024-12-18T00:29:43.2270390Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-12-18T00:29:43.2271062Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2271712Z GITHUB_ACTOR=malfet 2024-12-18T00:29:43.2271968Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1 2024-12-18T00:29:43.2272246Z PR_NUMBER= 2024-12-18T00:29:43.2272465Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:29:43.2272730Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:29:43.2273074Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:29:43.2273419Z TERM=vt100 2024-12-18T00:29:43.2273653Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1 2024-12-18T00:29:43.2273968Z INSTALLED_VISION=yes 2024-12-18T00:29:43.2274217Z BRANCH=release/2.6 2024-12-18T00:29:43.2274472Z SCCACHE_REGION=us-east-1 2024-12-18T00:29:43.2274747Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:29:43.2275046Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:29:43.2275376Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:29:43.2275888Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:29:43.2276456Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:29:43.2276824Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:29:43.2277165Z REENABLED_ISSUES= 2024-12-18T00:29:43.2277395Z SHLVL=1 2024-12-18T00:29:43.2277599Z MAX_JOBS=14 2024-12-18T00:29:43.2277827Z NV_CUDA_LIB_VERSION=12.4.1-1 2024-12-18T00:29:43.2278097Z NVARCH=x86_64 2024-12-18T00:29:43.2278322Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:29:43.2278668Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2279058Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:29:43.2279360Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-12-18T00:29:43.2279813Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:29:43.2280240Z GITHUB_JOB=test 2024-12-18T00:29:43.2280514Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4 2024-12-18T00:29:43.2280931Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:29:43.2281314Z NO_TEST_TIMEOUT=False 2024-12-18T00:29:43.2281573Z TD_DISTRIBUTED=False 2024-12-18T00:29:43.2281837Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 2024-12-18T00:29:43.2282162Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:29:43.2282466Z NV_NVPROF_VERSION=12.4.127-1 2024-12-18T00:29:43.2282745Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:29:43.2283007Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:29:43.2283280Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:29:43.2284036Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:29:43.2284797Z GITHUB_BASE_REF= 2024-12-18T00:29:43.2285267Z ARTIFACTS_FILE_SUFFIX=test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T00:29:43.2285808Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:29:43.2286083Z CI=true 2024-12-18T00:29:43.2286310Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:29:43.2286622Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:29:43.2286906Z JOB_ID=34567152522 2024-12-18T00:29:43.2287148Z INSTALLED_PROTOBUF=yes 2024-12-18T00:29:43.2287397Z GITHUB_HEAD_REF= 2024-12-18T00:29:43.2287635Z GITHUB_ACTION_REF= 2024-12-18T00:29:43.2287928Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:29:43.2288283Z TEST_SHOWLOCALS=False 2024-12-18T00:29:43.2288538Z GITHUB_WORKFLOW=inductor 2024-12-18T00:29:43.2288817Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:29:43.2289425Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2290040Z NO_TD=False 2024-12-18T00:29:43.2290273Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:29:43.2290552Z _=/usr/bin/env 2024-12-18T00:29:43.2291796Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T00:29:43.2472730Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T00:29:43.2474448Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T00:29:43.2475646Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T00:29:43.2476829Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T00:29:43.2477385Z + BUILD_DIR=build 2024-12-18T00:29:43.2477642Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T00:29:43.2478013Z + BUILD_BIN_DIR=build/bin 2024-12-18T00:29:43.2478283Z + SHARD_NUMBER=1 2024-12-18T00:29:43.2478525Z + NUM_TEST_SHARDS=1 2024-12-18T00:29:43.2478814Z + export VALGRIND=ON 2024-12-18T00:29:43.2479094Z + VALGRIND=ON 2024-12-18T00:29:43.2479405Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-12-18T00:29:43.2479918Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-12-18T00:29:43.2480273Z + [[ 0 == \1 ]] 2024-12-18T00:29:43.2480551Z + [[ False == \1 ]] 2024-12-18T00:29:43.2480886Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-12-18T00:29:43.2481295Z ++ realpath build/custom_test_artifacts 2024-12-18T00:29:43.2490516Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-12-18T00:29:43.2491043Z + [[ -n '' ]] 2024-12-18T00:29:43.2491328Z + echo 'Environment variables' 2024-12-18T00:29:43.2491652Z Environment variables 2024-12-18T00:29:43.2491901Z + env 2024-12-18T00:29:43.2501137Z INSTALLED_DB=yes 2024-12-18T00:29:43.2501505Z NV_LIBCUBLAS_VERSION=12.4.5.8-1 2024-12-18T00:29:43.2501865Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:29:43.2502236Z NV_NVML_DEV_VERSION=12.4.127-1 2024-12-18T00:29:43.2502819Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:29:43.2503394Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:29:43.2503851Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4 2024-12-18T00:29:43.2504337Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:29:43.2504856Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:29:43.2505359Z HOSTNAME=2fd5f2f4acae 2024-12-18T00:29:43.2505985Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2506603Z GITHUB_ACTION=__self 2024-12-18T00:29:43.2506880Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:29:43.2511314Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-12-18T00:29:43.2516645Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1 2024-12-18T00:29:43.2517052Z NV_NVTX_VERSION=12.4.127-1 2024-12-18T00:29:43.2517335Z GITHUB_RUN_NUMBER=109429 2024-12-18T00:29:43.2517616Z TEST_CONFIG=inductor_huggingface 2024-12-18T00:29:43.2517919Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:29:43.2518251Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:29:43.2518798Z IS_A100_RUNNER=0 2024-12-18T00:29:43.2519067Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 2024-12-18T00:29:43.2519512Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1 2024-12-18T00:29:43.2519977Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:29:43.2520272Z NV_LIBNPP_VERSION=12.2.5.30-1 2024-12-18T00:29:43.2520573Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:29:43.2520914Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:29:43.2521274Z GITHUB_REF_TYPE=branch 2024-12-18T00:29:43.2521545Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:29:43.2521819Z NCCL_VERSION=2.21.5-1 2024-12-18T00:29:43.2522127Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2522468Z XLA_CUDA= 2024-12-18T00:29:43.2522808Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:29:43.2523283Z *** 2024-12-18T00:29:43.2523505Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:29:43.2523813Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:29:43.2524100Z GITHUB_ACTIONS=true 2024-12-18T00:29:43.2524375Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:29:43.2524718Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1 2024-12-18T00:29:43.2525085Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1 2024-12-18T00:29:43.2525447Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2525797Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:29:43.2526162Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2526771Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6 2024-12-18T00:29:43.2527275Z UCC_HOME=/usr 2024-12-18T00:29:43.2527516Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 2024-12-18T00:29:43.2527823Z VERBOSE_TEST_LOGS=False 2024-12-18T00:29:43.2528090Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:29:43.2528411Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-12-18T00:29:43.2528766Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:29:43.2529072Z NV_CUDA_CUDART_VERSION=12.4.127-1 2024-12-18T00:29:43.2529360Z SHARD_NUMBER=1 2024-12-18T00:29:43.2529606Z GITHUB_REF_PROTECTED=true 2024-12-18T00:29:43.2529882Z HOME=/var/lib/jenkins 2024-12-18T00:29:43.2530163Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:29:43.2530503Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:29:43.2530856Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:29:43.2531215Z SCCACHE_S3_KEY_PREFIX=inductor 2024-12-18T00:29:43.2531501Z CUDA_VERSION=12.4.1 2024-12-18T00:29:43.2531778Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1 2024-12-18T00:29:43.2532115Z NUM_TEST_SHARDS=1 2024-12-18T00:29:43.2532353Z UCX_HOME=/usr 2024-12-18T00:29:43.2532705Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1 2024-12-18T00:29:43.2533454Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2534299Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T00:29:43.2535132Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2535920Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:29:43.2536425Z GITHUB_EVENT_NAME=push 2024-12-18T00:29:43.2536693Z DASHBOARD_TAG= 2024-12-18T00:29:43.2536938Z GITHUB_RUN_ID=12383255690 2024-12-18T00:29:43.2537253Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1 2024-12-18T00:29:43.2537662Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-12-18T00:29:43.2538342Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2538997Z GITHUB_ACTOR=malfet 2024-12-18T00:29:43.2539258Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1 2024-12-18T00:29:43.2539549Z PR_NUMBER= 2024-12-18T00:29:43.2539777Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:29:43.2540022Z VALGRIND=ON 2024-12-18T00:29:43.2540259Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:29:43.2540606Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:29:43.2541061Z TERM=vt100 2024-12-18T00:29:43.2541302Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1 2024-12-18T00:29:43.2541616Z INSTALLED_VISION=yes 2024-12-18T00:29:43.2541951Z BRANCH=release/2.6 2024-12-18T00:29:43.2542204Z SCCACHE_REGION=us-east-1 2024-12-18T00:29:43.2542485Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:29:43.2542787Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:29:43.2543111Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:29:43.2543628Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:29:43.2544199Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:29:43.2544565Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:29:43.2544912Z REENABLED_ISSUES= 2024-12-18T00:29:43.2545148Z SHLVL=1 2024-12-18T00:29:43.2545359Z MAX_JOBS=14 2024-12-18T00:29:43.2545589Z NV_CUDA_LIB_VERSION=12.4.1-1 2024-12-18T00:29:43.2545862Z NVARCH=x86_64 2024-12-18T00:29:43.2546093Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:29:43.2546445Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:29:43.2546841Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:29:43.2547148Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-12-18T00:29:43.2547602Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:29:43.2548028Z GITHUB_JOB=test 2024-12-18T00:29:43.2548416Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4 2024-12-18T00:29:43.2548834Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:29:43.2549220Z NO_TEST_TIMEOUT=False 2024-12-18T00:29:43.2549479Z TD_DISTRIBUTED=False 2024-12-18T00:29:43.2549750Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 2024-12-18T00:29:43.2550082Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:29:43.2550387Z NV_NVPROF_VERSION=12.4.127-1 2024-12-18T00:29:43.2550670Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:29:43.2550945Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:29:43.2551214Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:29:43.2551977Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:29:43.2552745Z GITHUB_BASE_REF= 2024-12-18T00:29:43.2553219Z ARTIFACTS_FILE_SUFFIX=test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T00:29:43.2553772Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:29:43.2554052Z CI=true 2024-12-18T00:29:43.2554287Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:29:43.2554604Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:29:43.2554888Z JOB_ID=34567152522 2024-12-18T00:29:43.2555136Z INSTALLED_PROTOBUF=yes 2024-12-18T00:29:43.2555390Z GITHUB_HEAD_REF= 2024-12-18T00:29:43.2555631Z GITHUB_ACTION_REF= 2024-12-18T00:29:43.2555934Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:29:43.2556294Z TEST_SHOWLOCALS=False 2024-12-18T00:29:43.2556559Z GITHUB_WORKFLOW=inductor 2024-12-18T00:29:43.2556837Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:29:43.2557462Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_189cc028-5aee-49a9-b8c1-271b451b732c 2024-12-18T00:29:43.2558084Z NO_TD=False 2024-12-18T00:29:43.2558328Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:29:43.2558607Z _=/usr/bin/env 2024-12-18T00:29:43.2558837Z + echo 'Testing pytorch' 2024-12-18T00:29:43.2559108Z Testing pytorch 2024-12-18T00:29:43.2559355Z + export LANG=C.UTF-8 2024-12-18T00:29:43.2559602Z + LANG=C.UTF-8 2024-12-18T00:29:43.2559830Z + PR_NUMBER= 2024-12-18T00:29:43.2560082Z + [[ inductor_huggingface == \d\e\f\a\u\l\t ]] 2024-12-18T00:29:43.2560466Z + [[ inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T00:29:43.2560835Z + [[ inductor_huggingface == \s\l\o\w ]] 2024-12-18T00:29:43.2561253Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-12-18T00:29:43.2561722Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-12-18T00:29:43.2562114Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:29:43.2562469Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:29:43.2562900Z + [[ inductor_huggingface == *crossref* ]] 2024-12-18T00:29:43.2563283Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-12-18T00:29:43.2563777Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-12-18T00:29:43.2564198Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-12-18T00:29:43.2564586Z + pip_install --user ninja==1.10.2 2024-12-18T00:29:43.2564979Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:29:43.2565469Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T00:29:44.4409189Z Collecting ninja==1.10.2 2024-12-18T00:29:44.4584051Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T00:29:44.5181661Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T00:29:45.1411873Z Installing collected packages: ninja 2024-12-18T00:29:45.1490344Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T00:29:45.1491459Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T00:29:45.2026356Z Successfully installed ninja-1.10.2 2024-12-18T00:29:45.2872015Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:29:45.2875135Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:29:45.2876984Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-12-18T00:29:45.2877355Z + install_tlparse 2024-12-18T00:29:45.2877621Z + pip_install --user tlparse==0.3.25 2024-12-18T00:29:45.2878020Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:29:45.2878520Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T00:29:45.7278437Z Collecting tlparse==0.3.25 2024-12-18T00:29:45.7449741Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T00:29:45.8047534Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T00:29:46.4844821Z Installing collected packages: tlparse 2024-12-18T00:29:46.5181994Z Successfully installed tlparse-0.3.25 2024-12-18T00:29:46.5962790Z ++ python -m site --user-base 2024-12-18T00:29:46.6206708Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:29:46.6207933Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-12-18T00:29:46.6208379Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-12-18T00:29:46.6209298Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-12-18T00:29:46.6210151Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-12-18T00:29:46.6211011Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-12-18T00:29:46.6211512Z + cd test 2024-12-18T00:29:46.6211861Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T00:29:48.3098276Z + [[ inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T00:29:48.3099136Z + [[ inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T00:29:48.3103494Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T00:29:48.3104026Z + [[ inductor_huggingface == *pr_time_benchmarks* ]] 2024-12-18T00:29:48.3104612Z + [[ inductor_huggingface == *dynamo_eager* ]] 2024-12-18T00:29:48.3105145Z + [[ inductor_huggingface == *aot_eager* ]] 2024-12-18T00:29:48.3105669Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-12-18T00:29:48.3106587Z + [[ inductor_huggingface == *inductor* ]] 2024-12-18T00:29:48.3107109Z + [[ inductor_huggingface != *perf* ]] 2024-12-18T00:29:48.3107824Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-12-18T00:29:48.3108483Z + [[ inductor_huggingface == *dynamic* ]] 2024-12-18T00:29:48.3108986Z + [[ inductor_huggingface == *cpu* ]] 2024-12-18T00:29:48.3109463Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T00:29:48.3138612Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-12-18T00:29:48.3139098Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-12-18T00:29:48.3142841Z + cd test 2024-12-18T00:29:48.3143144Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T00:29:49.8202595Z PyTorch built with: 2024-12-18T00:29:49.8202914Z - GCC 9.4 2024-12-18T00:29:49.8203150Z - C++ Version: 201703 2024-12-18T00:29:49.8203704Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:29:49.8204454Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:29:49.8204905Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:29:49.8205263Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T00:29:49.8205601Z - NNPACK is enabled 2024-12-18T00:29:49.8205874Z - CPU capability usage: AVX2 2024-12-18T00:29:49.8206160Z - CUDA Runtime 12.4 2024-12-18T00:29:49.8206519Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-12-18T00:29:49.8206913Z - CuDNN 90.1 2024-12-18T00:29:49.8207149Z - Magma 2.6.1 2024-12-18T00:29:49.8212106Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T00:29:49.8217209Z 2024-12-18T00:29:50.1584727Z + cd test 2024-12-18T00:29:50.1585126Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T00:29:51.5296780Z ATen/Parallel: 2024-12-18T00:29:51.5297093Z at::get_num_threads() : 8 2024-12-18T00:29:51.5297410Z at::get_num_interop_threads() : 16 2024-12-18T00:29:51.5297738Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:29:51.5298052Z omp_get_max_threads() : 8 2024-12-18T00:29:51.5298612Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:29:51.5299494Z mkl_get_max_threads() : 8 2024-12-18T00:29:51.5299898Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:29:51.5300360Z std::thread::hardware_concurrency() : 16 2024-12-18T00:29:51.5300694Z Environment variables: 2024-12-18T00:29:51.5300975Z OMP_NUM_THREADS : [not set] 2024-12-18T00:29:51.5301258Z MKL_NUM_THREADS : [not set] 2024-12-18T00:29:51.5301551Z ATen parallel backend: OpenMP 2024-12-18T00:29:51.5301751Z 2024-12-18T00:29:51.8131714Z + [[ inductor_huggingface == *numpy_2* ]] 2024-12-18T00:29:51.8132590Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-12-18T00:29:51.8133031Z + [[ inductor_huggingface == *backward* ]] 2024-12-18T00:29:51.8133542Z + [[ inductor_huggingface == *xla* ]] 2024-12-18T00:29:51.8133874Z + [[ inductor_huggingface == *executorch* ]] 2024-12-18T00:29:51.8134244Z + [[ inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T00:29:51.8134667Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-12-18T00:29:51.8135073Z + [[ inductor_huggingface == distributed ]] 2024-12-18T00:29:51.8135453Z + [[ inductor_huggingface == *inductor_distributed* ]] 2024-12-18T00:29:51.8135845Z + [[ inductor_huggingface == *inductor-halide* ]] 2024-12-18T00:29:51.8136247Z + [[ inductor_huggingface == *inductor-triton-cpu* ]] 2024-12-18T00:29:51.8136678Z + [[ inductor_huggingface == *inductor-micro-benchmark* ]] 2024-12-18T00:29:51.8137085Z + [[ inductor_huggingface == *huggingface* ]] 2024-12-18T00:29:51.8137415Z + install_torchvision 2024-12-18T00:29:51.8137665Z + local orig_preload 2024-12-18T00:29:51.8137920Z + local commit 2024-12-18T00:29:51.8138169Z ++ get_pinned_commit vision 2024-12-18T00:29:51.8138465Z ++ cat .github/ci_commit_pins/vision.txt 2024-12-18T00:29:51.8155497Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:51.8155833Z + orig_preload= 2024-12-18T00:29:51.8156067Z + '[' -n '' ']' 2024-12-18T00:29:51.8156686Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:51.8157379Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:29:51.8158166Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:52.1543089Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:52.1547302Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-e9ikdd3j 2024-12-18T00:29:52.1578588Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-e9ikdd3j 2024-12-18T00:29:53.6743329Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-12-18T00:29:53.6769693Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:55.0550366Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:55.4206823Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-12-18T00:29:58.0365970Z Preparing metadata (setup.py) ... [?25l- \ done 2024-12-18T00:29:58.0398744Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-12-18T00:29:58.0402635Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.6.0a0+git0cdf8b1) 2024-12-18T00:29:58.0407372Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0) 2024-12-18T00:29:58.0477191Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1) 2024-12-18T00:29:58.0481381Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-12-18T00:29:58.0485137Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-12-18T00:29:58.0488488Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-12-18T00:29:58.0492251Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0) 2024-12-18T00:29:58.0497292Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-12-18T00:29:58.0512778Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-12-18T00:29:58.0994458Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2) 2024-12-18T00:29:58.1058870Z Building wheels for collected packages: torchvision 2024-12-18T00:31:12.7724513Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-12-18T00:31:12.7756267Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2027674 sha256=5a4999896ebf3cde40c82aebb12e58d29028755ea77de372d961b507951c3ef3 2024-12-18T00:31:12.7757684Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-12-18T00:31:12.7791030Z Successfully built torchvision 2024-12-18T00:31:13.2770393Z Installing collected packages: torchvision 2024-12-18T00:31:13.6925815Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-12-18T00:31:13.8327252Z + '[' -n '' ']' 2024-12-18T00:31:13.8327736Z + id=0 2024-12-18T00:31:13.8328308Z + test_dynamo_benchmark huggingface 0 2024-12-18T00:31:13.8330190Z ++ pwd 2024-12-18T00:31:13.8333662Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-12-18T00:31:13.8334092Z + local suite=huggingface 2024-12-18T00:31:13.8334352Z + shift 2024-12-18T00:31:13.8334554Z + local shard_id=0 2024-12-18T00:31:13.8334881Z + shift 2024-12-18T00:31:13.8335230Z + [[ inductor_huggingface == *perf_compare* ]] 2024-12-18T00:31:13.8335611Z + [[ inductor_huggingface == *perf* ]] 2024-12-18T00:31:13.8335934Z + [[ inductor_huggingface == *cpu* ]] 2024-12-18T00:31:13.8336262Z + [[ inductor_huggingface == *aot_inductor* ]] 2024-12-18T00:31:13.8336837Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16 2024-12-18T00:31:13.8337404Z ++ pwd 2024-12-18T00:31:13.8339260Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-12-18T00:31:13.8339749Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-12-18T00:31:13.8394958Z + local name=inference 2024-12-18T00:31:13.8395291Z + shift 2024-12-18T00:31:13.8395521Z + local suite=huggingface 2024-12-18T00:31:13.8395842Z + shift 2024-12-18T00:31:13.8396069Z + local shard_id=0 2024-12-18T00:31:13.8396293Z + shift 2024-12-18T00:31:13.8396507Z + partition_flags=() 2024-12-18T00:31:13.8396763Z + local partition_flags 2024-12-18T00:31:13.8397033Z + [[ -n 1 ]] 2024-12-18T00:31:13.8397255Z + [[ -n 0 ]] 2024-12-18T00:31:13.8397670Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-12-18T00:31:13.8398204Z + [[ inductor_huggingface == *perf_compare* ]] 2024-12-18T00:31:13.8398563Z + [[ inductor_huggingface == *perf* ]] 2024-12-18T00:31:13.8399224Z + [[ inductor_huggingface == *_avx2* ]] 2024-12-18T00:31:13.8399554Z + [[ inductor_huggingface == *_avx512* ]] 2024-12-18T00:31:13.8400656Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2024-12-18T00:31:17.3541065Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T00:31:17.3543522Z warnings.warn( 2024-12-18T00:31:17.4333397Z 2024-12-18T00:31:17.4333985Z config.json: 0% 0.00/694 [00:00 will be ignored 2024-12-18T00:59:36.5099826Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T00:59:45.7331030Z pass 2024-12-18T00:59:45.7677877Z TIMING: entire_frame_compile:25.21967 _recursive_pre_grad_passes:0.0215 _recursive_joint_graph_passes:2.41 _recursive_post_grad_passes:0.81209 async_compile.wait:5.97155 code_gen:13.08457 inductor_compile:20.37608 backend_compile:21.38514 entire_backward_compile:9.10774 total_wall_time:34.3274 2024-12-18T00:59:45.7681046Z STATS: call_* op count: 585 | FakeTensorMode.__torch_dispatch__:40401 | FakeTensor.__torch_dispatch__:7553 | ProxyTorchDispatchMode.__torch_dispatch__:20178 2024-12-18T00:59:45.7681931Z Dynamo produced 2 graphs covering 585 ops with 5 graph breaks (4 unique) 2024-12-18T00:59:51.3640882Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T00:59:51.3642097Z warnings.warn( 2024-12-18T00:59:51.6455796Z 2024-12-18T00:59:54.5107639Z loading model: 0it [00:00, ?it/s] 2024-12-18T00:59:54.5108063Z loading model: 0it [00:02, ?it/s] 2024-12-18T00:59:54.5108485Z cuda train AllenaiLongformerBase 2024-12-18T01:00:03.5949560Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-12-18T01:00:03.5950616Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-12-18T01:00:03.5951431Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] or: 2024-12-18T01:00:03.5952218Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-12-18T01:00:03.5953158Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] to include these operations in the captured graph. 2024-12-18T01:00:03.5953932Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 2024-12-18T01:00:03.5954670Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] Graph break: from user code at: 2024-12-18T01:00:03.5955945Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:00:03.5957231Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] pred = mod(**cloned_inputs) 2024-12-18T01:00:03.5958499Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward 2024-12-18T01:00:03.5959750Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] outputs = self.longformer( 2024-12-18T01:00:03.5961474Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward 2024-12-18T01:00:03.5962739Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] encoder_outputs = self.encoder( 2024-12-18T01:00:03.5964167Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward 2024-12-18T01:00:03.5965495Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] is_global_attn = is_index_global_attn.flatten().any().item() 2024-12-18T01:00:03.5966306Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 2024-12-18T01:00:03.5966969Z W1218 01:00:03.593000 11397 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 2024-12-18T01:01:19.1547281Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:01:19.1550189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1318, in torch_dynamo_resume_in_forward_at_1291 2024-12-18T01:01:19.1561082Z layer_outputs = layer_module( 2024-12-18T01:01:19.1561857Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1246, in forward 2024-12-18T01:01:19.1562599Z self_attn_outputs = self.attention( 2024-12-18T01:01:19.1563324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1182, in forward 2024-12-18T01:01:19.1564045Z self_outputs = self.self( 2024-12-18T01:01:19.1564728Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 571, in forward 2024-12-18T01:01:19.1565487Z attn_scores = self._sliding_chunks_query_key_matmul( 2024-12-18T01:01:19.1566360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 836, in _sliding_chunks_query_key_matmul 2024-12-18T01:01:19.1567333Z query = self._chunk(query, window_overlap, getattr(self.config, "onnx_export", False)) 2024-12-18T01:01:19.1568263Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 778, in _chunk 2024-12-18T01:01:19.1569080Z return hidden_states.as_strided(size=chunk_size, stride=chunk_stride) 2024-12-18T01:01:19.1569406Z 2024-12-18T01:01:19.8644636Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:01:19.8645553Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1734, in forward 2024-12-18T01:01:19.8646295Z embedding_output = self.embeddings( 2024-12-18T01:01:19.8647022Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 470, in forward 2024-12-18T01:01:19.8647805Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:01:19.8648072Z 2024-12-18T01:01:19.9159412Z W1218 01:01:19.914000 11397 site-packages/torch/_logging/_internal.py:1089] [16/0] Profiler function will be ignored 2024-12-18T01:01:21.9873388Z ('Grad tensors ["L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:02:10.3084093Z pass 2024-12-18T01:02:10.3516529Z TIMING: entire_frame_compile:101.42545 _recursive_pre_grad_passes:0.10243 _recursive_joint_graph_passes:3.66169 inductor_compile:57.93001 backend_compile:77.7414 _recursive_post_grad_passes:3.25247 async_compile.wait:9.18112 code_gen:34.76224 entire_backward_compile:18.5661 total_wall_time:119.99155 2024-12-18T01:02:10.3518291Z STATS: call_* op count: 2772 | FakeTensorMode.__torch_dispatch__:122826 | FakeTensor.__torch_dispatch__:18597 | ProxyTorchDispatchMode.__torch_dispatch__:59644 2024-12-18T01:02:10.3519157Z Dynamo produced 7 graphs covering 2772 ops with 9 graph breaks (5 unique) 2024-12-18T01:02:19.2050256Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:02:19.2052462Z warnings.warn( 2024-12-18T01:02:19.4959329Z 2024-12-18T01:02:23.9170029Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:02:23.9170391Z loading model: 0it [00:04, ?it/s] 2024-12-18T01:02:23.9170762Z cuda train BartForCausalLM 2024-12-18T01:02:38.6578745Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:02:38.6580574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-12-18T01:02:38.6581782Z return super().forward(positions + self.offset) 2024-12-18T01:02:38.6582069Z 2024-12-18T01:02:44.5769233Z pass 2024-12-18T01:02:44.7047464Z TIMING: entire_frame_compile:4.64447 _recursive_pre_grad_passes:0.00559 _recursive_joint_graph_passes:0.95797 inductor_compile:2.83511 backend_compile:3.82252 _recursive_post_grad_passes:0.09837 async_compile.precompile:0.15219 async_compile.wait:0.9121 code_gen:1.99295 entire_backward_compile:1.43112 total_wall_time:6.07559 2024-12-18T01:02:44.7049287Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:3860 | FakeTensor.__torch_dispatch__:725 | ProxyTorchDispatchMode.__torch_dispatch__:1670 2024-12-18T01:02:44.7050146Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique) 2024-12-18T01:02:49.3476571Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:02:49.3477836Z warnings.warn( 2024-12-18T01:02:49.5905929Z 2024-12-18T01:02:56.4711793Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:02:56.4712159Z loading model: 0it [00:06, ?it/s] 2024-12-18T01:02:56.4712513Z cuda train BartForConditionalGeneration 2024-12-18T01:03:14.8133120Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:03:14.8134068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-12-18T01:03:14.8134776Z return super().forward(positions + self.offset) 2024-12-18T01:03:14.8135044Z 2024-12-18T01:03:36.8517853Z pass 2024-12-18T01:03:37.0624809Z TIMING: entire_frame_compile:6.58454 _recursive_pre_grad_passes:0.00768 _recursive_joint_graph_passes:0.33513 inductor_compile:4.20989 backend_compile:5.01068 _recursive_post_grad_passes:0.20808 async_compile.precompile:0.08488 async_compile.wait:0.96519 code_gen:2.58175 entire_backward_compile:2.15232 total_wall_time:8.73686 2024-12-18T01:03:37.0629065Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:8218 | FakeTensor.__torch_dispatch__:1555 | ProxyTorchDispatchMode.__torch_dispatch__:3844 2024-12-18T01:03:37.0629910Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique) 2024-12-18T01:03:41.8881041Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:03:41.8882291Z warnings.warn( 2024-12-18T01:03:42.1667815Z 2024-12-18T01:03:44.2557729Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:03:44.2558099Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:03:44.2558447Z cuda train BertForMaskedLM 2024-12-18T01:04:18.6051112Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:04:18.6053458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:04:18.6054301Z pred = mod(**cloned_inputs) 2024-12-18T01:04:18.6055033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-12-18T01:04:18.6055769Z outputs = self.bert( 2024-12-18T01:04:18.6056449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-12-18T01:04:18.6057165Z embedding_output = self.embeddings( 2024-12-18T01:04:18.6057896Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-12-18T01:04:18.6058652Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:04:18.6058983Z 2024-12-18T01:04:18.7629832Z W1218 01:04:18.761000 12473 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:04:20.5083682Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:05:00.4794482Z pass 2024-12-18T01:05:00.5437607Z TIMING: entire_frame_compile:61.3543 _recursive_pre_grad_passes:0.06211 _recursive_joint_graph_passes:2.57627 _recursive_post_grad_passes:0.81684 async_compile.wait:3.52043 code_gen:19.16928 inductor_compile:34.32671 backend_compile:47.33123 entire_backward_compile:8.30991 total_wall_time:69.66421 2024-12-18T01:05:00.5439628Z STATS: call_* op count: 1401 | FakeTensorMode.__torch_dispatch__:62380 | FakeTensor.__torch_dispatch__:15583 | ProxyTorchDispatchMode.__torch_dispatch__:28569 2024-12-18T01:05:00.5440518Z Dynamo produced 2 graphs covering 1401 ops with 5 graph breaks (4 unique) 2024-12-18T01:05:07.7552408Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:05:07.7554941Z warnings.warn( 2024-12-18T01:05:08.0387727Z 2024-12-18T01:05:09.8683600Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:05:09.8683979Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:05:09.8684327Z cuda train BertForQuestionAnswering 2024-12-18T01:05:43.3722887Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:05:43.3725692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:05:43.3726605Z pred = mod(**cloned_inputs) 2024-12-18T01:05:43.3727264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1846, in forward 2024-12-18T01:05:43.3727932Z outputs = self.bert( 2024-12-18T01:05:43.3728579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-12-18T01:05:43.3729252Z embedding_output = self.embeddings( 2024-12-18T01:05:43.3729921Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-12-18T01:05:43.3730610Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:05:43.3730863Z 2024-12-18T01:05:43.5306083Z W1218 01:05:43.529000 12796 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:05:45.2434131Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:06:22.9715895Z pass 2024-12-18T01:06:23.0389409Z TIMING: entire_frame_compile:58.91624 _recursive_pre_grad_passes:0.06127 _recursive_joint_graph_passes:2.54762 _recursive_post_grad_passes:1.12197 async_compile.wait:1.47752 code_gen:16.98464 inductor_compile:32.05603 backend_compile:45.10243 entire_backward_compile:7.97187 total_wall_time:66.88812 2024-12-18T01:06:23.0391158Z STATS: call_* op count: 1393 | FakeTensorMode.__torch_dispatch__:61892 | FakeTensor.__torch_dispatch__:15413 | ProxyTorchDispatchMode.__torch_dispatch__:28358 2024-12-18T01:06:23.0392007Z Dynamo produced 2 graphs covering 1393 ops with 5 graph breaks (4 unique) 2024-12-18T01:06:30.2141126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:06:30.2142355Z warnings.warn( 2024-12-18T01:06:30.5332278Z 2024-12-18T01:06:59.3236029Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:06:59.3237682Z loading model: 0it [00:28, ?it/s] 2024-12-18T01:06:59.3238276Z cuda train BlenderbotForCausalLM 2024-12-18T01:06:59.3244143Z Traceback (most recent call last): 2024-12-18T01:06:59.3244851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model 2024-12-18T01:06:59.3245436Z self.model_iter_fn(model, example_inputs) 2024-12-18T01:06:59.3246078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in forward_and_backward_pass 2024-12-18T01:06:59.3246707Z pred = mod(**cloned_inputs) 2024-12-18T01:06:59.3247416Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:06:59.3248084Z return self._call_impl(*args, **kwargs) 2024-12-18T01:06:59.3248704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:06:59.3249324Z return forward_call(*args, **kwargs) 2024-12-18T01:06:59.3250043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward 2024-12-18T01:06:59.3250771Z outputs = self.model.decoder( 2024-12-18T01:06:59.3251402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:06:59.3252050Z return self._call_impl(*args, **kwargs) 2024-12-18T01:06:59.3252656Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:06:59.3253258Z return forward_call(*args, **kwargs) 2024-12-18T01:06:59.3253976Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward 2024-12-18T01:06:59.3254697Z layer_outputs = decoder_layer( 2024-12-18T01:06:59.3255326Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:06:59.3255975Z return self._call_impl(*args, **kwargs) 2024-12-18T01:06:59.3256579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:06:59.3257180Z return forward_call(*args, **kwargs) 2024-12-18T01:06:59.3257889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward 2024-12-18T01:06:59.3258716Z hidden_states, self_attn_weights, present_key_value = self.self_attn( 2024-12-18T01:06:59.3259886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:06:59.3260541Z return self._call_impl(*args, **kwargs) 2024-12-18T01:06:59.3261315Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:06:59.3261914Z return forward_call(*args, **kwargs) 2024-12-18T01:06:59.3262633Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward 2024-12-18T01:06:59.3263405Z query_states = self.q_proj(hidden_states) * self.scaling 2024-12-18T01:06:59.3264107Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:06:59.3264748Z return self._call_impl(*args, **kwargs) 2024-12-18T01:06:59.3265352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:06:59.3265950Z return forward_call(*args, **kwargs) 2024-12-18T01:06:59.3266536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward 2024-12-18T01:06:59.3267179Z return F.linear(input, self.weight, self.bias) 2024-12-18T01:06:59.3269313Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 68323 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-12-18T01:06:59.3271245Z 2024-12-18T01:06:59.3271464Z The above exception was the direct cause of the following exception: 2024-12-18T01:06:59.3271795Z 2024-12-18T01:06:59.3271912Z Traceback (most recent call last): 2024-12-18T01:06:59.3272403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run 2024-12-18T01:06:59.3272903Z ) = runner.load_model( 2024-12-18T01:06:59.3273420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 458, in load_model 2024-12-18T01:06:59.3274002Z self.validate_model(model, example_inputs) 2024-12-18T01:06:59.3274898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model 2024-12-18T01:06:59.3275476Z raise RuntimeError("Eager run failed") from e 2024-12-18T01:06:59.3275833Z RuntimeError: Eager run failed 2024-12-18T01:06:59.3276032Z 2024-12-18T01:06:59.3276128Z eager_fail_to_run 2024-12-18T01:07:03.5359899Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:07:03.5361183Z warnings.warn( 2024-12-18T01:07:03.8049384Z 2024-12-18T01:07:05.2844932Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:07:05.2845326Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:07:05.2845671Z cuda train BlenderbotSmallForCausalLM 2024-12-18T01:07:05.3057789Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine 2024-12-18T01:07:13.6114580Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:07:13.6115519Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-12-18T01:07:13.6116293Z return super().forward(positions) 2024-12-18T01:07:13.6116511Z 2024-12-18T01:07:17.8771412Z pass 2024-12-18T01:07:17.9040471Z TIMING: entire_frame_compile:5.22975 _recursive_pre_grad_passes:0.00659 _recursive_joint_graph_passes:0.90357 inductor_compile:3.16082 backend_compile:4.1921 _recursive_post_grad_passes:0.11684 async_compile.precompile:0.07819 async_compile.wait:1.04775 code_gen:2.18394 entire_backward_compile:1.4717 total_wall_time:6.70145 2024-12-18T01:07:17.9042393Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4480 | FakeTensor.__torch_dispatch__:786 | ProxyTorchDispatchMode.__torch_dispatch__:1907 2024-12-18T01:07:17.9043228Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-12-18T01:07:22.5803986Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:07:22.5805203Z warnings.warn( 2024-12-18T01:07:22.9447519Z 2024-12-18T01:07:24.8496386Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:07:24.8496804Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:07:24.8497188Z cuda train BlenderbotSmallForConditionalGeneration 2024-12-18T01:07:24.9218395Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine 2024-12-18T01:07:28.0221465Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:07:28.0222399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 90, in forward 2024-12-18T01:07:28.0223180Z return super().forward(positions) 2024-12-18T01:07:28.0223397Z 2024-12-18T01:07:46.0903590Z pass 2024-12-18T01:07:46.1219267Z TIMING: entire_frame_compile:7.04273 _recursive_pre_grad_passes:0.0087 _recursive_joint_graph_passes:0.38354 inductor_compile:4.75228 backend_compile:5.3975 async_compile.precompile:0.02192 async_compile.wait:1.01231 _recursive_post_grad_passes:0.28049 code_gen:2.70747 entire_backward_compile:2.27543 total_wall_time:9.31816 2024-12-18T01:07:46.1221052Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9248 | FakeTensor.__torch_dispatch__:1664 | ProxyTorchDispatchMode.__torch_dispatch__:4249 2024-12-18T01:07:46.1221897Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique) 2024-12-18T01:07:50.9556514Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:07:50.9559102Z warnings.warn( 2024-12-18T01:07:51.2461828Z 2024-12-18T01:07:53.3490071Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:07:53.3490445Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:07:53.3490782Z cuda train CamemBert 2024-12-18T01:08:27.4917108Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:08:27.4919582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:08:27.4920447Z pred = mod(**cloned_inputs) 2024-12-18T01:08:27.4921230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 979, in forward 2024-12-18T01:08:27.4922033Z outputs = self.roberta( 2024-12-18T01:08:27.4922785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 881, in forward 2024-12-18T01:08:27.4923583Z embedding_output = self.embeddings( 2024-12-18T01:08:27.4924374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/camembert/modeling_camembert.py", line 139, in forward 2024-12-18T01:08:27.4925200Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:08:27.4925467Z 2024-12-18T01:08:27.6510516Z W1218 01:08:27.650000 13573 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:08:29.3879133Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:09:09.2791897Z pass 2024-12-18T01:09:09.3440494Z TIMING: entire_frame_compile:61.38903 _recursive_pre_grad_passes:0.06198 _recursive_joint_graph_passes:2.60036 _recursive_post_grad_passes:0.82177 async_compile.wait:3.20542 code_gen:18.89221 inductor_compile:34.08319 backend_compile:47.27339 entire_backward_compile:8.07456 total_wall_time:69.46358 2024-12-18T01:09:09.3442171Z STATS: call_* op count: 1409 | FakeTensorMode.__torch_dispatch__:62499 | FakeTensor.__torch_dispatch__:15605 | ProxyTorchDispatchMode.__torch_dispatch__:28617 2024-12-18T01:09:09.3443051Z Dynamo produced 2 graphs covering 1409 ops with 5 graph breaks (4 unique) 2024-12-18T01:09:16.6106539Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:09:16.6107833Z warnings.warn( 2024-12-18T01:09:16.9123431Z 2024-12-18T01:09:19.3859900Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:09:19.3860258Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:09:19.3860590Z cuda train DebertaForMaskedLM 2024-12-18T01:09:55.9804451Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:09:55.9807239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:09:55.9808022Z pred = mod(**cloned_inputs) 2024-12-18T01:09:55.9808764Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1062, in forward 2024-12-18T01:09:55.9809622Z outputs = self.deberta( 2024-12-18T01:09:55.9810513Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-12-18T01:09:55.9811300Z embedding_output = self.embeddings( 2024-12-18T01:09:55.9811981Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-12-18T01:09:55.9812692Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:09:55.9812943Z 2024-12-18T01:09:56.1305802Z W1218 01:09:56.129000 13872 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:09:57.5126462Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:10:32.3683253Z pass 2024-12-18T01:10:32.4299664Z TIMING: entire_frame_compile:57.88093 _recursive_pre_grad_passes:0.06391 _recursive_joint_graph_passes:2.44407 _recursive_post_grad_passes:1.12849 async_compile.wait:5.12806 code_gen:19.65867 inductor_compile:33.50757 backend_compile:44.66845 entire_backward_compile:8.59662 total_wall_time:66.47756 2024-12-18T01:10:32.4301380Z STATS: call_* op count: 1650 | FakeTensorMode.__torch_dispatch__:64235 | FakeTensor.__torch_dispatch__:14397 | ProxyTorchDispatchMode.__torch_dispatch__:29686 2024-12-18T01:10:32.4302266Z Dynamo produced 2 graphs covering 1650 ops with 5 graph breaks (4 unique) 2024-12-18T01:10:39.4337756Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:10:39.4339007Z warnings.warn( 2024-12-18T01:10:39.9363630Z 2024-12-18T01:10:42.0008651Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:10:42.0009019Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:10:42.0009366Z cuda train DebertaForQuestionAnswering 2024-12-18T01:11:16.5636212Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:11:16.5637111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:11:16.5637837Z pred = mod(**cloned_inputs) 2024-12-18T01:11:16.5638524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 1388, in forward 2024-12-18T01:11:16.5639212Z outputs = self.deberta( 2024-12-18T01:11:16.5639891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 966, in forward 2024-12-18T01:11:16.5640582Z embedding_output = self.embeddings( 2024-12-18T01:11:16.5641273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta/modeling_deberta.py", line 780, in forward 2024-12-18T01:11:16.5641997Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:11:16.5642250Z 2024-12-18T01:11:16.7074568Z W1218 01:11:16.706000 14259 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:11:18.0700828Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:11:50.5845584Z pass 2024-12-18T01:11:50.6524178Z TIMING: entire_frame_compile:54.27167 _recursive_pre_grad_passes:0.06297 _recursive_joint_graph_passes:1.74015 _recursive_post_grad_passes:1.12175 async_compile.wait:2.77147 code_gen:17.04062 inductor_compile:30.66646 backend_compile:41.16157 entire_backward_compile:8.38126 total_wall_time:62.65292 2024-12-18T01:11:50.6527648Z STATS: call_* op count: 1642 | FakeTensorMode.__torch_dispatch__:63705 | ProxyTorchDispatchMode.__torch_dispatch__:29466 | FakeTensor.__torch_dispatch__:14215 2024-12-18T01:11:50.6529390Z Dynamo produced 2 graphs covering 1642 ops with 5 graph breaks (4 unique) 2024-12-18T01:11:57.5755868Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:11:57.5757799Z warnings.warn( 2024-12-18T01:11:57.8535448Z 2024-12-18T01:12:10.3923384Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:12:10.3923748Z loading model: 0it [00:12, ?it/s] 2024-12-18T01:12:10.3924083Z cuda train DebertaV2ForMaskedLM 2024-12-18T01:12:10.4151560Z pass_due_to_skip 2024-12-18T01:12:10.6376493Z TIMING: total_wall_time:0 2024-12-18T01:12:10.6376781Z STATS: call_* op count: 0 2024-12-18T01:12:10.6377172Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-12-18T01:12:14.7488866Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:12:14.7490300Z warnings.warn( 2024-12-18T01:12:15.0732479Z 2024-12-18T01:12:25.8532789Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:12:25.8533171Z loading model: 0it [00:10, ?it/s] 2024-12-18T01:12:25.8533524Z cuda train DebertaV2ForQuestionAnswering 2024-12-18T01:12:36.5377499Z ERROR:common: 2024-12-18T01:12:36.5377811Z Traceback (most recent call last): 2024-12-18T01:12:36.5378376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2967, in check_accuracy 2024-12-18T01:12:36.5378952Z correct_result = self.run_n_iterations( 2024-12-18T01:12:36.5379511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2770, in run_n_iterations 2024-12-18T01:12:36.5380157Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-12-18T01:12:36.5380837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in forward_and_backward_pass 2024-12-18T01:12:36.5381481Z pred = mod(**cloned_inputs) 2024-12-18T01:12:36.5382108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5382760Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5383365Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5383961Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5384674Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward 2024-12-18T01:12:36.5385431Z outputs = self.deberta( 2024-12-18T01:12:36.5386041Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5386683Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5387299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5387901Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5388683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward 2024-12-18T01:12:36.5389390Z encoder_outputs = self.encoder( 2024-12-18T01:12:36.5390018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5390663Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5391264Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5391861Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5392566Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward 2024-12-18T01:12:36.5393275Z output_states = layer_module( 2024-12-18T01:12:36.5393897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5394542Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5395146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5395741Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5396446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward 2024-12-18T01:12:36.5397158Z attention_output = self.attention( 2024-12-18T01:12:36.5397795Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5398667Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5399529Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5400291Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5400987Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward 2024-12-18T01:12:36.5401679Z self_output = self.self( 2024-12-18T01:12:36.5402280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl 2024-12-18T01:12:36.5402918Z return self._call_impl(*args, **kwargs) 2024-12-18T01:12:36.5403516Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl 2024-12-18T01:12:36.5404108Z return forward_call(*args, **kwargs) 2024-12-18T01:12:36.5404804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward 2024-12-18T01:12:36.5405637Z attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1) 2024-12-18T01:12:36.5406333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply 2024-12-18T01:12:36.5406977Z return super().apply(*args, **kwargs) # type: ignore[misc] 2024-12-18T01:12:36.5407740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward 2024-12-18T01:12:36.5408447Z output = torch.softmax(output, self.dim) 2024-12-18T01:12:36.5410484Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 70046 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-12-18T01:12:36.5955043Z eager_1st_run_OOM 2024-12-18T01:12:36.5955470Z TIMING: total_wall_time:0 2024-12-18T01:12:36.5955778Z STATS: call_* op count: 0 2024-12-18T01:12:36.5956178Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-12-18T01:12:40.7705279Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:12:40.7706503Z warnings.warn( 2024-12-18T01:12:41.0275745Z 2024-12-18T01:12:42.5493384Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:12:42.5493742Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:12:42.5494122Z cuda train DistilBertForMaskedLM 2024-12-18T01:13:01.1001289Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:13:01.1002222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:13:01.1002993Z pred = mod(**cloned_inputs) 2024-12-18T01:13:01.1003713Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-12-18T01:13:01.1004432Z dlbrt_output = self.distilbert( 2024-12-18T01:13:01.1005140Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-12-18T01:13:01.1006004Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-12-18T01:13:01.1006865Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-12-18T01:13:01.1007961Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-12-18T01:13:01.1008457Z 2024-12-18T01:13:01.2365568Z W1218 01:13:01.235000 14598 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:13:02.1503297Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:13:24.6670327Z pass 2024-12-18T01:13:24.7071476Z TIMING: entire_frame_compile:34.51364 _recursive_pre_grad_passes:0.03532 _recursive_joint_graph_passes:1.91599 _recursive_post_grad_passes:0.71216 async_compile.wait:3.82529 code_gen:11.8626 inductor_compile:20.09493 backend_compile:27.48597 entire_backward_compile:4.1761 total_wall_time:38.68975 2024-12-18T01:13:24.7073535Z STATS: call_* op count: 752 | FakeTensorMode.__torch_dispatch__:33342 | FakeTensor.__torch_dispatch__:8096 | ProxyTorchDispatchMode.__torch_dispatch__:15052 2024-12-18T01:13:24.7074434Z Dynamo produced 2 graphs covering 752 ops with 5 graph breaks (4 unique) 2024-12-18T01:13:30.5372609Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:13:30.5373922Z warnings.warn( 2024-12-18T01:13:30.7817190Z 2024-12-18T01:13:32.1304625Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:13:32.1304973Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:13:32.1305322Z cuda train DistilBertForQuestionAnswering 2024-12-18T01:13:50.0518867Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:13:50.0519753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:13:50.0520479Z pred = mod(**cloned_inputs) 2024-12-18T01:13:50.0521216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1124, in forward 2024-12-18T01:13:50.0521962Z distilbert_output = self.distilbert( 2024-12-18T01:13:50.0522685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-12-18T01:13:50.0523545Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-12-18T01:13:50.0524410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-12-18T01:13:50.0525245Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-12-18T01:13:50.0525582Z 2024-12-18T01:13:50.1746872Z W1218 01:13:50.173000 14955 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:13:51.0789211Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:14:10.9611334Z pass 2024-12-18T01:14:11.0007565Z TIMING: entire_frame_compile:31.49922 _recursive_pre_grad_passes:0.03437 _recursive_joint_graph_passes:1.8902 _recursive_post_grad_passes:0.71574 async_compile.wait:1.46838 code_gen:9.15923 inductor_compile:17.22156 backend_compile:24.55141 entire_backward_compile:4.03823 total_wall_time:35.53745 2024-12-18T01:14:11.0009576Z STATS: call_* op count: 745 | FakeTensorMode.__torch_dispatch__:32891 | FakeTensor.__torch_dispatch__:7939 | ProxyTorchDispatchMode.__torch_dispatch__:14854 2024-12-18T01:14:11.0010441Z Dynamo produced 2 graphs covering 745 ops with 5 graph breaks (4 unique) 2024-12-18T01:14:16.8098756Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:14:16.8100321Z warnings.warn( 2024-12-18T01:14:17.2025708Z 2024-12-18T01:14:19.2562598Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:14:19.2562956Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:14:19.2563287Z cuda train DistillGPT2 2024-12-18T01:14:37.9191705Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:14:37.9192593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:14:37.9193325Z pred = mod(**cloned_inputs) 2024-12-18T01:14:37.9193979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-12-18T01:14:37.9194662Z transformer_outputs = self.transformer( 2024-12-18T01:14:37.9195334Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-12-18T01:14:37.9195991Z inputs_embeds = self.wte(input_ids) 2024-12-18T01:14:37.9196214Z 2024-12-18T01:14:38.0326900Z W1218 01:14:38.031000 15244 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:14:38.7053718Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:14:55.8308128Z pass 2024-12-18T01:14:55.8667841Z TIMING: entire_frame_compile:27.8634 _recursive_pre_grad_passes:0.03239 _recursive_joint_graph_passes:1.99223 _recursive_post_grad_passes:0.60009 async_compile.wait:3.98362 code_gen:10.82917 inductor_compile:17.30664 backend_compile:22.06393 entire_backward_compile:4.52117 total_wall_time:32.38457 2024-12-18T01:14:55.8669559Z STATS: call_* op count: 725 | FakeTensorMode.__torch_dispatch__:27066 | FakeTensor.__torch_dispatch__:6614 | ProxyTorchDispatchMode.__torch_dispatch__:12087 2024-12-18T01:14:55.8670408Z Dynamo produced 2 graphs covering 725 ops with 5 graph breaks (4 unique) 2024-12-18T01:15:01.4528877Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:15:01.4530108Z warnings.warn( 2024-12-18T01:15:01.6960336Z 2024-12-18T01:15:01.6969661Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2024-12-18T01:15:02.5005257Z 2024-12-18T01:15:02.5005837Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:15:02.5006226Z cuda train ElectraForCausalLM 2024-12-18T01:15:35.1512282Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:15:35.1514978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:15:35.1515727Z pred = mod(**cloned_inputs) 2024-12-18T01:15:35.1516417Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1617, in forward 2024-12-18T01:15:35.1517115Z outputs = self.electra( 2024-12-18T01:15:35.1517787Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-12-18T01:15:35.1518497Z hidden_states = self.embeddings( 2024-12-18T01:15:35.1519180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-12-18T01:15:35.1519908Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:15:35.1520175Z 2024-12-18T01:15:35.9418212Z pass 2024-12-18T01:15:35.9609041Z TIMING: entire_frame_compile:22.15699 _recursive_pre_grad_passes:0.01387 _recursive_joint_graph_passes:2.38833 _recursive_post_grad_passes:0.65203 async_compile.wait:0.60332 code_gen:7.37944 inductor_compile:14.47568 backend_compile:16.03142 entire_backward_compile:8.27135 total_wall_time:30.42834 2024-12-18T01:15:35.9610656Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:38741 | FakeTensor.__torch_dispatch__:6716 | ProxyTorchDispatchMode.__torch_dispatch__:17991 2024-12-18T01:15:35.9611497Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique) 2024-12-18T01:15:41.6018645Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:15:41.6019878Z warnings.warn( 2024-12-18T01:15:42.1028706Z 2024-12-18T01:15:42.8828410Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:15:42.8829097Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:15:42.8829580Z cuda train ElectraForQuestionAnswering 2024-12-18T01:16:15.1150618Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:16:15.1153669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:16:15.1154438Z pred = mod(**cloned_inputs) 2024-12-18T01:16:15.1155136Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 1385, in forward 2024-12-18T01:16:15.1155902Z discriminator_hidden_states = self.electra( 2024-12-18T01:16:15.1156624Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 902, in forward 2024-12-18T01:16:15.1157332Z hidden_states = self.embeddings( 2024-12-18T01:16:15.1158013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/electra/modeling_electra.py", line 203, in forward 2024-12-18T01:16:15.1158784Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:16:15.1159040Z 2024-12-18T01:16:15.2666750Z W1218 01:16:15.265000 15905 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:16:17.0041532Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:16:58.3485460Z pass 2024-12-18T01:16:58.3637887Z TIMING: entire_frame_compile:62.71558 _recursive_pre_grad_passes:0.06112 _recursive_joint_graph_passes:2.55932 _recursive_post_grad_passes:0.82888 async_compile.wait:4.78228 code_gen:20.42731 inductor_compile:35.61107 backend_compile:48.71602 entire_backward_compile:8.01023 total_wall_time:70.7258 2024-12-18T01:16:58.3641213Z STATS: call_* op count: 1404 | FakeTensorMode.__torch_dispatch__:62445 | FakeTensor.__torch_dispatch__:15564 | ProxyTorchDispatchMode.__torch_dispatch__:28571 2024-12-18T01:16:58.3642105Z Dynamo produced 2 graphs covering 1404 ops with 5 graph breaks (4 unique) 2024-12-18T01:17:05.5847475Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:17:05.5848706Z warnings.warn( 2024-12-18T01:17:05.9154944Z 2024-12-18T01:17:08.2658671Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:17:08.2659337Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:17:08.2660022Z cuda train GPT2ForSequenceClassification 2024-12-18T01:17:43.1373273Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:17:43.1374879Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-12-18T01:17:43.1376090Z transformer_outputs = self.transformer( 2024-12-18T01:17:43.1376770Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-12-18T01:17:43.1377445Z inputs_embeds = self.wte(input_ids) 2024-12-18T01:17:43.1377664Z 2024-12-18T01:17:43.2263351Z W1218 01:17:43.225000 16263 site-packages/torch/_logging/_internal.py:1089] [7/0] Profiler function will be ignored 2024-12-18T01:17:44.4894791Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:18:37.9476779Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:18:37.9480920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1426, in forward 2024-12-18T01:18:37.9481773Z transformer_outputs = self.transformer( 2024-12-18T01:18:37.9482460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-12-18T01:18:37.9483140Z inputs_embeds = self.wte(input_ids) 2024-12-18T01:18:37.9483366Z 2024-12-18T01:18:38.2508994Z pass 2024-12-18T01:18:38.3399565Z TIMING: entire_frame_compile:68.51592 _recursive_pre_grad_passes:0.0733 _recursive_joint_graph_passes:2.75505 inductor_compile:42.92767 backend_compile:52.87776 _recursive_post_grad_passes:1.27122 async_compile.wait:5.21649 code_gen:25.62488 entire_backward_compile:15.74316 total_wall_time:84.25908 2024-12-18T01:18:38.3401720Z STATS: call_* op count: 2034 | FakeTensorMode.__torch_dispatch__:85650 | FakeTensor.__torch_dispatch__:18729 | ProxyTorchDispatchMode.__torch_dispatch__:39488 2024-12-18T01:18:38.3402572Z Dynamo produced 6 graphs covering 2034 ops with 7 graph breaks (5 unique) 2024-12-18T01:18:46.0398658Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:18:46.0400367Z warnings.warn( 2024-12-18T01:18:46.3377347Z 2024-12-18T01:18:47.7369357Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only 2024-12-18T01:18:48.0007317Z 2024-12-18T01:18:48.0007902Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:18:48.0008313Z WARNING:common:Model GoogleFnet supports float32 only 2024-12-18T01:18:48.0018430Z cuda train GoogleFnet 2024-12-18T01:18:49.3665457Z WARNING:common:Model GoogleFnet supports float32 only 2024-12-18T01:19:06.5261400Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:19:06.5262318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:19:06.5263077Z pred = mod(**cloned_inputs) 2024-12-18T01:19:06.5263736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 749, in forward 2024-12-18T01:19:06.5264414Z outputs = self.fnet( 2024-12-18T01:19:06.5265030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 581, in forward 2024-12-18T01:19:06.5265702Z embedding_output = self.embeddings( 2024-12-18T01:19:06.5266358Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/fnet/modeling_fnet.py", line 148, in forward 2024-12-18T01:19:06.5267047Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:19:06.5267311Z 2024-12-18T01:19:06.6573044Z W1218 01:19:06.656000 16725 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:19:07.5591035Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:19:30.0325549Z pass 2024-12-18T01:19:30.0766340Z TIMING: entire_frame_compile:32.83876 _recursive_pre_grad_passes:0.03602 _recursive_joint_graph_passes:0.52865 _recursive_post_grad_passes:0.37086 async_compile.wait:4.0359 code_gen:12.64766 inductor_compile:20.69903 backend_compile:25.73043 entire_backward_compile:4.9373 total_wall_time:37.77607 2024-12-18T01:19:30.0767975Z STATS: call_* op count: 791 | FakeTensorMode.__torch_dispatch__:28270 | FakeTensor.__torch_dispatch__:8126 | ProxyTorchDispatchMode.__torch_dispatch__:13155 2024-12-18T01:19:30.0768825Z Dynamo produced 2 graphs covering 791 ops with 5 graph breaks (4 unique) 2024-12-18T01:19:35.9101954Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:19:35.9103338Z warnings.warn( 2024-12-18T01:19:36.3516644Z 2024-12-18T01:19:38.5042717Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:19:38.5043082Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:19:38.5043415Z cuda train LayoutLMForMaskedLM 2024-12-18T01:20:14.5974821Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:20:14.5977432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:20:14.5978182Z pred = mod(**cloned_inputs) 2024-12-18T01:20:14.5978889Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 938, in forward 2024-12-18T01:20:14.5979582Z outputs = self.layoutlm( 2024-12-18T01:20:14.5980280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-12-18T01:20:14.5980995Z embedding_output = self.embeddings( 2024-12-18T01:20:14.5981687Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-12-18T01:20:14.5982411Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:20:14.5982664Z 2024-12-18T01:20:14.7571558Z W1218 01:20:14.756000 17033 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:20:16.5434287Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:20:58.8586827Z pass 2024-12-18T01:20:58.9249970Z TIMING: entire_frame_compile:65.14257 _recursive_pre_grad_passes:0.06915 _recursive_joint_graph_passes:1.93644 _recursive_post_grad_passes:0.83499 async_compile.wait:5.83695 code_gen:21.98003 inductor_compile:37.48897 backend_compile:49.83278 entire_backward_compile:8.69694 total_wall_time:73.83951 2024-12-18T01:20:58.9251999Z STATS: call_* op count: 1447 | FakeTensorMode.__torch_dispatch__:63827 | FakeTensor.__torch_dispatch__:15848 | ProxyTorchDispatchMode.__torch_dispatch__:29173 2024-12-18T01:20:58.9253085Z Dynamo produced 2 graphs covering 1447 ops with 5 graph breaks (4 unique) 2024-12-18T01:21:06.1833466Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:21:06.1834951Z warnings.warn( 2024-12-18T01:21:06.5243507Z 2024-12-18T01:21:08.4202438Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:21:08.4202992Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:21:08.4203514Z cuda train LayoutLMForSequenceClassification 2024-12-18T01:21:47.2437214Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:21:47.2440035Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-12-18T01:21:47.2440772Z outputs = self.layoutlm( 2024-12-18T01:21:47.2441465Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-12-18T01:21:47.2442166Z embedding_output = self.embeddings( 2024-12-18T01:21:47.2442862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-12-18T01:21:47.2443588Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:21:47.2443848Z 2024-12-18T01:21:47.4128570Z W1218 01:21:47.411000 17353 site-packages/torch/_logging/_internal.py:1089] [7/0] Profiler function will be ignored 2024-12-18T01:21:49.1998288Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:22:55.5430905Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:22:55.5432660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 1060, in forward 2024-12-18T01:22:55.5433717Z outputs = self.layoutlm( 2024-12-18T01:22:55.5434692Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 821, in forward 2024-12-18T01:22:55.5435985Z embedding_output = self.embeddings( 2024-12-18T01:22:55.5437011Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 99, in forward 2024-12-18T01:22:55.5438054Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:22:55.5438423Z 2024-12-18T01:22:55.9157964Z pass 2024-12-18T01:22:56.0056364Z TIMING: entire_frame_compile:86.85789 _recursive_pre_grad_passes:0.08077 _recursive_joint_graph_passes:3.61757 inductor_compile:48.84791 backend_compile:63.75917 _recursive_post_grad_passes:1.42994 async_compile.wait:4.13698 code_gen:26.53266 entire_backward_compile:16.27113 total_wall_time:103.12903 2024-12-18T01:22:56.0058088Z STATS: call_* op count: 1836 | FakeTensorMode.__torch_dispatch__:102401 | FakeTensor.__torch_dispatch__:21905 | ProxyTorchDispatchMode.__torch_dispatch__:47947 2024-12-18T01:22:56.0058966Z Dynamo produced 6 graphs covering 1836 ops with 7 graph breaks (5 unique) 2024-12-18T01:23:04.6025521Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:23:04.6029357Z warnings.warn( 2024-12-18T01:23:04.9163043Z 2024-12-18T01:23:15.3990481Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:23:15.3991165Z loading model: 0it [00:10, ?it/s] 2024-12-18T01:23:15.3991858Z cuda train M2M100ForConditionalGeneration 2024-12-18T01:23:15.5010720Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine 2024-12-18T01:24:27.2858824Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:24:27.2860573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:24:27.2861947Z pred = mod(**cloned_inputs) 2024-12-18T01:24:27.2862608Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1275, in forward 2024-12-18T01:24:27.2863271Z outputs = self.model( 2024-12-18T01:24:27.2863897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1162, in forward 2024-12-18T01:24:27.2864563Z encoder_outputs = self.encoder( 2024-12-18T01:24:27.2865216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 763, in forward 2024-12-18T01:24:27.2865955Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-12-18T01:24:27.2866264Z 2024-12-18T01:24:28.9888084Z pass 2024-12-18T01:24:29.0869774Z TIMING: entire_frame_compile:51.21116 _recursive_pre_grad_passes:0.03837 _recursive_joint_graph_passes:2.90383 _recursive_post_grad_passes:3.54061 async_compile.wait:1.12415 code_gen:16.5187 inductor_compile:34.38886 backend_compile:38.69941 entire_backward_compile:17.48781 total_wall_time:68.69897 2024-12-18T01:24:29.0871435Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:99986 | FakeTensor.__torch_dispatch__:14339 | ProxyTorchDispatchMode.__torch_dispatch__:47008 2024-12-18T01:24:29.0872291Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique) 2024-12-18T01:24:36.5027262Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:24:36.5028592Z warnings.warn( 2024-12-18T01:24:36.7429957Z 2024-12-18T01:24:40.7116359Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:24:40.7116747Z loading model: 0it [00:03, ?it/s] 2024-12-18T01:24:40.7117078Z cuda train MBartForCausalLM 2024-12-18T01:24:40.7420382Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine 2024-12-18T01:24:44.1950837Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:24:44.1951774Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-12-18T01:24:44.1952487Z return super().forward(positions + self.offset) 2024-12-18T01:24:44.1952747Z 2024-12-18T01:24:53.8870520Z pass 2024-12-18T01:24:53.9502246Z TIMING: entire_frame_compile:3.78162 _recursive_pre_grad_passes:0.00656 _recursive_joint_graph_passes:0.21388 inductor_compile:2.08761 backend_compile:2.87266 _recursive_post_grad_passes:0.09964 async_compile.precompile:0.11065 async_compile.wait:0.5622 code_gen:1.31243 entire_backward_compile:1.0571 total_wall_time:4.83872 2024-12-18T01:24:53.9503975Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3894 | FakeTensor.__torch_dispatch__:651 | ProxyTorchDispatchMode.__torch_dispatch__:1583 2024-12-18T01:24:53.9504816Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-12-18T01:24:58.6224650Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:24:58.6225885Z warnings.warn( 2024-12-18T01:24:58.8815877Z 2024-12-18T01:25:05.8522199Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:25:05.8522676Z loading model: 0it [00:06, ?it/s] 2024-12-18T01:25:05.8523140Z cuda train MBartForConditionalGeneration 2024-12-18T01:25:05.9475978Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine 2024-12-18T01:25:11.4429386Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:25:11.4430277Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 135, in forward 2024-12-18T01:25:11.4430983Z return super().forward(positions + self.offset) 2024-12-18T01:25:11.4431238Z 2024-12-18T01:25:35.1095810Z pass 2024-12-18T01:25:35.2137186Z TIMING: entire_frame_compile:7.01226 _recursive_pre_grad_passes:0.00907 _recursive_joint_graph_passes:0.38549 inductor_compile:4.70789 backend_compile:5.32879 _recursive_post_grad_passes:0.28923 async_compile.precompile:0.11833 async_compile.wait:0.80264 code_gen:2.8073 entire_backward_compile:2.42139 total_wall_time:9.43365 2024-12-18T01:25:35.2140671Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:9575 | FakeTensor.__torch_dispatch__:1674 | ProxyTorchDispatchMode.__torch_dispatch__:4358 2024-12-18T01:25:35.2141614Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique) 2024-12-18T01:25:40.0994028Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:25:40.0995287Z warnings.warn( 2024-12-18T01:25:40.6823012Z 2024-12-18T01:25:44.7619632Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:25:44.7619992Z loading model: 0it [00:04, ?it/s] 2024-12-18T01:25:44.7620339Z cuda train MT5ForConditionalGeneration 2024-12-18T01:26:37.9282115Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:26:37.9283010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:26:37.9283741Z pred = mod(**cloned_inputs) 2024-12-18T01:26:37.9284904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 1722, in forward 2024-12-18T01:26:37.9285581Z encoder_outputs = self.encoder( 2024-12-18T01:26:37.9286414Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mt5/modeling_mt5.py", line 989, in forward 2024-12-18T01:26:37.9287094Z inputs_embeds = self.embed_tokens(input_ids) 2024-12-18T01:26:37.9287338Z 2024-12-18T01:26:38.0779606Z W1218 01:26:38.076000 18447 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:26:40.0738559Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:27:22.4950614Z pass 2024-12-18T01:27:22.6407639Z TIMING: entire_frame_compile:77.38032 _recursive_pre_grad_passes:0.0765 _recursive_joint_graph_passes:2.29258 _recursive_post_grad_passes:1.31103 async_compile.wait:4.52297 code_gen:22.74292 inductor_compile:42.92433 backend_compile:57.4261 entire_backward_compile:12.88554 total_wall_time:90.26586 2024-12-18T01:27:22.6409517Z STATS: call_* op count: 2145 | FakeTensorMode.__torch_dispatch__:97601 | ProxyTorchDispatchMode.__torch_dispatch__:46228 | FakeTensor.__torch_dispatch__:19942 2024-12-18T01:27:22.6410405Z Dynamo produced 2 graphs covering 2145 ops with 5 graph breaks (4 unique) 2024-12-18T01:27:30.4907076Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:27:30.4908410Z warnings.warn( 2024-12-18T01:27:30.7873245Z 2024-12-18T01:27:30.7912399Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2024-12-18T01:27:35.6401987Z 2024-12-18T01:27:35.6402532Z loading model: 0it [00:04, ?it/s] 2024-12-18T01:27:35.6402936Z cuda train MegatronBertForCausalLM 2024-12-18T01:28:40.4296538Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:28:40.4299027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:28:40.4299791Z pred = mod(**cloned_inputs) 2024-12-18T01:28:40.4300619Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1199, in forward 2024-12-18T01:28:40.4301358Z outputs = self.bert( 2024-12-18T01:28:40.4302053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-12-18T01:28:40.4302790Z embedding_output = self.embeddings( 2024-12-18T01:28:40.4304036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-12-18T01:28:40.4304819Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:28:40.4305316Z 2024-12-18T01:28:40.6442145Z W1218 01:28:40.643000 18797 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:28:44.0430899Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][314].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][316].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad", "L['self'].param_groups[0]['params'][329].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][331].grad", "L['self'].param_groups[0]['params'][332].grad", "L['self'].param_groups[0]['params'][333].grad", "L['self'].param_groups[0]['params'][334].grad", "L['self'].param_groups[0]['params'][335].grad", "L['self'].param_groups[0]['params'][336].grad", "L['self'].param_groups[0]['params'][337].grad", "L['self'].param_groups[0]['params'][338].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][340].grad", "L['self'].param_groups[0]['params'][341].grad", "L['self'].param_groups[0]['params'][342].grad", "L['self'].param_groups[0]['params'][343].grad", "L['self'].param_groups[0]['params'][344].grad", "L['self'].param_groups[0]['params'][345].grad", "L['self'].param_groups[0]['params'][346].grad", "L['self'].param_groups[0]['params'][347].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][349].grad", "L['self'].param_groups[0]['params'][350].grad", "L['self'].param_groups[0]['params'][351].grad", "L['self'].param_groups[0]['params'][352].grad", "L['self'].param_groups[0]['params'][353].grad", "L['self'].param_groups[0]['params'][354].grad", "L['self'].param_groups[0]['params'][355].grad", "L['self'].param_groups[0]['params'][356].grad", "L['self'].param_groups[0]['params'][357].grad", "L['self'].param_groups[0]['params'][358].grad", "L['self'].param_groups[0]['params'][359].grad", "L['self'].param_groups[0]['params'][360].grad", "L['self'].param_groups[0]['params'][361].grad", "L['self'].param_groups[0]['params'][362].grad", "L['self'].param_groups[0]['params'][363].grad", "L['self'].param_groups[0]['params'][364].grad", "L['self'].param_groups[0]['params'][365].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][367].grad", "L['self'].param_groups[0]['params'][368].grad", "L['self'].param_groups[0]['params'][369].grad", "L['self'].param_groups[0]['params'][370].grad", "L['self'].param_groups[0]['params'][371].grad", "L['self'].param_groups[0]['params'][372].grad", "L['self'].param_groups[0]['params'][373].grad", "L['self'].param_groups[0]['params'][374].grad", "L['self'].param_groups[0]['params'][375].grad", "L['self'].param_groups[0]['params'][376].grad", "L['self'].param_groups[0]['params'][377].grad", "L['self'].param_groups[0]['params'][378].grad", "L['self'].param_groups[0]['params'][379].grad", "L['self'].param_groups[0]['params'][380].grad", "L['self'].param_groups[0]['params'][381].grad", "L['self'].param_groups[0]['params'][382].grad", "L['self'].param_groups[0]['params'][383].grad", "L['self'].param_groups[0]['params'][384].grad", "L['self'].param_groups[0]['params'][385].grad", "L['self'].param_groups[0]['params'][386].grad", "L['self'].param_groups[0]['params'][387].grad", "L['self'].param_groups[0]['params'][388].grad", "L['self'].param_groups[0]['params'][389].grad", "L['self'].param_groups[0]['params'][390].grad", "L['self'].param_groups[0]['params'][391].grad", "L['self'].param_groups[0]['params'][392].grad", "L['self'].param_groups[0]['params'][393].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:30:06.6965177Z pass 2024-12-18T01:30:06.8979851Z TIMING: entire_frame_compile:121.68613 _recursive_pre_grad_passes:0.11498 _recursive_joint_graph_passes:2.77167 _recursive_post_grad_passes:1.62534 async_compile.wait:7.09336 code_gen:40.35999 inductor_compile:69.28207 backend_compile:93.56336 entire_backward_compile:15.06573 total_wall_time:136.75186 2024-12-18T01:30:06.8981531Z STATS: call_* op count: 2712 | FakeTensorMode.__torch_dispatch__:121284 | FakeTensor.__torch_dispatch__:30288 | ProxyTorchDispatchMode.__torch_dispatch__:55799 2024-12-18T01:30:06.8982388Z Dynamo produced 2 graphs covering 2712 ops with 5 graph breaks (4 unique) 2024-12-18T01:30:16.9444495Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:30:16.9445919Z warnings.warn( 2024-12-18T01:30:17.2165415Z 2024-12-18T01:30:21.6244376Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:30:21.6244736Z loading model: 0it [00:04, ?it/s] 2024-12-18T01:30:21.6245090Z cuda train MegatronBertForQuestionAnswering 2024-12-18T01:31:25.9956778Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:31:25.9957801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:31:25.9958554Z pred = mod(**cloned_inputs) 2024-12-18T01:31:25.9959316Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1792, in forward 2024-12-18T01:31:25.9960050Z outputs = self.bert( 2024-12-18T01:31:25.9960749Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 967, in forward 2024-12-18T01:31:25.9961486Z embedding_output = self.embeddings( 2024-12-18T01:31:25.9962219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 179, in forward 2024-12-18T01:31:25.9962989Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:31:25.9963252Z 2024-12-18T01:31:26.2148585Z W1218 01:31:26.213000 19224 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:31:29.6208565Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][314].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][316].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad", "L['self'].param_groups[0]['params'][329].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][331].grad", "L['self'].param_groups[0]['params'][332].grad", "L['self'].param_groups[0]['params'][333].grad", "L['self'].param_groups[0]['params'][334].grad", "L['self'].param_groups[0]['params'][335].grad", "L['self'].param_groups[0]['params'][336].grad", "L['self'].param_groups[0]['params'][337].grad", "L['self'].param_groups[0]['params'][338].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][340].grad", "L['self'].param_groups[0]['params'][341].grad", "L['self'].param_groups[0]['params'][342].grad", "L['self'].param_groups[0]['params'][343].grad", "L['self'].param_groups[0]['params'][344].grad", "L['self'].param_groups[0]['params'][345].grad", "L['self'].param_groups[0]['params'][346].grad", "L['self'].param_groups[0]['params'][347].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][349].grad", "L['self'].param_groups[0]['params'][350].grad", "L['self'].param_groups[0]['params'][351].grad", "L['self'].param_groups[0]['params'][352].grad", "L['self'].param_groups[0]['params'][353].grad", "L['self'].param_groups[0]['params'][354].grad", "L['self'].param_groups[0]['params'][355].grad", "L['self'].param_groups[0]['params'][356].grad", "L['self'].param_groups[0]['params'][357].grad", "L['self'].param_groups[0]['params'][358].grad", "L['self'].param_groups[0]['params'][359].grad", "L['self'].param_groups[0]['params'][360].grad", "L['self'].param_groups[0]['params'][361].grad", "L['self'].param_groups[0]['params'][362].grad", "L['self'].param_groups[0]['params'][363].grad", "L['self'].param_groups[0]['params'][364].grad", "L['self'].param_groups[0]['params'][365].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][367].grad", "L['self'].param_groups[0]['params'][368].grad", "L['self'].param_groups[0]['params'][369].grad", "L['self'].param_groups[0]['params'][370].grad", "L['self'].param_groups[0]['params'][371].grad", "L['self'].param_groups[0]['params'][372].grad", "L['self'].param_groups[0]['params'][373].grad", "L['self'].param_groups[0]['params'][374].grad", "L['self'].param_groups[0]['params'][375].grad", "L['self'].param_groups[0]['params'][376].grad", "L['self'].param_groups[0]['params'][377].grad", "L['self'].param_groups[0]['params'][378].grad", "L['self'].param_groups[0]['params'][379].grad", "L['self'].param_groups[0]['params'][380].grad", "L['self'].param_groups[0]['params'][381].grad", "L['self'].param_groups[0]['params'][382].grad", "L['self'].param_groups[0]['params'][383].grad", "L['self'].param_groups[0]['params'][384].grad", "L['self'].param_groups[0]['params'][385].grad", "L['self'].param_groups[0]['params'][386].grad", "L['self'].param_groups[0]['params'][387].grad", "L['self'].param_groups[0]['params'][388].grad", "L['self'].param_groups[0]['params'][389].grad", "L['self'].param_groups[0]['params'][390].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:32:46.6048283Z pass 2024-12-18T01:32:46.8102375Z TIMING: entire_frame_compile:116.31403 _recursive_pre_grad_passes:0.11389 _recursive_joint_graph_passes:4.18285 _recursive_post_grad_passes:1.62508 async_compile.wait:2.57382 code_gen:34.59881 inductor_compile:62.86424 backend_compile:88.32571 entire_backward_compile:14.74731 total_wall_time:131.06134 2024-12-18T01:32:46.8104086Z STATS: call_* op count: 2700 | FakeTensorMode.__torch_dispatch__:120674 | FakeTensor.__torch_dispatch__:30110 | ProxyTorchDispatchMode.__torch_dispatch__:55544 2024-12-18T01:32:46.8105117Z Dynamo produced 2 graphs covering 2700 ops with 5 graph breaks (4 unique) 2024-12-18T01:32:56.8387205Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:32:56.8388506Z warnings.warn( 2024-12-18T01:32:57.2313612Z 2024-12-18T01:32:58.6699010Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:32:58.6699370Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:32:58.6699717Z cuda train MobileBertForMaskedLM 2024-12-18T01:34:56.0853384Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:34:56.0854704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:34:56.0855822Z pred = mod(**cloned_inputs) 2024-12-18T01:34:56.0856880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1089, in forward 2024-12-18T01:34:56.0857919Z outputs = self.mobilebert( 2024-12-18T01:34:56.0858617Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-12-18T01:34:56.0859347Z embedding_output = self.embeddings( 2024-12-18T01:34:56.0860061Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-12-18T01:34:56.0860808Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:34:56.0861074Z 2024-12-18T01:34:58.5806927Z pass 2024-12-18T01:34:58.6288497Z TIMING: entire_frame_compile:86.65155 _recursive_pre_grad_passes:0.04799 _recursive_joint_graph_passes:5.7779 _recursive_post_grad_passes:3.74221 async_compile.wait:0.86393 code_gen:22.88153 inductor_compile:47.90083 backend_compile:57.65113 entire_backward_compile:26.52473 total_wall_time:113.17627 2024-12-18T01:34:58.6290247Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:154134 | FakeTensor.__torch_dispatch__:22285 | ProxyTorchDispatchMode.__torch_dispatch__:74442 2024-12-18T01:34:58.6291113Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique) 2024-12-18T01:35:07.7367893Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:35:07.7369125Z warnings.warn( 2024-12-18T01:35:08.0340549Z 2024-12-18T01:35:09.3211414Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:35:09.3212794Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:35:09.3213161Z cuda train MobileBertForQuestionAnswering 2024-12-18T01:37:05.1442235Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:37:05.1443311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:37:05.1444039Z pred = mod(**cloned_inputs) 2024-12-18T01:37:05.1444843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1390, in forward 2024-12-18T01:37:05.1445817Z outputs = self.mobilebert( 2024-12-18T01:37:05.1446584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 895, in forward 2024-12-18T01:37:05.1447308Z embedding_output = self.embeddings( 2024-12-18T01:37:05.1448054Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 218, in forward 2024-12-18T01:37:05.1448803Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:37:05.1449068Z 2024-12-18T01:37:07.6251271Z pass 2024-12-18T01:37:07.6587134Z TIMING: entire_frame_compile:85.5494 _recursive_pre_grad_passes:0.04803 _recursive_joint_graph_passes:5.72868 _recursive_post_grad_passes:4.58833 async_compile.wait:0.31295 code_gen:21.97835 inductor_compile:46.80963 backend_compile:56.76981 entire_backward_compile:26.10513 total_wall_time:111.65453 2024-12-18T01:37:07.6588864Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:153943 | ProxyTorchDispatchMode.__torch_dispatch__:74369 | FakeTensor.__torch_dispatch__:22266 2024-12-18T01:37:07.6589751Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique) 2024-12-18T01:37:16.6389512Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:37:16.6390754Z warnings.warn( 2024-12-18T01:37:17.0377321Z 2024-12-18T01:37:19.5688809Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:37:19.5689160Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:37:19.5689490Z cuda train OPTForCausalLM 2024-12-18T01:37:19.5906249Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine 2024-12-18T01:37:28.5726329Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:37:28.5727435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 109, in forward 2024-12-18T01:37:28.5728359Z return super().forward(positions + self.offset) 2024-12-18T01:37:28.5737668Z 2024-12-18T01:37:34.8684475Z pass 2024-12-18T01:37:34.8922114Z TIMING: entire_frame_compile:4.73925 _recursive_pre_grad_passes:0.00818 _recursive_joint_graph_passes:0.23684 inductor_compile:3.42838 backend_compile:3.64524 _recursive_post_grad_passes:0.1293 async_compile.precompile:0.09164 async_compile.wait:1.17221 code_gen:2.42107 entire_backward_compile:1.68443 total_wall_time:6.42368 2024-12-18T01:37:34.8923881Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:4851 | ProxyTorchDispatchMode.__torch_dispatch__:2031 | FakeTensor.__torch_dispatch__:809 2024-12-18T01:37:34.8924721Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique) 2024-12-18T01:37:39.6507645Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:37:39.6508924Z warnings.warn( 2024-12-18T01:37:39.9383780Z 2024-12-18T01:37:41.8997555Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:37:41.8997930Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:37:41.8999512Z cuda train PLBartForCausalLM 2024-12-18T01:37:41.9173955Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine 2024-12-18T01:37:50.3810903Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:37:50.3811761Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-12-18T01:37:50.3812487Z return super().forward(positions + self.offset) 2024-12-18T01:37:50.3812752Z 2024-12-18T01:37:53.8067416Z pass 2024-12-18T01:37:53.8380047Z TIMING: entire_frame_compile:5.06333 _recursive_pre_grad_passes:0.00735 _recursive_joint_graph_passes:1.10449 inductor_compile:3.23791 backend_compile:4.15425 async_compile.precompile:0.09324 async_compile.wait:0.9398 _recursive_post_grad_passes:0.12058 code_gen:2.28463 entire_backward_compile:1.78499 total_wall_time:6.84832 2024-12-18T01:37:53.8381895Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:4428 | FakeTensor.__torch_dispatch__:790 | ProxyTorchDispatchMode.__torch_dispatch__:1868 2024-12-18T01:37:53.8382741Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-12-18T01:37:58.5162202Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:37:58.5163424Z warnings.warn( 2024-12-18T01:37:58.8466857Z 2024-12-18T01:38:02.2399745Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:38:02.2400227Z loading model: 0it [00:03, ?it/s] 2024-12-18T01:38:02.2400582Z cuda train PLBartForConditionalGeneration 2024-12-18T01:38:02.3020200Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine 2024-12-18T01:38:05.8910477Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:38:05.8911331Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 106, in forward 2024-12-18T01:38:05.8912065Z return super().forward(positions + self.offset) 2024-12-18T01:38:05.8912332Z 2024-12-18T01:38:22.4752906Z pass 2024-12-18T01:38:22.5128137Z TIMING: entire_frame_compile:7.36381 _recursive_pre_grad_passes:0.00921 _recursive_joint_graph_passes:0.38985 inductor_compile:4.9814 backend_compile:5.55427 async_compile.precompile:0.03329 async_compile.wait:1.12217 _recursive_post_grad_passes:0.30278 code_gen:3.03263 entire_backward_compile:2.64448 total_wall_time:10.00829 2024-12-18T01:38:22.5131538Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:9549 | FakeTensor.__torch_dispatch__:1733 | ProxyTorchDispatchMode.__torch_dispatch__:4348 2024-12-18T01:38:22.5133167Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique) 2024-12-18T01:38:27.3987777Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:38:27.3989074Z warnings.warn( 2024-12-18T01:38:27.8036632Z 2024-12-18T01:38:34.1682029Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:38:34.1682386Z loading model: 0it [00:06, ?it/s] 2024-12-18T01:38:34.1682713Z cuda train PegasusForCausalLM 2024-12-18T01:38:34.1980033Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine 2024-12-18T01:38:48.3810889Z pass 2024-12-18T01:38:48.4589121Z TIMING: entire_frame_compile:4.78492 _recursive_pre_grad_passes:0.00656 _recursive_joint_graph_passes:0.91144 inductor_compile:2.75549 backend_compile:3.89639 async_compile.precompile:0.09101 async_compile.wait:0.76079 _recursive_post_grad_passes:0.11353 code_gen:1.82654 entire_backward_compile:1.33293 total_wall_time:6.11784 2024-12-18T01:38:48.4590881Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:4276 | ProxyTorchDispatchMode.__torch_dispatch__:1815 | FakeTensor.__torch_dispatch__:780 2024-12-18T01:38:48.4591918Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique) 2024-12-18T01:38:53.1714826Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:38:53.1716027Z warnings.warn( 2024-12-18T01:38:53.4219443Z 2024-12-18T01:39:03.6867739Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:39:03.6868270Z loading model: 0it [00:10, ?it/s] 2024-12-18T01:39:03.6868632Z cuda train PegasusForConditionalGeneration 2024-12-18T01:39:03.7758926Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine 2024-12-18T01:39:30.8839034Z pass 2024-12-18T01:39:31.0134407Z TIMING: entire_frame_compile:6.83141 _recursive_pre_grad_passes:0.00875 _recursive_joint_graph_passes:0.38376 inductor_compile:4.66959 backend_compile:5.22631 async_compile.precompile:0.00983 async_compile.wait:0.87254 _recursive_post_grad_passes:0.28163 code_gen:2.58431 entire_backward_compile:2.35191 total_wall_time:9.18332 2024-12-18T01:39:31.0136172Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:9202 | ProxyTorchDispatchMode.__torch_dispatch__:4249 | FakeTensor.__torch_dispatch__:1698 2024-12-18T01:39:31.0137021Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique) 2024-12-18T01:39:35.8867137Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:39:35.8868486Z warnings.warn( 2024-12-18T01:39:36.2268615Z 2024-12-18T01:39:36.2297899Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2024-12-18T01:39:38.2376667Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-12-18T01:39:38.2378338Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-12-18T01:39:38.5986322Z 2024-12-18T01:39:38.5986714Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:39:38.5987103Z cuda train RobertaForCausalLM 2024-12-18T01:40:12.4329262Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:40:12.4330171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:40:12.4330900Z pred = mod(**cloned_inputs) 2024-12-18T01:40:12.4331591Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 953, in forward 2024-12-18T01:40:12.4332276Z outputs = self.roberta( 2024-12-18T01:40:12.4332929Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-12-18T01:40:12.4333619Z embedding_output = self.embeddings( 2024-12-18T01:40:12.4334305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-12-18T01:40:12.4335014Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:40:12.4335276Z 2024-12-18T01:40:12.5920795Z W1218 01:40:12.591000 21173 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:40:14.3210222Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:40:54.2052825Z pass 2024-12-18T01:40:54.2601720Z TIMING: entire_frame_compile:60.62822 _recursive_pre_grad_passes:0.0622 _recursive_joint_graph_passes:1.93575 _recursive_post_grad_passes:1.13607 async_compile.wait:3.1338 code_gen:19.03751 inductor_compile:34.27771 backend_compile:46.6458 entire_backward_compile:8.14981 total_wall_time:68.77803 2024-12-18T01:40:54.2603385Z STATS: call_* op count: 1413 | FakeTensorMode.__torch_dispatch__:62605 | FakeTensor.__torch_dispatch__:15612 | ProxyTorchDispatchMode.__torch_dispatch__:28657 2024-12-18T01:40:54.2604242Z Dynamo produced 2 graphs covering 1413 ops with 5 graph breaks (4 unique) 2024-12-18T01:41:01.4984603Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:41:01.4986991Z warnings.warn( 2024-12-18T01:41:01.9151840Z 2024-12-18T01:41:03.5347953Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2024-12-18T01:41:03.5349467Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2024-12-18T01:41:03.9208062Z 2024-12-18T01:41:03.9208559Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:41:03.9208952Z cuda train RobertaForQuestionAnswering 2024-12-18T01:41:36.5695016Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:41:36.5695901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:41:36.5696638Z pred = mod(**cloned_inputs) 2024-12-18T01:41:36.5697333Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 1500, in forward 2024-12-18T01:41:36.5698026Z outputs = self.roberta( 2024-12-18T01:41:36.5699391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 828, in forward 2024-12-18T01:41:36.5700276Z embedding_output = self.embeddings( 2024-12-18T01:41:36.5700965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/roberta/modeling_roberta.py", line 125, in forward 2024-12-18T01:41:36.5701735Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:41:36.5701989Z 2024-12-18T01:41:36.7261755Z W1218 01:41:36.725000 21417 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:41:38.4306686Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:42:15.3037356Z pass 2024-12-18T01:42:15.3690481Z TIMING: entire_frame_compile:57.24994 _recursive_pre_grad_passes:0.06169 _recursive_joint_graph_passes:1.89086 _recursive_post_grad_passes:0.82757 async_compile.wait:0.5703 code_gen:16.00077 inductor_compile:31.00609 backend_compile:43.40333 entire_backward_compile:7.83216 total_wall_time:65.08209 2024-12-18T01:42:15.3692113Z STATS: call_* op count: 1400 | FakeTensorMode.__torch_dispatch__:61982 | FakeTensor.__torch_dispatch__:15432 | ProxyTorchDispatchMode.__torch_dispatch__:28393 2024-12-18T01:42:15.3692973Z Dynamo produced 2 graphs covering 1400 ops with 5 graph breaks (4 unique) 2024-12-18T01:42:22.5824990Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:42:22.5826204Z warnings.warn( 2024-12-18T01:42:22.8670247Z 2024-12-18T01:42:23.5675670Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:42:23.5676035Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:42:23.5676382Z cuda train Speech2Text2ForCausalLM 2024-12-18T01:42:23.5843142Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine 2024-12-18T01:42:34.0090136Z pass 2024-12-18T01:42:34.0144630Z TIMING: entire_frame_compile:4.42507 _recursive_pre_grad_passes:0.00768 _recursive_joint_graph_passes:0.37464 inductor_compile:2.79163 backend_compile:3.51014 async_compile.precompile:0.1132 async_compile.wait:0.80991 _recursive_post_grad_passes:0.11963 code_gen:1.87688 entire_backward_compile:1.23412 total_wall_time:5.65919 2024-12-18T01:42:34.0147926Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:4447 | ProxyTorchDispatchMode.__torch_dispatch__:1868 | FakeTensor.__torch_dispatch__:739 2024-12-18T01:42:34.0149026Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique) 2024-12-18T01:42:38.7040824Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:42:38.7042152Z warnings.warn( 2024-12-18T01:42:39.0234940Z 2024-12-18T01:42:40.5281686Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:42:40.5282059Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:42:40.5282410Z cuda train T5ForConditionalGeneration 2024-12-18T01:43:19.8579613Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:43:19.8580555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:43:19.8581300Z pred = mod(**cloned_inputs) 2024-12-18T01:43:19.8581936Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-12-18T01:43:19.8582585Z encoder_outputs = self.encoder( 2024-12-18T01:43:19.8583213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-12-18T01:43:19.8583879Z inputs_embeds = self.embed_tokens(input_ids) 2024-12-18T01:43:19.8584133Z 2024-12-18T01:43:19.9830281Z W1218 01:43:19.981000 21810 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:43:21.3473875Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:43:50.6617437Z pass 2024-12-18T01:43:50.7218673Z TIMING: entire_frame_compile:54.2555 _recursive_pre_grad_passes:0.05484 _recursive_joint_graph_passes:1.64909 _recursive_post_grad_passes:0.83548 async_compile.wait:4.50706 code_gen:16.8065 inductor_compile:29.82758 backend_compile:40.34031 entire_backward_compile:8.47119 total_wall_time:62.72668 2024-12-18T01:43:50.7220333Z STATS: call_* op count: 1487 | FakeTensorMode.__torch_dispatch__:67183 | ProxyTorchDispatchMode.__torch_dispatch__:31555 | FakeTensor.__torch_dispatch__:12984 2024-12-18T01:43:50.7221224Z Dynamo produced 2 graphs covering 1487 ops with 5 graph breaks (4 unique) 2024-12-18T01:43:57.4513748Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:43:57.4514980Z warnings.warn( 2024-12-18T01:43:57.7442506Z 2024-12-18T01:43:59.2377351Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:43:59.2377712Z loading model: 0it [00:01, ?it/s] 2024-12-18T01:43:59.2378033Z cuda train T5Small 2024-12-18T01:44:32.6916681Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:44:32.6917576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:44:32.6918301Z pred = mod(**cloned_inputs) 2024-12-18T01:44:32.6918931Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1706, in forward 2024-12-18T01:44:32.6919572Z encoder_outputs = self.encoder( 2024-12-18T01:44:32.6920224Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1016, in forward 2024-12-18T01:44:32.6920893Z inputs_embeds = self.embed_tokens(input_ids) 2024-12-18T01:44:32.6921142Z 2024-12-18T01:44:32.8174283Z W1218 01:44:32.816000 22171 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:44:34.1970867Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:44:45.3159729Z pass 2024-12-18T01:44:45.3761264Z TIMING: entire_frame_compile:31.43916 _recursive_pre_grad_passes:0.05485 _recursive_joint_graph_passes:1.52278 async_compile.wait:0.40159 inductor_compile:9.24662 backend_compile:17.28075 _recursive_post_grad_passes:0.49426 code_gen:3.43316 entire_backward_compile:7.95769 total_wall_time:39.39685 2024-12-18T01:44:45.3763003Z STATS: call_* op count: 1487 | FakeTensorMode.__torch_dispatch__:48635 | ProxyTorchDispatchMode.__torch_dispatch__:24589 | FakeTensor.__torch_dispatch__:5970 2024-12-18T01:44:45.3763867Z Dynamo produced 2 graphs covering 1487 ops with 5 graph breaks (4 unique) 2024-12-18T01:44:50.8060430Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:44:50.8061670Z warnings.warn( 2024-12-18T01:44:51.1373778Z 2024-12-18T01:44:55.0780546Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:44:55.0781030Z loading model: 0it [00:03, ?it/s] 2024-12-18T01:44:55.0781519Z cuda train TrOCRForCausalLM 2024-12-18T01:44:55.1087528Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2024-12-18T01:45:03.7703275Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:45:03.7704112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 66, in forward 2024-12-18T01:45:03.7704830Z return super().forward(positions + self.offset) 2024-12-18T01:45:03.7705095Z 2024-12-18T01:45:09.9093421Z pass 2024-12-18T01:45:09.9894589Z TIMING: entire_frame_compile:5.15363 _recursive_pre_grad_passes:0.00667 _recursive_joint_graph_passes:0.9303 inductor_compile:3.25654 backend_compile:4.22658 _recursive_post_grad_passes:0.11929 async_compile.precompile:0.15055 async_compile.wait:1.03984 code_gen:2.28215 entire_backward_compile:1.56599 total_wall_time:6.71961 2024-12-18T01:45:09.9896345Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:4531 | FakeTensor.__torch_dispatch__:790 | ProxyTorchDispatchMode.__torch_dispatch__:1916 2024-12-18T01:45:09.9897183Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-12-18T01:45:14.6505454Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:45:14.6506679Z warnings.warn( 2024-12-18T01:45:14.8966686Z 2024-12-18T01:45:24.5842611Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:45:24.5842964Z loading model: 0it [00:09, ?it/s] 2024-12-18T01:45:24.5843295Z cuda train XGLMForCausalLM 2024-12-18T01:45:24.6373169Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2024-12-18T01:45:28.6811940Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 2024-12-18T01:45:28.6813232Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 174, in forward 2024-12-18T01:45:28.6814102Z position_ids += self.offset 2024-12-18T01:45:28.6814299Z 2024-12-18T01:45:45.8162229Z pass 2024-12-18T01:45:45.9093422Z TIMING: entire_frame_compile:4.69342 _recursive_pre_grad_passes:0.00754 _recursive_joint_graph_passes:0.23533 inductor_compile:3.37044 backend_compile:3.72969 async_compile.precompile:0.10651 async_compile.wait:1.06068 _recursive_post_grad_passes:0.12366 code_gen:2.38215 entire_backward_compile:1.52109 total_wall_time:6.21451 2024-12-18T01:45:45.9095559Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:4610 | ProxyTorchDispatchMode.__torch_dispatch__:1946 | FakeTensor.__torch_dispatch__:835 2024-12-18T01:45:45.9096405Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique) 2024-12-18T01:45:50.6444108Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:45:50.6445462Z warnings.warn( 2024-12-18T01:45:51.0261786Z 2024-12-18T01:45:55.5885520Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:45:55.5885890Z loading model: 0it [00:04, ?it/s] 2024-12-18T01:45:55.5886221Z cuda train XLNetLMHeadModel 2024-12-18T01:46:49.6041825Z W1218 01:46:49.603000 22778 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program 2024-12-18T01:47:05.2173911Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat). Found from : 2024-12-18T01:47:05.2174895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:47:05.2175618Z pred = mod(**cloned_inputs) 2024-12-18T01:47:05.2176312Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1446, in forward 2024-12-18T01:47:05.2177012Z transformer_outputs = self.transformer( 2024-12-18T01:47:05.2177711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1203, in forward 2024-12-18T01:47:05.2178470Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2024-12-18T01:47:05.2179318Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1055, in relative_positional_encoding 2024-12-18T01:47:05.2180150Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2024-12-18T01:47:05.2180964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1014, in positional_embedding 2024-12-18T01:47:05.2181686Z pos_emb = pos_emb[:, None, :] 2024-12-18T01:47:05.2181889Z 2024-12-18T01:47:26.3216942Z W1218 01:47:26.320000 22778 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:47:31.4320886Z ('Grad tensors ["L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad", "L['self'].param_groups[0]['params'][329].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][332].grad", "L['self'].param_groups[0]['params'][334].grad", "L['self'].param_groups[0]['params'][335].grad", "L['self'].param_groups[0]['params'][336].grad", "L['self'].param_groups[0]['params'][337].grad", "L['self'].param_groups[0]['params'][338].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][340].grad", "L['self'].param_groups[0]['params'][341].grad", "L['self'].param_groups[0]['params'][342].grad", "L['self'].param_groups[0]['params'][343].grad", "L['self'].param_groups[0]['params'][344].grad", "L['self'].param_groups[0]['params'][345].grad", "L['self'].param_groups[0]['params'][346].grad", "L['self'].param_groups[0]['params'][347].grad", "L['self'].param_groups[0]['params'][349].grad", "L['self'].param_groups[0]['params'][351].grad", "L['self'].param_groups[0]['params'][352].grad", "L['self'].param_groups[0]['params'][353].grad", "L['self'].param_groups[0]['params'][354].grad", "L['self'].param_groups[0]['params'][355].grad", "L['self'].param_groups[0]['params'][356].grad", "L['self'].param_groups[0]['params'][357].grad", "L['self'].param_groups[0]['params'][358].grad", "L['self'].param_groups[0]['params'][359].grad", "L['self'].param_groups[0]['params'][360].grad", "L['self'].param_groups[0]['params'][361].grad", "L['self'].param_groups[0]['params'][362].grad", "L['self'].param_groups[0]['params'][363].grad", "L['self'].param_groups[0]['params'][364].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][368].grad", "L['self'].param_groups[0]['params'][369].grad", "L['self'].param_groups[0]['params'][370].grad", "L['self'].param_groups[0]['params'][371].grad", "L['self'].param_groups[0]['params'][372].grad", "L['self'].param_groups[0]['params'][373].grad", "L['self'].param_groups[0]['params'][374].grad", "L['self'].param_groups[0]['params'][375].grad", "L['self'].param_groups[0]['params'][376].grad", "L['self'].param_groups[0]['params'][377].grad", "L['self'].param_groups[0]['params'][378].grad", "L['self'].param_groups[0]['params'][379].grad", "L['self'].param_groups[0]['params'][380].grad", "L['self'].param_groups[0]['params'][381].grad", "L['self'].param_groups[0]['params'][383].grad", "L['self'].param_groups[0]['params'][385].grad", "L['self'].param_groups[0]['params'][386].grad", "L['self'].param_groups[0]['params'][387].grad", "L['self'].param_groups[0]['params'][388].grad", "L['self'].param_groups[0]['params'][389].grad", "L['self'].param_groups[0]['params'][390].grad", "L['self'].param_groups[0]['params'][391].grad", "L['self'].param_groups[0]['params'][392].grad", "L['self'].param_groups[0]['params'][393].grad", "L['self'].param_groups[0]['params'][394].grad", "L['self'].param_groups[0]['params'][395].grad", "L['self'].param_groups[0]['params'][396].grad", "L['self'].param_groups[0]['params'][397].grad", "L['self'].param_groups[0]['params'][398].grad", "L['self'].param_groups[0]['params'][400].grad", "L['self'].param_groups[0]['params'][402].grad", "L['self'].param_groups[0]['params'][403].grad", "L['self'].param_groups[0]['params'][404].grad", "L['self'].param_groups[0]['params'][405].grad", "L['self'].param_groups[0]['params'][406].grad", "L['self'].param_groups[0]['params'][407].grad", "L['self'].param_groups[0]['params'][408].grad", "L['self'].param_groups[0]['params'][409].grad", "L['self'].param_groups[0]['params'][410].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:48:50.0883854Z pass 2024-12-18T01:48:50.2951460Z TIMING: entire_frame_compile:141.99476 _recursive_pre_grad_passes:0.11025 _recursive_joint_graph_passes:3.90745 _recursive_post_grad_passes:3.30475 async_compile.wait:7.32763 code_gen:42.60135 inductor_compile:77.64891 backend_compile:108.57126 entire_backward_compile:18.63328 total_wall_time:160.62804 2024-12-18T01:48:50.2953166Z STATS: call_* op count: 2599 | FakeTensorMode.__torch_dispatch__:166628 | FakeTensor.__torch_dispatch__:31374 | ProxyTorchDispatchMode.__torch_dispatch__:69620 2024-12-18T01:48:50.2954049Z Dynamo produced 2 graphs covering 2599 ops with 5 graph breaks (4 unique) 2024-12-18T01:49:00.8091606Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`. 2024-12-18T01:49:00.8092827Z warnings.warn( 2024-12-18T01:49:01.0785422Z 2024-12-18T01:49:03.3606189Z loading model: 0it [00:00, ?it/s] 2024-12-18T01:49:03.3606553Z loading model: 0it [00:02, ?it/s] 2024-12-18T01:49:03.3606882Z cuda train YituTechConvBert 2024-12-18T01:49:54.5356061Z skipping cudagraphs due to deterministic index put. Found from : 2024-12-18T01:49:54.5357213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 528, in torch_dynamo_resume_in_forward_and_backward_pass_at_526 2024-12-18T01:49:54.5357939Z pred = mod(**cloned_inputs) 2024-12-18T01:49:54.5358639Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 918, in forward 2024-12-18T01:49:54.5359348Z generator_hidden_states = self.convbert( 2024-12-18T01:49:54.5360055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 834, in forward 2024-12-18T01:49:54.5360746Z hidden_states = self.embeddings( 2024-12-18T01:49:54.5361431Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/convbert/modeling_convbert.py", line 230, in forward 2024-12-18T01:49:54.5362156Z inputs_embeds = self.word_embeddings(input_ids) 2024-12-18T01:49:54.5362409Z 2024-12-18T01:49:54.8277112Z W1218 01:49:54.826000 23312 site-packages/torch/_logging/_internal.py:1089] [4/0] Profiler function will be ignored 2024-12-18T01:49:57.5029450Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',) 2024-12-18T01:50:57.8911553Z pass 2024-12-18T01:50:57.9463460Z TIMING: entire_frame_compile:93.54071 _recursive_pre_grad_passes:0.08894 _recursive_joint_graph_passes:3.93159 _recursive_post_grad_passes:1.45823 async_compile.wait:5.607 code_gen:30.34665 inductor_compile:52.65333 backend_compile:71.76516 entire_backward_compile:12.75933 total_wall_time:106.30004 2024-12-18T01:50:57.9466654Z STATS: call_* op count: 2085 | FakeTensorMode.__torch_dispatch__:93071 | FakeTensor.__torch_dispatch__:22131 | ProxyTorchDispatchMode.__torch_dispatch__:42192 2024-12-18T01:50:57.9467592Z Dynamo produced 2 graphs covering 2085 ops with 5 graph breaks (4 unique) 2024-12-18T01:51:03.3111496Z accuracy pass_rate=93.48% 2024-12-18T01:51:03.3113773Z calls_captured gmean=0.00x mean=973.674x 2024-12-18T01:51:03.3117784Z unique_graphs gmean=0.00x mean=3.391x 2024-12-18T01:51:03.3121668Z graph_breaks gmean=0.00x mean=5.196x 2024-12-18T01:51:03.3125469Z unique_graph_breaks gmean=0.00x mean=4.043x 2024-12-18T01:51:03.3129210Z autograd_captures gmean=0.00x mean=0.000x 2024-12-18T01:51:03.3133036Z autograd_compiles gmean=0.00x mean=0.000x 2024-12-18T01:51:03.3136962Z cudagraph_skips gmean=0.00x mean=0.935x 2024-12-18T01:51:04.8786061Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_huggingface_training.csv 2024-12-18T01:51:05.1428581Z AlbertForMaskedLM PASS 2024-12-18T01:51:05.1433262Z AlbertForQuestionAnswering PASS 2024-12-18T01:51:05.1438221Z AllenaiLongformerBase PASS 2024-12-18T01:51:05.1442105Z BartForCausalLM PASS 2024-12-18T01:51:05.1446594Z BartForConditionalGeneration PASS 2024-12-18T01:51:05.1451128Z BertForMaskedLM PASS 2024-12-18T01:51:05.1455672Z BertForQuestionAnswering PASS 2024-12-18T01:51:05.1460221Z BlenderbotForCausalLM XFAIL 2024-12-18T01:51:05.1464858Z BlenderbotSmallForCausalLM PASS 2024-12-18T01:51:05.1469343Z BlenderbotSmallForConditionalGeneration PASS 2024-12-18T01:51:05.1474093Z CamemBert PASS 2024-12-18T01:51:05.1478528Z DebertaForMaskedLM PASS 2024-12-18T01:51:05.1483027Z DebertaForQuestionAnswering PASS 2024-12-18T01:51:05.1487476Z DebertaV2ForMaskedLM XFAIL 2024-12-18T01:51:05.1491947Z DebertaV2ForQuestionAnswering XFAIL 2024-12-18T01:51:05.1496459Z DistilBertForMaskedLM PASS 2024-12-18T01:51:05.1501332Z DistilBertForQuestionAnswering PASS 2024-12-18T01:51:05.1505767Z DistillGPT2 PASS 2024-12-18T01:51:05.1510317Z ElectraForCausalLM PASS 2024-12-18T01:51:05.1514947Z ElectraForQuestionAnswering PASS 2024-12-18T01:51:05.1519329Z GPT2ForSequenceClassification PASS 2024-12-18T01:51:05.1523754Z GoogleFnet PASS 2024-12-18T01:51:05.1528369Z LayoutLMForMaskedLM PASS 2024-12-18T01:51:05.1532800Z LayoutLMForSequenceClassification PASS 2024-12-18T01:51:05.1537205Z M2M100ForConditionalGeneration PASS 2024-12-18T01:51:05.1541969Z MBartForCausalLM PASS 2024-12-18T01:51:05.1546185Z MBartForConditionalGeneration PASS 2024-12-18T01:51:05.1550876Z MT5ForConditionalGeneration PASS 2024-12-18T01:51:05.1555221Z MegatronBertForCausalLM PASS 2024-12-18T01:51:05.1559743Z MegatronBertForQuestionAnswering PASS 2024-12-18T01:51:05.1564125Z MobileBertForMaskedLM PASS 2024-12-18T01:51:05.1568653Z MobileBertForQuestionAnswering PASS 2024-12-18T01:51:05.1573084Z OPTForCausalLM PASS 2024-12-18T01:51:05.1577664Z PLBartForCausalLM PASS 2024-12-18T01:51:05.1582098Z PLBartForConditionalGeneration PASS 2024-12-18T01:51:05.1586692Z PegasusForCausalLM PASS 2024-12-18T01:51:05.1591235Z PegasusForConditionalGeneration PASS 2024-12-18T01:51:05.1595751Z RobertaForCausalLM PASS 2024-12-18T01:51:05.1600454Z RobertaForQuestionAnswering PASS 2024-12-18T01:51:05.1605037Z Speech2Text2ForCausalLM PASS 2024-12-18T01:51:05.1609416Z T5ForConditionalGeneration PASS 2024-12-18T01:51:05.1613920Z T5Small PASS 2024-12-18T01:51:05.1618429Z TrOCRForCausalLM PASS 2024-12-18T01:51:05.1623063Z XGLMForCausalLM PASS 2024-12-18T01:51:05.1627429Z XLNetLMHeadModel PASS 2024-12-18T01:51:05.1632394Z YituTechConvBert PASS 2024-12-18T01:51:05.2051614Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_huggingface_training.csv 2024-12-18T01:51:05.4690422Z AlbertForMaskedLM PASS 2024-12-18T01:51:05.4696150Z AlbertForQuestionAnswering PASS 2024-12-18T01:51:05.4699925Z AllenaiLongformerBase PASS 2024-12-18T01:51:05.4704186Z BartForCausalLM PASS 2024-12-18T01:51:05.4709162Z BartForConditionalGeneration PASS 2024-12-18T01:51:05.4714109Z BertForMaskedLM PASS 2024-12-18T01:51:05.4718619Z BertForQuestionAnswering PASS 2024-12-18T01:51:05.4723332Z BlenderbotForCausalLM PASS 2024-12-18T01:51:05.4728238Z BlenderbotSmallForCausalLM PASS 2024-12-18T01:51:05.4732979Z BlenderbotSmallForConditionalGeneration PASS 2024-12-18T01:51:05.4737665Z CamemBert PASS 2024-12-18T01:51:05.4742405Z DebertaForMaskedLM PASS 2024-12-18T01:51:05.4747177Z DebertaForQuestionAnswering PASS 2024-12-18T01:51:05.4752442Z DebertaV2ForMaskedLM PASS 2024-12-18T01:51:05.4757384Z DebertaV2ForQuestionAnswering PASS 2024-12-18T01:51:05.4762392Z DistilBertForMaskedLM PASS 2024-12-18T01:51:05.4767484Z DistilBertForQuestionAnswering PASS 2024-12-18T01:51:05.4772415Z DistillGPT2 PASS 2024-12-18T01:51:05.4777457Z ElectraForCausalLM PASS 2024-12-18T01:51:05.4782415Z ElectraForQuestionAnswering PASS 2024-12-18T01:51:05.4787475Z GPT2ForSequenceClassification PASS 2024-12-18T01:51:05.4792681Z GoogleFnet PASS 2024-12-18T01:51:05.4797601Z LayoutLMForMaskedLM PASS 2024-12-18T01:51:05.4803113Z LayoutLMForSequenceClassification PASS 2024-12-18T01:51:05.4808051Z M2M100ForConditionalGeneration PASS 2024-12-18T01:51:05.4813145Z MBartForCausalLM PASS 2024-12-18T01:51:05.4818069Z MBartForConditionalGeneration PASS 2024-12-18T01:51:05.4823111Z MT5ForConditionalGeneration PASS 2024-12-18T01:51:05.4827957Z MegatronBertForCausalLM PASS 2024-12-18T01:51:05.4833359Z MegatronBertForQuestionAnswering PASS 2024-12-18T01:51:05.4838025Z MobileBertForMaskedLM PASS 2024-12-18T01:51:05.4843008Z MobileBertForQuestionAnswering PASS 2024-12-18T01:51:05.4847952Z OPTForCausalLM PASS 2024-12-18T01:51:05.4853007Z PLBartForCausalLM PASS 2024-12-18T01:51:05.4858076Z PLBartForConditionalGeneration PASS 2024-12-18T01:51:05.4862963Z PegasusForCausalLM PASS 2024-12-18T01:51:05.4867815Z PegasusForConditionalGeneration PASS 2024-12-18T01:51:05.4872828Z RobertaForCausalLM PASS 2024-12-18T01:51:05.4877720Z RobertaForQuestionAnswering PASS 2024-12-18T01:51:05.4882719Z Speech2Text2ForCausalLM PASS 2024-12-18T01:51:05.4887689Z T5ForConditionalGeneration PASS 2024-12-18T01:51:05.4892766Z T5Small PASS 2024-12-18T01:51:05.4897691Z TrOCRForCausalLM PASS 2024-12-18T01:51:05.4904352Z XGLMForCausalLM PASS 2024-12-18T01:51:05.4909468Z XLNetLMHeadModel PASS 2024-12-18T01:51:05.4914535Z YituTechConvBert PASS 2024-12-18T01:51:05.5341207Z + cleanup_workspace 2024-12-18T01:51:05.5341727Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-12-18T01:51:05.5342526Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-12-18T01:51:05.5343147Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-12-18T01:51:05.5343617Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T01:51:05.5344151Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-12-18T01:51:05.5344747Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-12-18T01:51:05.5345221Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-12-18T01:51:06.2647909Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6 2024-12-18T01:51:06.2648424Z with: 2024-12-18T01:51:06.2648716Z benchmark-results-dir: test/test-reports 2024-12-18T01:51:06.2649068Z dry-run: false 2024-12-18T01:51:06.2659603Z schema-version: v3 2024-12-18T01:51:06.2660100Z github-token: *** 2024-12-18T01:51:06.2660347Z env: 2024-12-18T01:51:06.2660571Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:06.2660924Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:06.2661698Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:06.2662292Z ##[endgroup] 2024-12-18T01:51:06.2690465Z ##[group]Run set -eux 2024-12-18T01:51:06.2690738Z set -eux 2024-12-18T01:51:06.2691025Z python3 -mpip install boto3==1.35.33 2024-12-18T01:51:06.2705548Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:06.2705911Z env: 2024-12-18T01:51:06.2706156Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:06.2706494Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:06.2707038Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:06.2707522Z ##[endgroup] 2024-12-18T01:51:06.2743334Z + python3 -mpip install boto3==1.35.33 2024-12-18T01:51:06.5218924Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T01:51:07.6712987Z Collecting boto3==1.35.33 2024-12-18T01:51:07.7081633Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2024-12-18T01:51:08.9986992Z Collecting botocore<1.36.0,>=1.35.33 2024-12-18T01:51:09.0030592Z Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB) 2024-12-18T01:51:09.1473369Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2024-12-18T01:51:09.2177670Z Collecting s3transfer<0.11.0,>=0.10.0 2024-12-18T01:51:09.2220498Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2024-12-18T01:51:09.2312886Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2024-12-18T01:51:09.2323742Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2024-12-18T01:51:09.3730699Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2024-12-18T01:51:09.4572861Z Installing collected packages: botocore, s3transfer, boto3 2024-12-18T01:51:10.0645737Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4 2024-12-18T01:51:10.1763510Z ##[group]Run set -eux 2024-12-18T01:51:10.1763792Z set -eux 2024-12-18T01:51:10.1764026Z  2024-12-18T01:51:10.1764267Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2024-12-18T01:51:10.1764637Z  echo "Missing github-token input" 2024-12-18T01:51:10.1764959Z  exit 1 2024-12-18T01:51:10.1765191Z fi 2024-12-18T01:51:10.1774831Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:10.1775201Z env: 2024-12-18T01:51:10.1775415Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:10.1775757Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:10.1776311Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:10.1777164Z GITHUB_TOKEN: *** 2024-12-18T01:51:10.1777428Z ##[endgroup] 2024-12-18T01:51:10.1808418Z + [[ -z *** ]] 2024-12-18T01:51:10.1856999Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2024-12-18T01:51:10.1857436Z with: 2024-12-18T01:51:10.1857781Z github-token: *** 2024-12-18T01:51:10.1858024Z env: 2024-12-18T01:51:10.1858238Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:10.1858579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:10.1859356Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:10.1859857Z ##[endgroup] 2024-12-18T01:51:10.1879335Z ##[group]Run set -eux 2024-12-18T01:51:10.1879622Z set -eux 2024-12-18T01:51:10.1879871Z  2024-12-18T01:51:10.1880344Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T01:51:10.1889263Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:10.1889694Z env: 2024-12-18T01:51:10.1889923Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:10.1890270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:10.1890843Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:10.1891649Z GITHUB_TOKEN: *** 2024-12-18T01:51:10.1891892Z ##[endgroup] 2024-12-18T01:51:10.1921227Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-096ab043c4c0d1de7 2024-12-18T01:51:11.4315271Z setting job-id=34567152522 2024-12-18T01:51:11.4315841Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T01:51:11.4419174Z ##[group]Run set -eux 2024-12-18T01:51:11.4419507Z set -eux 2024-12-18T01:51:11.4419738Z  2024-12-18T01:51:11.4420128Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2024-12-18T01:51:11.4420648Z  --schema-version "${SCHEMA_VERSION}" \ 2024-12-18T01:51:11.4420994Z  --repo "${REPO}" \ 2024-12-18T01:51:11.4421299Z  --head-branch "${HEAD_BRANCH}" \ 2024-12-18T01:51:11.4421631Z  --head-sha "${HEAD_SHA}" \ 2024-12-18T01:51:11.4421971Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2024-12-18T01:51:11.4422323Z  --run-attempt "${RUN_ATTEMPT}" \ 2024-12-18T01:51:11.4422662Z  --job-id "${JOB_ID}" \ 2024-12-18T01:51:11.4422978Z  --job-name "${JOB_NAME}" 2024-12-18T01:51:11.4431911Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:11.4432279Z env: 2024-12-18T01:51:11.4432488Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:11.4432821Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:11.4433372Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:11.4433858Z SCHEMA_VERSION: v3 2024-12-18T01:51:11.4434102Z REPO: pytorch/pytorch 2024-12-18T01:51:11.4434571Z HEAD_BRANCH: refs/heads/release/2.6 2024-12-18T01:51:11.4434924Z HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T01:51:11.4435274Z WORKFLOW_RUN_ID: 12383255690 2024-12-18T01:51:11.4435549Z RUN_ATTEMPT: 1 2024-12-18T01:51:11.4435780Z JOB_ID: 34567152522 2024-12-18T01:51:11.4436255Z JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) 2024-12-18T01:51:11.4436785Z ##[endgroup] 2024-12-18T01:51:11.4467194Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567152522 --job-name 'cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)' 2024-12-18T01:51:11.4804116Z ##[group]Run set -eux 2024-12-18T01:51:11.4804410Z set -eux 2024-12-18T01:51:11.4804643Z  2024-12-18T01:51:11.4804905Z # TODO (huydhn): Implement this part 2024-12-18T01:51:11.4805281Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2024-12-18T01:51:11.4814142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:11.4814529Z env: 2024-12-18T01:51:11.4814748Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:11.4815300Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:11.4815863Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:11.4816359Z ##[endgroup] 2024-12-18T01:51:11.4843241Z + echo 'runners=[]' 2024-12-18T01:51:11.4871179Z ##[group]Run set -eux 2024-12-18T01:51:11.4871446Z set -eux 2024-12-18T01:51:11.4871672Z  2024-12-18T01:51:11.4871931Z # TODO (huydhn): Implement this part 2024-12-18T01:51:11.4872319Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2024-12-18T01:51:11.4880412Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:11.4880868Z env: 2024-12-18T01:51:11.4881091Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:11.4881428Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:11.4881986Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:11.4882473Z ##[endgroup] 2024-12-18T01:51:11.4909149Z + echo 'dependencies={}' 2024-12-18T01:51:11.4934742Z ##[group]Run set -eux 2024-12-18T01:51:11.4935024Z set -eux 2024-12-18T01:51:11.4935259Z  2024-12-18T01:51:11.4935534Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2024-12-18T01:51:11.4935974Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2024-12-18T01:51:11.4936460Z  # We don't want the job to fail if the directory doesn't exist 2024-12-18T01:51:11.4936851Z  exit 0 2024-12-18T01:51:11.4937080Z fi 2024-12-18T01:51:11.4937308Z  2024-12-18T01:51:11.4937547Z if [[ "${DRY_RUN}" == "true" ]]; then 2024-12-18T01:51:11.4938020Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T01:51:11.4938579Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T01:51:11.4939008Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T01:51:11.4939369Z  --runners "${RUNNER_INFO}" \ 2024-12-18T01:51:11.4939725Z  --dependencies "${DEPENDENCIES}" \ 2024-12-18T01:51:11.4940109Z  --dry-run 2024-12-18T01:51:11.4940362Z else 2024-12-18T01:51:11.4940741Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T01:51:11.4941283Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T01:51:11.4941703Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T01:51:11.4942057Z  --runners "${RUNNER_INFO}" \ 2024-12-18T01:51:11.4942557Z  --dependencies "${DEPENDENCIES}" 2024-12-18T01:51:11.4942871Z fi 2024-12-18T01:51:11.4951033Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:11.4951405Z env: 2024-12-18T01:51:11.4951617Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:11.4951958Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:11.4952510Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:11.4953029Z BENCHMARK_RESULTS_DIR: test/test-reports 2024-12-18T01:51:11.4953346Z DRY_RUN: false 2024-12-18T01:51:11.4954662Z BENCHMARK_METADATA: {"timestamp": 1734486671, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567152522} 2024-12-18T01:51:11.4956017Z RUNNER_INFO: [] 2024-12-18T01:51:11.4956253Z DEPENDENCIES: {} 2024-12-18T01:51:11.4956493Z ##[endgroup] 2024-12-18T01:51:11.4984602Z + [[ ! -d test/test-reports ]] 2024-12-18T01:51:11.4984895Z + [[ false == \t\r\u\e ]] 2024-12-18T01:51:11.4987409Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734486671, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567152522}' --runners '[]' --dependencies '{}' 2024-12-18T01:51:11.6583291Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152522/inference_huggingface.json 2024-12-18T01:51:11.6996389Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2024-12-18T01:51:11.8950554Z INFO:root:Upload test/test-reports/inference_huggingface_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152522/inference_huggingface_graph_breaks.json 2024-12-18T01:51:12.0159654Z INFO:root:Upload test/test-reports/training_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152522/training_huggingface.json 2024-12-18T01:51:12.1904151Z INFO:root:Upload test/test-reports/training_huggingface_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152522/training_huggingface_graph_breaks.json 2024-12-18T01:51:12.4155314Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T01:51:12.4155708Z cat test/**/*_toprint.log || true 2024-12-18T01:51:12.4164039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:12.4164421Z env: 2024-12-18T01:51:12.4164628Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.4164960Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.4165512Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.4165997Z ##[endgroup] 2024-12-18T01:51:12.4251076Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T01:51:12.4283569Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-12-18T01:51:12.4283921Z kill "$MONITOR_SCRIPT_PID" 2024-12-18T01:51:12.4291903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:12.4292283Z env: 2024-12-18T01:51:12.4292505Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.4292855Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.4293409Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.4293909Z MONITOR_SCRIPT_PID: 53443 2024-12-18T01:51:12.4294182Z ##[endgroup] 2024-12-18T01:51:12.4438192Z Prepare all required actions 2024-12-18T01:51:12.4438748Z Getting action download info 2024-12-18T01:51:12.5868965Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T01:51:12.9052441Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T01:51:12.9052808Z with: 2024-12-18T01:51:12.9053235Z file-suffix: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T01:51:12.9053740Z s3-bucket: gha-artifacts 2024-12-18T01:51:12.9054013Z env: 2024-12-18T01:51:12.9054235Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.9054580Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.9055138Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.9055633Z ##[endgroup] 2024-12-18T01:51:12.9085125Z ##[group]Run # Remove any previous test jsons if they exist 2024-12-18T01:51:12.9085680Z # Remove any previous test jsons if they exist 2024-12-18T01:51:12.9086057Z rm -f test-jsons-*.zip 2024-12-18T01:51:12.9086481Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-12-18T01:51:12.9095370Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:12.9095737Z env: 2024-12-18T01:51:12.9095953Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.9096287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.9096826Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.9097507Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T01:51:12.9097987Z ##[endgroup] 2024-12-18T01:51:12.9301163Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2024-12-18T01:51:12.9306495Z adding: test/test-reports/inference_huggingface_graph_breaks.json (deflated 97%) 2024-12-18T01:51:12.9367340Z adding: test/test-reports/training_huggingface.json (deflated 99%) 2024-12-18T01:51:12.9499300Z adding: test/test-reports/training_huggingface_graph_breaks.json (deflated 99%) 2024-12-18T01:51:12.9536309Z ##[group]Run # Remove any previous test reports if they exist 2024-12-18T01:51:12.9536950Z # Remove any previous test reports if they exist 2024-12-18T01:51:12.9537439Z rm -f test-reports-*.zip 2024-12-18T01:51:12.9538037Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-12-18T01:51:12.9562123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:12.9562520Z env: 2024-12-18T01:51:12.9562750Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.9563090Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.9563642Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.9564323Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T01:51:12.9564809Z ##[endgroup] 2024-12-18T01:51:12.9623148Z adding: test/test-reports/inference_huggingface.csv (deflated 76%) 2024-12-18T01:51:12.9623954Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%) 2024-12-18T01:51:12.9624840Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%) 2024-12-18T01:51:12.9625541Z adding: test/test-reports/training_huggingface.csv (deflated 74%) 2024-12-18T01:51:12.9632542Z adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%) 2024-12-18T01:51:12.9633432Z adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%) 2024-12-18T01:51:12.9661930Z ##[group]Run # Remove any previous usage logs if they exist 2024-12-18T01:51:12.9662390Z # Remove any previous usage logs if they exist 2024-12-18T01:51:12.9662757Z rm -f logs-*.zip 2024-12-18T01:51:12.9663232Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-12-18T01:51:12.9664103Z # so check to see if the file exists first 2024-12-18T01:51:12.9664461Z if [ -f 'usage_log.txt' ]; then 2024-12-18T01:51:12.9664832Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-12-18T01:51:12.9665177Z fi 2024-12-18T01:51:12.9665554Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-12-18T01:51:12.9666111Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-12-18T01:51:12.9666513Z fi 2024-12-18T01:51:12.9674608Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:12.9674986Z env: 2024-12-18T01:51:12.9675207Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:12.9675549Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:12.9676106Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:12.9676799Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T01:51:12.9677283Z ##[endgroup] 2024-12-18T01:51:12.9945772Z adding: usage_log.txt (deflated 98%) 2024-12-18T01:51:12.9998445Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-12-18T01:51:12.9999369Z # Remove any previous debugging artifacts if they exist 2024-12-18T01:51:12.9999773Z rm -f debug-*.zip 2024-12-18T01:51:13.0000063Z if [ -d 'test/debug' ]; then 2024-12-18T01:51:13.0000423Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-12-18T01:51:13.0000757Z fi 2024-12-18T01:51:13.0009237Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:13.0009614Z env: 2024-12-18T01:51:13.0009837Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:13.0010206Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:13.0010781Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:13.0011480Z FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522 2024-12-18T01:51:13.0011985Z ##[endgroup] 2024-12-18T01:51:13.0115098Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:51:13.0115439Z with: 2024-12-18T01:51:13.0115662Z s3-bucket: gha-artifacts 2024-12-18T01:51:13.0115993Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:13.0116341Z retention-days: 14 2024-12-18T01:51:13.0116609Z if-no-files-found: warn 2024-12-18T01:51:13.0116890Z path: test-jsons-*.zip 2024-12-18T01:51:13.0117146Z name: artifact 2024-12-18T01:51:13.0117377Z region: us-east-1 2024-12-18T01:51:13.0117608Z env: 2024-12-18T01:51:13.0117821Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:13.0118163Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:13.0118715Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:13.0119203Z ##[endgroup] 2024-12-18T01:51:13.3908915Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:51:13.3909499Z With the provided path, there will be 1 file uploaded 2024-12-18T01:51:13.3909963Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:13.3967734Z Starting upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:13.5126841Z Finished upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:13.5412183Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:51:13.5412529Z with: 2024-12-18T01:51:13.5412747Z s3-bucket: gha-artifacts 2024-12-18T01:51:13.5413083Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:13.5413436Z retention-days: 14 2024-12-18T01:51:13.5413705Z if-no-files-found: error 2024-12-18T01:51:13.5413996Z path: test-reports-*.zip 2024-12-18T01:51:13.5414264Z name: artifact 2024-12-18T01:51:13.5414505Z region: us-east-1 2024-12-18T01:51:13.5414740Z env: 2024-12-18T01:51:13.5414957Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:13.5415674Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:13.5416243Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:13.5416737Z ##[endgroup] 2024-12-18T01:51:13.8685785Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:51:13.8686393Z With the provided path, there will be 1 file uploaded 2024-12-18T01:51:13.8686999Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:13.8744727Z Starting upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:13.9777014Z Finished upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:14.0060281Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:51:14.0060624Z with: 2024-12-18T01:51:14.0060843Z s3-bucket: gha-artifacts 2024-12-18T01:51:14.0061172Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:14.0061555Z retention-days: 14 2024-12-18T01:51:14.0061823Z if-no-files-found: ignore 2024-12-18T01:51:14.0062106Z path: logs-*.zip 2024-12-18T01:51:14.0062339Z name: artifact 2024-12-18T01:51:14.0062575Z region: us-east-1 2024-12-18T01:51:14.0062812Z env: 2024-12-18T01:51:14.0063031Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:14.0063378Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:14.0063936Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:14.0064428Z ##[endgroup] 2024-12-18T01:51:14.3332997Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:51:14.3333872Z With the provided path, there will be 1 file uploaded 2024-12-18T01:51:14.3334757Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:14.3391009Z Starting upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:14.5288311Z Finished upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_34567152522.zip 2024-12-18T01:51:14.5575908Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:51:14.5576249Z with: 2024-12-18T01:51:14.5576469Z s3-bucket: gha-artifacts 2024-12-18T01:51:14.5576805Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:51:14.5577160Z retention-days: 14 2024-12-18T01:51:14.5577426Z if-no-files-found: ignore 2024-12-18T01:51:14.5577708Z path: debug-*.zip 2024-12-18T01:51:14.5577944Z name: artifact 2024-12-18T01:51:14.5578178Z region: us-east-1 2024-12-18T01:51:14.5578412Z env: 2024-12-18T01:51:14.5578633Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:14.5578977Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:14.5579525Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:14.5580018Z ##[endgroup] 2024-12-18T01:51:14.8777899Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-12-18T01:51:14.9090953Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T01:51:14.9091339Z # shellcheck disable=SC2156 2024-12-18T01:51:14.9091907Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T01:51:14.9102107Z shell: /usr/bin/bash -e {0} 2024-12-18T01:51:14.9102388Z env: 2024-12-18T01:51:14.9102616Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:14.9102973Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:14.9103542Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:14.9104034Z ##[endgroup] 2024-12-18T01:51:15.1823115Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6 2024-12-18T01:51:15.1823586Z with: 2024-12-18T01:51:15.1823798Z env: 2024-12-18T01:51:15.1824010Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:15.1824360Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:15.1824924Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:15.1825610Z ##[endgroup] 2024-12-18T01:51:15.1845947Z ##[group]Run set -eou pipefail 2024-12-18T01:51:15.1846276Z set -eou pipefail 2024-12-18T01:51:15.1846555Z  2024-12-18T01:51:15.1846938Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-12-18T01:51:15.1847387Z for _ in $(seq 1440); do 2024-12-18T01:51:15.1847731Z  # Break if no ssh session exists anymore 2024-12-18T01:51:15.1848084Z  if [ "$(who)" = "" ]; then 2024-12-18T01:51:15.1848390Z  break 2024-12-18T01:51:15.1848662Z  fi 2024-12-18T01:51:15.1848896Z  echo "." 2024-12-18T01:51:15.1849144Z  sleep 5 2024-12-18T01:51:15.1849391Z done 2024-12-18T01:51:15.1858008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:15.1858389Z env: 2024-12-18T01:51:15.1858615Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:15.1858975Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:15.1859530Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:15.1860028Z ##[endgroup] 2024-12-18T01:51:15.1887593Z Holding runner for 2 hours until all ssh sessions have logged out 2024-12-18T01:51:15.1977384Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T01:51:15.1977944Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T01:51:15.1978373Z # shellcheck disable=SC2046 2024-12-18T01:51:15.1978730Z docker stop $(docker ps -q) || true 2024-12-18T01:51:15.1979089Z # Prune all of the docker images 2024-12-18T01:51:15.1979431Z docker system prune -af 2024-12-18T01:51:15.1988085Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:15.1988572Z env: 2024-12-18T01:51:15.1988798Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:51:15.1989149Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:51:15.1989706Z DOCKER_CONTAINER_ID: 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:15.1990202Z ##[endgroup] 2024-12-18T01:51:15.8075409Z 2fd5f2f4acae 2024-12-18T01:51:16.6029279Z Deleted Containers: 2024-12-18T01:51:16.6029720Z 2fd5f2f4acae129b745c788d7531cfec98fa87d404d43021896127d727c48f99 2024-12-18T01:51:16.6030053Z 2024-12-18T01:51:28.5730286Z Deleted Images: 2024-12-18T01:51:28.5731408Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T01:51:28.5733023Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb 2024-12-18T01:51:28.5734187Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b 2024-12-18T01:51:28.5734851Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8 2024-12-18T01:51:28.5735516Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12 2024-12-18T01:51:28.5736166Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7 2024-12-18T01:51:28.5736846Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8 2024-12-18T01:51:28.5737475Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1 2024-12-18T01:51:28.5738401Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472 2024-12-18T01:51:28.5739022Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d 2024-12-18T01:51:28.5739635Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11 2024-12-18T01:51:28.5740280Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b 2024-12-18T01:51:28.5741189Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d 2024-12-18T01:51:28.5741817Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d 2024-12-18T01:51:28.5742439Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99 2024-12-18T01:51:28.5743070Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f 2024-12-18T01:51:28.5743712Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93 2024-12-18T01:51:28.5744346Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b 2024-12-18T01:51:28.5745071Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116 2024-12-18T01:51:28.5745738Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0 2024-12-18T01:51:28.5746601Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28 2024-12-18T01:51:28.5747432Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497 2024-12-18T01:51:28.5748292Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f 2024-12-18T01:51:28.5748906Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d 2024-12-18T01:51:28.5749527Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4 2024-12-18T01:51:28.5750141Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036 2024-12-18T01:51:28.5750766Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83 2024-12-18T01:51:28.5751381Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881 2024-12-18T01:51:28.5752001Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f 2024-12-18T01:51:28.5752626Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29 2024-12-18T01:51:28.5753237Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5 2024-12-18T01:51:28.5753855Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e 2024-12-18T01:51:28.5754468Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f 2024-12-18T01:51:28.5755100Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9 2024-12-18T01:51:28.5755729Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151 2024-12-18T01:51:28.5756344Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a 2024-12-18T01:51:28.5756970Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce 2024-12-18T01:51:28.5757598Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444 2024-12-18T01:51:28.5758214Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af 2024-12-18T01:51:28.5758826Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde 2024-12-18T01:51:28.5759432Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479 2024-12-18T01:51:28.5760044Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193 2024-12-18T01:51:28.5760674Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4 2024-12-18T01:51:28.5761303Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f 2024-12-18T01:51:28.5761975Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a 2024-12-18T01:51:28.5762596Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908 2024-12-18T01:51:28.5763380Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c 2024-12-18T01:51:28.5764011Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0 2024-12-18T01:51:28.5764635Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830 2024-12-18T01:51:28.5765265Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2 2024-12-18T01:51:28.5765977Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85 2024-12-18T01:51:28.5766592Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb 2024-12-18T01:51:28.5767218Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3 2024-12-18T01:51:28.5767835Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d 2024-12-18T01:51:28.5768447Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9 2024-12-18T01:51:28.5769061Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f 2024-12-18T01:51:28.5769681Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51 2024-12-18T01:51:28.5770302Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9 2024-12-18T01:51:28.5770925Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e 2024-12-18T01:51:28.5771547Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b 2024-12-18T01:51:28.5772164Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef 2024-12-18T01:51:28.5772780Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7 2024-12-18T01:51:28.5773399Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845 2024-12-18T01:51:28.5774026Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23 2024-12-18T01:51:28.5774648Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a 2024-12-18T01:51:28.5775260Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8 2024-12-18T01:51:28.5775882Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f 2024-12-18T01:51:28.5776506Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec 2024-12-18T01:51:28.5777135Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1 2024-12-18T01:51:28.5777766Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab 2024-12-18T01:51:28.5778391Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b 2024-12-18T01:51:28.5779006Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a 2024-12-18T01:51:28.5779634Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a 2024-12-18T01:51:28.5780257Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac 2024-12-18T01:51:28.5780897Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268 2024-12-18T01:51:28.5781525Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a 2024-12-18T01:51:28.5782153Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25 2024-12-18T01:51:28.5782785Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b 2024-12-18T01:51:28.5783426Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da 2024-12-18T01:51:28.5784060Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6 2024-12-18T01:51:28.5784692Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0 2024-12-18T01:51:28.5785326Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2 2024-12-18T01:51:28.5785962Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c 2024-12-18T01:51:28.5786681Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e 2024-12-18T01:51:28.5787319Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82 2024-12-18T01:51:28.5787697Z 2024-12-18T01:51:28.5787828Z Total reclaimed space: 42.91GB 2024-12-18T01:51:28.5850769Z Post job cleanup. 2024-12-18T01:51:28.5893517Z Post job cleanup. 2024-12-18T01:51:28.6704549Z [command]/usr/bin/git version 2024-12-18T01:51:28.6744693Z git version 2.40.1 2024-12-18T01:51:28.6784654Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ecf2f141-aca3-467d-9ef6-c082757f65df' before making global git config changes 2024-12-18T01:51:28.6785588Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T01:51:28.6789192Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T01:51:28.6823088Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T01:51:28.6852783Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T01:51:28.7215037Z Entering 'android/libs/fbjni' 2024-12-18T01:51:28.7282149Z Entering 'third_party/FP16' 2024-12-18T01:51:28.7349788Z Entering 'third_party/FXdiv' 2024-12-18T01:51:28.7417278Z Entering 'third_party/NNPACK' 2024-12-18T01:51:28.7486868Z Entering 'third_party/NVTX' 2024-12-18T01:51:28.7556835Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T01:51:28.7626076Z Entering 'third_party/XNNPACK' 2024-12-18T01:51:28.7707622Z Entering 'third_party/benchmark' 2024-12-18T01:51:28.7775768Z Entering 'third_party/composable_kernel' 2024-12-18T01:51:28.7849699Z Entering 'third_party/cpp-httplib' 2024-12-18T01:51:28.7917209Z Entering 'third_party/cpuinfo' 2024-12-18T01:51:28.7984584Z Entering 'third_party/cudnn_frontend' 2024-12-18T01:51:28.8053051Z Entering 'third_party/cutlass' 2024-12-18T01:51:28.8128108Z Entering 'third_party/eigen' 2024-12-18T01:51:28.8197179Z Entering 'third_party/fbgemm' 2024-12-18T01:51:28.8264716Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T01:51:28.8332135Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T01:51:28.8398007Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T01:51:28.8469874Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T01:51:28.8538289Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T01:51:28.8611006Z Entering 'third_party/flatbuffers' 2024-12-18T01:51:28.8681869Z Entering 'third_party/fmt' 2024-12-18T01:51:28.8750897Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T01:51:28.8819797Z Entering 'third_party/gloo' 2024-12-18T01:51:28.8888053Z Entering 'third_party/googletest' 2024-12-18T01:51:28.8957291Z Entering 'third_party/ideep' 2024-12-18T01:51:28.9023934Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T01:51:28.9098220Z Entering 'third_party/ittapi' 2024-12-18T01:51:28.9167622Z Entering 'third_party/kineto' 2024-12-18T01:51:28.9234785Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T01:51:28.9299859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T01:51:28.9368497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T01:51:28.9433813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T01:51:28.9500773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T01:51:28.9567449Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T01:51:28.9638654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T01:51:28.9705950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T01:51:28.9773372Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T01:51:28.9841679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T01:51:28.9911874Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T01:51:28.9981252Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T01:51:29.0052754Z Entering 'third_party/mimalloc' 2024-12-18T01:51:29.0120486Z Entering 'third_party/nccl/nccl' 2024-12-18T01:51:29.0188339Z Entering 'third_party/nlohmann' 2024-12-18T01:51:29.0257013Z Entering 'third_party/onnx' 2024-12-18T01:51:29.0339577Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T01:51:29.0412119Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T01:51:29.0479228Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T01:51:29.0544726Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T01:51:29.0615304Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T01:51:29.0683187Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T01:51:29.0752124Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T01:51:29.0818887Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T01:51:29.0885440Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T01:51:29.0952032Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T01:51:29.1021195Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T01:51:29.1091094Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T01:51:29.1183281Z Entering 'third_party/pocketfft' 2024-12-18T01:51:29.1253776Z Entering 'third_party/protobuf' 2024-12-18T01:51:29.1324277Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T01:51:29.1397698Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T01:51:29.1472315Z Entering 'third_party/psimd' 2024-12-18T01:51:29.1541522Z Entering 'third_party/pthreadpool' 2024-12-18T01:51:29.1610390Z Entering 'third_party/pybind11' 2024-12-18T01:51:29.1678697Z Entering 'third_party/python-peachpy' 2024-12-18T01:51:29.1747346Z Entering 'third_party/sleef' 2024-12-18T01:51:29.1819634Z Entering 'third_party/tensorpipe' 2024-12-18T01:51:29.1886266Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T01:51:29.1952414Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T01:51:29.2017524Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T01:51:29.2083632Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T01:51:29.2146709Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T01:51:29.2242876Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T01:51:29.2264139Z http.https://github.com/.extraheader 2024-12-18T01:51:29.2273683Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T01:51:29.2304621Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T01:51:29.2657803Z Entering 'android/libs/fbjni' 2024-12-18T01:51:29.2704000Z http.https://github.com/.extraheader 2024-12-18T01:51:29.2747845Z Entering 'third_party/FP16' 2024-12-18T01:51:29.2791641Z http.https://github.com/.extraheader 2024-12-18T01:51:29.2835560Z Entering 'third_party/FXdiv' 2024-12-18T01:51:29.2879700Z http.https://github.com/.extraheader 2024-12-18T01:51:29.2923663Z Entering 'third_party/NNPACK' 2024-12-18T01:51:29.2967289Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3010960Z Entering 'third_party/NVTX' 2024-12-18T01:51:29.3054355Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3098267Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T01:51:29.3141336Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3184732Z Entering 'third_party/XNNPACK' 2024-12-18T01:51:29.3229594Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3287789Z Entering 'third_party/benchmark' 2024-12-18T01:51:29.3332490Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3376247Z Entering 'third_party/composable_kernel' 2024-12-18T01:51:29.3421246Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3471212Z Entering 'third_party/cpp-httplib' 2024-12-18T01:51:29.3514660Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3558647Z Entering 'third_party/cpuinfo' 2024-12-18T01:51:29.3603244Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3647305Z Entering 'third_party/cudnn_frontend' 2024-12-18T01:51:29.3691395Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3735747Z Entering 'third_party/cutlass' 2024-12-18T01:51:29.3779289Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3830586Z Entering 'third_party/eigen' 2024-12-18T01:51:29.3873568Z http.https://github.com/.extraheader 2024-12-18T01:51:29.3920966Z Entering 'third_party/fbgemm' 2024-12-18T01:51:29.3968592Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4011508Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T01:51:29.4054637Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4097591Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T01:51:29.4140714Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4183136Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T01:51:29.4226075Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4275809Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T01:51:29.4318837Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4360737Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T01:51:29.4404185Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4448122Z Entering 'third_party/flatbuffers' 2024-12-18T01:51:29.4492567Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4538471Z Entering 'third_party/fmt' 2024-12-18T01:51:29.4582341Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4626261Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T01:51:29.4673523Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4716717Z Entering 'third_party/gloo' 2024-12-18T01:51:29.4760083Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4804342Z Entering 'third_party/googletest' 2024-12-18T01:51:29.4847865Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4891969Z Entering 'third_party/ideep' 2024-12-18T01:51:29.4938687Z http.https://github.com/.extraheader 2024-12-18T01:51:29.4980778Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T01:51:29.5023956Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5075365Z Entering 'third_party/ittapi' 2024-12-18T01:51:29.5118710Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5161144Z Entering 'third_party/kineto' 2024-12-18T01:51:29.5206970Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5249482Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T01:51:29.5291886Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5335002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T01:51:29.5386043Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5431497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T01:51:29.5474768Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5518430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T01:51:29.5560460Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5604311Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T01:51:29.5646552Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5688265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T01:51:29.5732591Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5782093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T01:51:29.5826380Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5869726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T01:51:29.5914043Z http.https://github.com/.extraheader 2024-12-18T01:51:29.5957892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T01:51:29.6000224Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6044530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T01:51:29.6086765Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6133970Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T01:51:29.6176702Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6220497Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T01:51:29.6262595Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6308931Z Entering 'third_party/mimalloc' 2024-12-18T01:51:29.6352403Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6394524Z Entering 'third_party/nccl/nccl' 2024-12-18T01:51:29.6440224Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6483955Z Entering 'third_party/nlohmann' 2024-12-18T01:51:29.6527140Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6571548Z Entering 'third_party/onnx' 2024-12-18T01:51:29.6614450Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6671316Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T01:51:29.6715054Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6763151Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T01:51:29.6807496Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6851584Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T01:51:29.6894221Z http.https://github.com/.extraheader 2024-12-18T01:51:29.6938602Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T01:51:29.6981050Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7023153Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T01:51:29.7064687Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7108649Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T01:51:29.7150207Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7193429Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T01:51:29.7235914Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7278608Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T01:51:29.7321805Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7363582Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T01:51:29.7405984Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7447030Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T01:51:29.7490437Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7535669Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T01:51:29.7578601Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7625798Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T01:51:29.7669667Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7733619Z Entering 'third_party/pocketfft' 2024-12-18T01:51:29.7777934Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7821763Z Entering 'third_party/protobuf' 2024-12-18T01:51:29.7865291Z http.https://github.com/.extraheader 2024-12-18T01:51:29.7914205Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T01:51:29.7957240Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8000571Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T01:51:29.8043906Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8090276Z Entering 'third_party/psimd' 2024-12-18T01:51:29.8134143Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8176624Z Entering 'third_party/pthreadpool' 2024-12-18T01:51:29.8222185Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8265868Z Entering 'third_party/pybind11' 2024-12-18T01:51:29.8314449Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8359361Z Entering 'third_party/python-peachpy' 2024-12-18T01:51:29.8403343Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8445740Z Entering 'third_party/sleef' 2024-12-18T01:51:29.8489679Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8533139Z Entering 'third_party/tensorpipe' 2024-12-18T01:51:29.8576526Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8620270Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T01:51:29.8662597Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8705512Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T01:51:29.8752205Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8794395Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T01:51:29.8842915Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8885415Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T01:51:29.8927872Z http.https://github.com/.extraheader 2024-12-18T01:51:29.8969025Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T01:51:29.9012241Z http.https://github.com/.extraheader 2024-12-18T01:51:29.9168815Z A job completed hook has been configured by the self-hosted runner administrator 2024-12-18T01:51:29.9195056Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-12-18T01:51:29.9203413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:51:29.9203798Z ##[endgroup] 2024-12-18T01:51:29.9301142Z [!ALERT!] Swap in detected! [!ALERT!] 2024-12-18T01:51:41.5057799Z [!ALERT!] Swap out detected [!ALERT!] 2024-12-18T01:52:00.0591022Z Cleaning up orphan processes