Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

layer_norm_grad for npu #10560

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

layer_norm_grad for npu #10560

wants to merge 2 commits into from

Commits on Nov 7, 2024

  1. layer_norm_grad npu

    @fpzh2011
    fpzh2011 committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    df98f12 View commit details
    Browse the repository at this point in the history

Commits on Nov 11, 2024

  1. fix layernorm grad sbp (#10561) 

    Fix SBP settings for LayerNormGradOp to ensure correct gradient
aggregation for gamma_diff and beta_diff

Changes

- Updated SBP strategy in LayerNormGradOp: Set gamma_diff and beta_diff
to use PartialSum instead of Split to avoid dimension mismatches during
distributed training.
- Added consistency check for begin_norm_axis and begin_params_axis:
Enforce equality to ensure proper alignment of normalization and
parameter dimensions.
    @ShawnXuan
    ShawnXuan authored Nov 11, 2024
    Configuration menu
    Copy the full SHA
    dfe78fb View commit details
    Browse the repository at this point in the history