CosineSimilarityLoss value of self-supervised model training with byol

[Xenon1997]

When I was training self-supervised model training with byol, I found that the CosineSimilarityLoss value can be lower to -1 and remain stable around -3.

According to the definition of CosineSimilarityLoss and the document/source code，I thought the value should be with in [-1 , 1]. Could anyone tell me the reason?

"""Cosine similarity loss function.

Compute the similarity between two features and optimize that similarity as
loss.

Args:
    shift_factor (float): The shift factor of cosine similarity.
        Default: 0.0.
    scale_factor (float): The scale factor of cosine similarity.
        Default: 1.0.
"""

def __init__(self,
             shift_factor: float = 0.0,
             scale_factor: float = 1.0) -> None:
    super().__init__()
    self.shift_factor = shift_factor
    self.scale_factor = scale_factor

def forward(self,
            pred: torch.Tensor,
            target: torch.Tensor,
            mask: Optional[torch.Tensor] = None) -> torch.Tensor:
    """Forward function of cosine similarity loss.

    Args:
        pred (torch.Tensor): The predicted features.
        target (torch.Tensor): The target features.

    Returns:
        torch.Tensor: The cosine similarity loss.
    """
    pred_norm = nn.functional.normalize(pred, dim=-1)
    target_norm = nn.functional.normalize(target, dim=-1)
    loss = self.shift_factor - self.scale_factor * (
        pred_norm * target_norm).sum(dim=-1)

    if mask is None:
        loss = loss.mean()
    else:
        loss = (loss * mask).sum() / mask.sum()
    return loss

[fangyixiao18]

You could refer to https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/models/selfsup/byol.py#L85