Causal language model pipeline?

[GeorgeS2019]

I wonder if this discussion here is also relevant for TorchSharp?
Could this be part of TorchText?
@LittleLittleCloud

Add GenAI packages

The GenAI packages will provide torchsharp implementation for a series of popular GenAI models.

src/Microsoft.ML.GenAI.Core/Extension/CausalLMPipelineExtension.cs

What is a causal language model pipeline?

The causal language model pipeline is a utility class which wraps a tokenizer and a causal language model and provides a uniformed interface for various decoding method to generate text. The pipeline is designed to be easy to use and requires only a few lines of code to generate text.

In Microsoft.ML.GenAI, we will provide a generic CausalLMPipeline class plus a typed CausalLMPipeline class which specifies the type parameters for the tokenizer and the causal language model. The typed CausalLMPipeline class make it easier to develop consuming method for semantic kernel. see here for more details

Contract

public abstract class CausalLMPipeline
{
    public virtual (
        Tensor, // output token ids [batch_size, sequence_length]
        Tensor // output logits [batch_size, sequence_length, vocab_size]
    ) Generate(
        Tensor inputIds, // input token ids [batch_size, sequence_length]
        Tensor attentionMask, // attention mask [batch_size, sequence_length]
        float temperature = 0.7f,
        float topP = 0.9f,
        int maxLen = 128,
        int[][]? stopTokenSequence = null,
        bool echo = false); // echo the input token ids in the output token ids
}

public CasualLMPipeline<TTokenizer, TCausalLM> : CausalLMPipeline
    where TTokenizer : ITokenizer
    where TCausalLM : nn.Module<CausalLanguageModelInput, CausalLanguageModelOutput>
{
    public CausalLMPipeline<LLama2Tokenizer, Phi3ForCasualLM> Create(LLama2Tokenizer tokenizer, Phi3ForCasualLM model);

}

Usage

LLama2Tokenizer tokenizer;
Phi3ForCausalLM model;

var pipeline = CausalLMPipeline.Create(tokenizer, model);
var prompt = "Once upon a time";
// top-k sampling
var output = pipeline.Generate(
    prompt: prompt,
    maxLen: 100,
    temperature: 0.7f,
    topP: 0.9f,
    stopSequences: null,
    device: "cuda",
    bos: true, // add bos token to the prompt
    eos: false, // do not add eos token to the prompt
    echo: true // echo the prompt in the generated text
);

Sampling methods

The CausalLMPipeline provides a uniformed interface for various decoding methods to generate text. This saves our effort to implement different decoding methods for each model.

Sampling

public virtual (
        Tensor, // output token ids [batch_size, sequence_length]
        Tensor // output logits [batch_size, sequence_length, vocab_size]
    ) Generate(
        Tensor inputIds, // input token ids [batch_size, sequence_length]
        Tensor attentionMask, // attention mask [batch_size, sequence_length]
        float temperature = 0.7f,
        float topP = 0.9f,
        int maxLen = 128,
        int[][]? stopTokenSequence = null,
        bool echo = false); // echo the input token ids in the output token ids

Note

The Greedy search and beam search are not implemented in the pipeline yet. They will be added in the future.

Greedy Search

public (
    Tensor, // output token ids [batch_size, sequence_length]
    Tensor // output logits [batch_size, sequence_length, vocab_size]
) GreedySearch(
    Tensor inputIds, // input token ids [batch_size, sequence_length]
    Tensor attentionMask, // attention mask [batch_size, sequence_length]
    int maxLen = 128,
    int[][]? stopTokenSequence = null,
    bool echo = false); // echo the input token ids in the output token ids

Beam Search

public (
    Tensor, // output token ids [batch_size, sequence_length]
    Tensor // output logits [batch_size, sequence_length, vocab_size]
) BeamSearch(
    Tensor inputIds, // input token ids [batch_size, sequence_length]
    Tensor attentionMask, // attention mask [batch_size, sequence_length]
    int maxLen = 128,
    int[][]? stopTokenSequence = null,
    int beamSize = 5,
    float lengthPenalty = 1.0f,
    bool echo = false); // echo the input token ids in the output token ids

I

The extension method for CausalLMPipeline

The extension Generate method provides a even-easier way to generate text without the necessary to generate the input tensor. The method takes a prompt string and other optional parameters to generate text.

public static string Generate(
    this CasualLMPipeline pipeline,
    string prompt,
    int maxLen = 128,
    float temperature = 0.7f,
    float topP = 0.9f,
    string[]? stopSequences = null,
    string device = "cpu",
    bool bos = true,
    bool eos = false,
    bool echo = false)

[LittleLittleCloud]

Hey @GeorgeS2019 nice to meet you

For the TorchText are you refering to https://pytorch.org/text/stable/index.html?

[GeorgeS2019]

Yes, right now we have TorchVision, TorchAudio, but not TorchText.

Previously we could not proceed, because many utility class are only available in pytorch. With your work on Causal langauge model pipeline and others, many of these utility classes are now FINALLY available in TorchSharp

It is very important to give new users of Torchsharp that this framework provides complete libraries.
It would be great some thought is put into creating a basic framework of TorchText, especially with basic tutorials or examples provided by the official PyTorch.

This will finally position TorchSharp as the Deep Learning framework for .NET users, so all things, [audio, vision and text], can all stay within .NET and no need to venture out to python framework.

[GeorgeS2019]

An effort to cover all aspects of e.g. Introduction to Pytorch for Deep Learning will give Universities to consider using Torchsharp in teaching Deep Learning using c#