Research: bench of the llamacpp

[jumbo-q]

Research Stage

• Background Research (Let's try to avoid reinventing the wheel)
• Hypothesis Formed (How do you think this will work and it's effect?)
• Strategy / Implementation Forming
• Analysis of results
• Debrief / Documentation (So people in the future can learn from us)

Previous existing literature and research

theres two Questions:

Is the content of this batch input self-defined, similar to Some other Infer framework or is there a specific dataset for it? Or other operations?
The output time only provides the average and variance for each token. How is this time calculated? Is it the mean and variance over multiple runs? Also, what part of the execution is being timed? From which point to which point is the timing measured?

Hypothesis

No response

Implementation

No response

Analysis

No response

Relevant log output

No response