Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding evaluations for the LLM AI implementation of ChatState #357

Conversation

mahomedalid
Copy link
Contributor

@mahomedalid mahomedalid commented May 8, 2024

Overview

This PR enhances the eShop reference sample by incorporating evaluations for the implementation of ChatState utilizing OpenAI's Large Language Model (LLM) AI. The goal is to assess the functionality of the ChatState feature, particularly focusing in coherence, groundedness and relevance.

Key Additions:

  1. Evaluations with xUnit Runner: Introduces evaluations for ChatState, leveraging xUnit as the test runner.
  2. Integration with Azure OpenAI or local models: Adds configuration to run the evaluations using local models or Azure OpenAI.
  3. GitHub Actions Workflow: Implements a GitHub Actions workflow to automate the testing process.

Notes:

  • Fix an issue on ChatState that when a kernel is reused and ChatState constructor is called multiple times (for example on multiple initializations) it will throw an error when trying to add multiple times the same plugins/functions.

@mahomedalid mahomedalid marked this pull request as ready for review May 11, 2024 05:10
Copy link
Contributor

@luisquintanilla luisquintanilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mahomedalid looks great.

Just had one question.

@@ -48,6 +49,15 @@ public ChatState(ICatalogService catalogService, IBasketState basketState, Claim
Messages.AddAssistantMessage("Hi! I'm the Northern Mountains Concierge. How can I help?");
}

private void ConfigureKernel()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mahomedalid what is this for?

Copy link
Contributor

@luisquintanilla luisquintanilla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@luisquintanilla luisquintanilla merged commit 3f02b78 into dotnet:may-2024-ai-updates May 16, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants