diff --git a/docs/_static/app-ui.png b/docs/_static/app-ui.png new file mode 100644 index 0000000..5ce3a24 Binary files /dev/null and b/docs/_static/app-ui.png differ diff --git a/docs/_static/sic-soc-llm.png b/docs/_static/sic-soc-llm.png new file mode 100644 index 0000000..0ba73fe Binary files /dev/null and b/docs/_static/sic-soc-llm.png differ diff --git a/docs/method.qmd b/docs/method.qmd index 4d7601f..359d70e 100644 --- a/docs/method.qmd +++ b/docs/method.qmd @@ -5,7 +5,7 @@ format: html ## Summary -A proof-of-concept large language model (LLM) application was created to assess whether an LLM could improve SIC autocoding performance for survey data. This was applied to sample of anonimized survey data and evaluated by comparing the results to clerical coding and to logistic regression model. The LLM showed marginal improvement over the logistic regression in the level of agreement with clerical coding at the 5-digit SIC level. It is likely that refinement of the method would improve performance further. Note that the evaluation scripts are out of scope for this repository. The methodology of the main SIC autocoding module is described bellow. For more information see Data science campus [blog](https://datasciencecampus.ons.gov.uk/category/projects/). +A proof-of-concept large language model (LLM) application was created to assess whether an LLM could improve SIC autocoding performance for survey data. This was applied to sample of anonimized survey data and evaluated by comparing the results to clerical coding and to logistic regression model. The LLM showed marginal improvement over the logistic regression in the level of agreement with clerical coding at the 5-digit SIC level. It is likely that refinement of the method would improve performance further. Note that the evaluation scripts are out of scope for this repository. The methodology of the main SIC autocoding module is described bellow. For more information see Data science campus [blog](https://datasciencecampus.ons.gov.uk/classifai-exploring-the-use-of-large-language-models-llms-to-assign-free-text-to-commonly-used-classifications/). ## RAG based classification diff --git a/docs/tutorials/3_soc_classifier.qmd b/docs/tutorials/3_soc_classifier.qmd index 8c6c71d..6749f4e 100644 --- a/docs/tutorials/3_soc_classifier.qmd +++ b/docs/tutorials/3_soc_classifier.qmd @@ -11,8 +11,6 @@ Demonstration notebook for the `ClassificationLLM` with Standard Occupational Cl ```{python} #| code-summary: "Code: Import methods and initialise" -from langchain.llms.fake import FakeListLLM - from sic_soc_llm import setup_logging from sic_soc_llm.llm import ClassificationLLM @@ -22,6 +20,7 @@ logger = setup_logging("soc_classifier") ```{python} #| echo: false #| code-summary: "Code: Create a fake Large Language Model (LLM) for demonstration purposes" +from langchain.llms.fake import FakeListLLM soc_demo_llm = FakeListLLM(responses=[ ''' diff --git a/pyproject.toml b/pyproject.toml index b4c232b..782d56f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -54,6 +54,7 @@ app = [ test = [ "pytest==6.2.5", "pytest-pythonpath==0.7.4", + "coverage==7.5.4", ] docs = ["quartodoc>=0.6.6",