Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add methods to export results in tabular format #280

Open
wants to merge 34 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
6b8d3d3
add print links method to LinkGraph, improve LinkGraph string represe…
liannette Oct 15, 2024
cdd26c3
feat: add a method to print tabular results files
liannette Oct 16, 2024
ec8b8ae
improve method names and docstrings, remove unused method to export g…
liannette Oct 16, 2024
2207df1
improve doctring and typing
liannette Oct 16, 2024
c6e166a
fix a failing test
liannette Oct 16, 2024
32ca3dd
refactor a little bit the spectrum method to covert to dict
liannette Oct 16, 2024
8e7945d
change the output format for gnps_annotations in metabolomics results…
liannette Oct 16, 2024
2592810
fix: convert int to str before using join
liannette Oct 17, 2024
7f53de8
change representation of empty values in output files for improved in…
liannette Oct 17, 2024
ad049c8
refactoring the export methods
liannette Oct 17, 2024
b220fb0
small refactor: specify staticmethod
liannette Oct 18, 2024
f98fa98
add more tests
liannette Oct 18, 2024
a8a8329
correct typing in doctrings
liannette Oct 18, 2024
c6c33e6
typing: changed typings to pass mypy static typing checks
liannette Oct 22, 2024
a260338
refactor: change the order of methods/functions
liannette Oct 22, 2024
3289683
restore the order of already existing functions and methods
liannette Nov 4, 2024
d2272e2
make dicts json compatible
liannette Nov 4, 2024
cb49209
rename functions and variables
liannette Nov 4, 2024
6a4da5f
refactor: changed the place when the index is added to the link dict
liannette Nov 4, 2024
edcc7db
use csv package to write the tabular output files
liannette Nov 4, 2024
05f9f76
make sure all elements of the input list have the same type of data.
liannette Nov 4, 2024
bff7731
shorten to long doc string lines, correct some doc strings
liannette Nov 4, 2024
d4bf9fb
tests: adapted the test to the changes
liannette Nov 4, 2024
2c05efb
remove a file that was committed by accident
liannette Nov 4, 2024
229a11d
Merge branch 'NPLinker:dev' into output_files
liannette Nov 5, 2024
32d78c3
Improve docstrings
liannette Nov 19, 2024
b04226b
Improve docstrings
liannette Nov 19, 2024
5fd4108
refactor: add method to convert a value to string for tabular output
liannette Nov 19, 2024
8137f7d
Merge branch 'output_files' of https://github.com/liannette/nplinker …
liannette Nov 19, 2024
940eb19
improve doctring, add a comment about key order of bgc dict represent…
liannette Nov 20, 2024
e551dcc
move to_string method to the BGC/Spectrum class, add a to_tabular method
liannette Nov 20, 2024
f9ae9f2
add tests for the to_string method
liannette Nov 20, 2024
1b00262
change to_tabular to it returns a list and not a string
liannette Nov 20, 2024
0d6bec3
refactor: to_tabular returns dict, to_string turned into private func…
liannette Nov 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/format-typing-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
- name: Install ruff and mypy
run: |
pip install ruff mypy typing_extensions \
types-Deprecated types-beautifulsoup4 types-jsonschema types-networkx pandas-stubs
types-Deprecated types-beautifulsoup4 types-jsonschema types-networkx types-tabulate pandas-stubs
- name: Get all changed python files
id: changed-python-files
uses: tj-actions/changed-files@v44
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ dev = [
"types-beautifulsoup4",
"types-jsonschema",
"types-networkx",
"types-tabulate",
"pandas-stubs",
# docs
"black",
Expand Down
31 changes: 31 additions & 0 deletions src/nplinker/genomics/bgc.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
from typing import Any
from deprecated import deprecated
from nplinker.strain import Strain
from .aa_pred import predict_aa
Expand Down Expand Up @@ -173,6 +174,36 @@ def is_mibig(self) -> bool:
"""
return self.id.startswith("BGC")

def to_dict(self) -> dict[str, Any]:
liannette marked this conversation as resolved.
Show resolved Hide resolved
"""Convert the BGC object to a dictionary for exporting results.
liannette marked this conversation as resolved.
Show resolved Hide resolved

Compiles relevant information from the BGC object and formats it into a dictionary.
Each key-value pair in the dictionary represents a specific attribute of the BGC.

liannette marked this conversation as resolved.
Show resolved Hide resolved
Returns:
A dictionary containing the following key-value pairs:
liannette marked this conversation as resolved.
Show resolved Hide resolved
- GCF_id (list[str]): A list of GCF IDs.
- GCF_bigscape_class (list[str | None]): A list of BiG-SCAPE classes.
liannette marked this conversation as resolved.
Show resolved Hide resolved
- strain_id (str | None): The ID of the strain.
- description (str | None): A description of the BGC.
- BGC_name (str): The name of the BGC.
- product_prediction (list[str]): (predicted) products or product classes of the BGC.
- mibig_bgc_class (list[str] | None): MIBiG biosynthetic classes.
- antismash_id (str | None): The antiSMASH ID.
- antismash_region (int | None): The antiSMASH region number.
"""
return {
"GCF_id": [gcf.id for gcf in self.parents if gcf.id is not None],
"GCF_bigscape_class": [bsc for bsc in self.bigscape_classes if bsc is not None],
"strain_id": self.strain.id if self.strain is not None else None,
"description": self.description,
"BGC_name": self.id,
"product_prediction": list(self.product_prediction),
"mibig_bgc_class": self.mibig_bgc_class,
"antismash_id": self.antismash_id,
"antismash_region": self.antismash_region,
}
liannette marked this conversation as resolved.
Show resolved Hide resolved

# CG: why not providing whole product but only amino acid as product monomer?
# this property is not used in NPLinker core business.
@property
Expand Down
27 changes: 27 additions & 0 deletions src/nplinker/metabolomics/spectrum.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the docstrings and comments based on my comments in bgc.py

Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
from __future__ import annotations
from functools import cached_property
from typing import TYPE_CHECKING
from typing import Any
import numpy as np
from nplinker.strain import Strain
from nplinker.strain import StrainCollection
Expand Down Expand Up @@ -97,3 +98,29 @@ def has_strain(self, strain: Strain) -> bool:
True when the given strain exist in the spectrum.
"""
return strain in self.strains

def to_dict(self) -> dict[str, Any]:
"""Convert the Spectrum object to a dictionary for exporting results.
liannette marked this conversation as resolved.
Show resolved Hide resolved

This method compiles relevant information from the Spectrum object into a dictionary format.
Each key-value pair in the dictionary represents a specific attribute of the Spectrum Object.

liannette marked this conversation as resolved.
Show resolved Hide resolved
Returns:
A dictionary containing containing the following key-value pairs:
liannette marked this conversation as resolved.
Show resolved Hide resolved
- "spectrum_id" (str): The unique identifier of the spectrum.
- "num_strains_with_spectrum" (int): The number of strains associated with the spectrum.
- "precursor_mz" (float): The precursor m/z value, rounded to four decimal places.
- "rt" (float): The retention time, rounded to three decimal places.
- "molecular_family" (str | None ): The identifier of the molecular family.
- "gnps_id" (str | None ): The GNPS identifier.
- "gnps_annotations" (dict[str, str]): A dictionary of GNPS annotations.
"""
return {
"spectrum_id": self.id,
"num_strains_with_spectrum": len(self.strains),
"precursor_mz": round(self.precursor_mz, 4),
"rt": round(self.rt, 3),
"molecular_family": self.family.id if self.family else None,
"gnps_id": self.gnps_id,
"gnps_annotations": self.gnps_annotations,
}
53 changes: 53 additions & 0 deletions src/nplinker/nplinker.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add unit tests for the new public methods objects_to_tsv and to_tsv in test_nplinker_local.py.

Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from __future__ import annotations
import csv
import logging
import pickle
from collections.abc import Sequence
Expand Down Expand Up @@ -355,3 +356,55 @@ def save_data(
data = (self.bgcs, self.gcfs, self.spectra, self.mfs, self.strains, links)
with open(file, "wb") as f:
pickle.dump(data, f)

def objects_to_tsv(self, objects: Sequence[BGC] | Sequence[Spectrum], filename: str) -> None:
"""Exports a list of BGC or Spectrum objects to a specified file in tab-separated format.
liannette marked this conversation as resolved.
Show resolved Hide resolved

Args:
objects (list): A list of BGC or a list of Spectrum objects to be exported.
filename (str): The name of the file where the data will be saved.
liannette marked this conversation as resolved.
Show resolved Hide resolved
"""
if not objects:
raise ValueError("No objects provided to export")

# Ensure all elements in the list are of the same type
obj_type = type(objects[0])
if not all(isinstance(obj, obj_type) for obj in objects):
raise TypeError("All objects in the list must be of the same type")

headers = objects[0].to_dict().keys()
with open(self._output_dir / filename, "w", newline="") as outfile:
CunliangGeng marked this conversation as resolved.
Show resolved Hide resolved
writer = csv.DictWriter(outfile, fieldnames=headers, delimiter="\t")
writer.writeheader()
for obj in objects:
row = obj.to_dict()
for header in headers:
value = row[header]
# Convert list, tuple, set to comma-separated string
if isinstance(value, (list, tuple, set)):
row[header] = ", ".join(map(str, value))
# Convert dict to comma-separated string
elif isinstance(value, dict):
row[header] = ", ".join([f"{k}:{v}" for k, v in value.items()])
# Convert anything else to string
else:
row[header] = str(value) if value else ""
# Replace tabs with 4 spaces
row[header] = row[header].replace("\t", " ")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part could be extracted to a to_tabular method with some modifications for both BGC and Spectrum classes.

Then you could use pandas dataframe to faciliate the process of a list of objects.

Copy link
Contributor Author

@liannette liannette Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved this part into a to_string method. Let me know if that's okay as well.

writer.writerow(row)

def to_tsv(self, lg: LinkGraph | None = None) -> None:
"""Exports the results to the output directory in tab-separated format.
liannette marked this conversation as resolved.
Show resolved Hide resolved

This method exports genomics and metabolomics data to their respective
TSV files in the specified output directory. If a LinkGraph object is
provided, it also exports the links data to a TSV file.
liannette marked this conversation as resolved.
Show resolved Hide resolved

Args:
lg (LinkGraph | None): An optional LinkGraph object. If provided,
the links data will be exported to 'links.tsv'.
"""
self.objects_to_tsv(self.bgcs, "genomics_data.tsv")
self.objects_to_tsv(self.spectra, "metabolomics_data.tsv")
if lg is not None:
lg.to_tsv(self._output_dir / "links.tsv")
132 changes: 103 additions & 29 deletions src/nplinker/scoring/link_graph.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
from __future__ import annotations
import csv
from collections.abc import Sequence
from functools import wraps
from os import PathLike
from typing import Any
from typing import Union
from networkx import Graph
from tabulate import tabulate
Expand Down Expand Up @@ -76,17 +79,17 @@ def __init__(self) -> None:

Display the empty LinkGraph object:
>>> lg
| | Object 1 | Object 2 | Metcalf Score | Rosetta Score |
|----|------------|------------|-----------------|-----------------|
| index | genomic_object_id | genomic_object_type | metabolomic_object_id | metabolomic_object_type | metcalf_score | rosetta_score |
|---------|---------------------|-----------------------|-------------------------|---------------------------|-----------------|-----------------|

Add a link between a GCF and a Spectrum object:
>>> lg.add_link(gcf, spectrum, metcalf=Score("metcalf", 1.0, {"cutoff": 0.5}))

Display all links in LinkGraph object:
>>> lg
| | Object 1 | Object 2 | Metcalf Score | Rosetta Score |
|----|--------------|------------------------|-----------------|-----------------|
| 1 | GCF(id=gcf1) | Spectrum(id=spectrum1) | 1 | - |
| index | genomic_object_id | genomic_object_type | metabolomic_object_id | metabolomic_object_type | metcalf_score | rosetta_score |
|---------|---------------------|-----------------------|-------------------------|---------------------------|-----------------|-----------------|
| 1 | 1 | GCF | 1 | Spectrum | 1.00 | |

Get all links for a given object:
>>> lg[gcf]
Expand All @@ -103,6 +106,18 @@ def __init__(self) -> None:
Get the link data between two objects:
>>> lg.get_link_data(gcf, spectrum)
{"metcalf": Score("metcalf", 1.0, {"cutoff": 0.5})}

Filter the links for `gcf1` and `gcf2`:
>>> new_lg = lg.filter([gcf1, gcf2])

Filter the links for `spectrum1` and `spectrum2`:
>>> new_lg = lg.filter([spectrum1, spectrum2])

Filter the links between two lists of objects:
>>> new_lg = lg.filter([gcf1, gcf2], [spectrum1, spectrum2])

Export the links to a file:
>>> lg.to_tsv("links.tsv")
"""
self._g: Graph = Graph()

Expand Down Expand Up @@ -267,6 +282,53 @@ def filter(self, u_nodes: Sequence[Entity], v_nodes: Sequence[Entity] = [], /) -

return lg

@staticmethod
def link_to_dict(link: LINK) -> dict[str, Any]:
"""Convert a link to a dictionary representation.

Args:
link: A tuple containing the link information (u, v, data).

Returns:
A dictionary containing the link information with the following keys:
liannette marked this conversation as resolved.
Show resolved Hide resolved
- genomic_object_id (str): The ID of the genomic object.
- genomic_object_type (str): The type of the genomic object.
- metabolomic_object_id (str): The ID of the metabolomic object.
- metabolomic_object_type (str): The type of the metabolomic object.
- metcalf_score (float | str): The Metcalf score, rounded to 2 decimal places.
- rosetta_score (float | str): The Rosetta score, rounded to 2 decimal places.
"""
u, v, data = link
genomic_types = (GCF,)
genomic_object = u if isinstance(u, genomic_types) else v
metabolomic_object = v if isinstance(u, genomic_types) else u
metcalf_score = data.get("metcalf")
rosetta_score = data.get("rosetta")
return {
liannette marked this conversation as resolved.
Show resolved Hide resolved
"genomic_object_id": genomic_object.id,
"genomic_object_type": genomic_object.__class__.__name__,
"metabolomic_object_id": metabolomic_object.id,
"metabolomic_object_type": metabolomic_object.__class__.__name__,
"metcalf_score": round(metcalf_score.value, 2) if metcalf_score else "",
"rosetta_score": round(rosetta_score.value, 2) if rosetta_score else "",
}

def to_tsv(self, file: str | PathLike) -> None:
"""Exports the links in the LinkGraph to a file in tab-separated format.
liannette marked this conversation as resolved.
Show resolved Hide resolved

Args:
file: the file to write the links to.
liannette marked this conversation as resolved.
Show resolved Hide resolved

Examples:
>>> lg.to_tsv("links.tsv")
"""
table_data = self._links_to_dicts()
headers = table_data[0].keys()
with open(file, "w", newline="") as f:
CunliangGeng marked this conversation as resolved.
Show resolved Hide resolved
writer = csv.DictWriter(f, fieldnames=headers, delimiter="\t")
writer.writeheader()
writer.writerows(table_data)

@validate_u
def _filter_one_node(self, u: Entity, lg: LinkGraph) -> None:
"""Filter the links for a given object and add them to the new LinkGraph object."""
Expand All @@ -285,35 +347,47 @@ def _filter_two_nodes(self, u: Entity, v: Entity, lg: LinkGraph) -> None:
if link_data is not None:
lg.add_link(u, v, **link_data)

def _get_table_repr(self) -> str:
def _get_table_repr(self, display_limit: int | None = 60) -> str:
"""Generate a table representation of the LinkGraph.

The table is truncated to 60 links.
"""
headers = ["", "Object 1", "Object 2", "Metcalf Score", "Rosetta Score"]
table_data = []
display_limit = 60
Args:
display_limit: The maximum number of links to display in the table. Defaults to 60.

for index, (u, v, data) in enumerate(self.links, start=1):
metcalf_score = data.get("metcalf")
rosetta_score = data.get("rosetta")
Returns:
str: A string representation of the table in GitHub-flavored markdown format. If the
liannette marked this conversation as resolved.
Show resolved Hide resolved
number of links exceeds the display limit, the table is truncated and an additional
line indicating the total number of links is appended.
"""
table = tabulate(
self._links_to_dicts(display_limit),
headers="keys",
tablefmt="github",
stralign="right",
)

if display_limit is not None and len(self.links) > display_limit:
truncated_info = f"...\n[ {len(self.links)} links ]"
table += f"\n{truncated_info}"

row = [
index,
str(u if isinstance(u, GCF) else v),
str(v if isinstance(u, GCF) else u),
f"{metcalf_score.value:.2f}" if metcalf_score else "-",
f"{rosetta_score.value:.2f}" if rosetta_score else "-",
]
table_data.append(row)
return table

if index == display_limit:
break
def _links_to_dicts(self, display_limit: int | None = None) -> list[dict[str, Any]]:
"""Generate the table data for the LinkGraph.

table = tabulate(table_data, headers=headers, tablefmt="github", stralign="right")
This method iterates over the links in the LinkGraph and constructs a table
containing information about genomic and metabolomic objects, as well as their
associated scores. Each row in the table represents a link between a genomic
object and a metabolomic object.

if len(self.links) > display_limit:
truncated_info = f"...\n[ {len(self.links)} links ]"
return f"{table}\n{truncated_info}"
Args:
display_limit (int | None): The maximum number of rows to include in the
table. If None, all rows are included.

return table
Returns:
A list of dictionaries containing the table data.
"""
links = self.links[:display_limit] if display_limit else self.links
link_dicts = []
for idx, link in enumerate(links):
link_dicts.append({"index": idx + 1, **self.link_to_dict(link)})
return link_dicts
34 changes: 34 additions & 0 deletions tests/unit/genomics/test_bgc.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's required to test public methods to_tabluar and optional to test private method _to_string. So please add unit tests for the public methods.

Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,37 @@ def test_add_and_detach_parent():
assert bgc.parents == {gcf}
bgc.detach_parent(gcf)
assert bgc.parents == set()


def test_to_dict():
bgc = BGC("BGC0000001", "Polyketide", "NRP")
bgc.strain = Strain("sample_strain")
bgc.description = "Sample description"

dict_repr = bgc.to_dict()
assert dict_repr["GCF_id"] == list()
assert dict_repr["GCF_bigscape_class"] == list()
assert dict_repr["BGC_name"] == "BGC0000001"
assert dict_repr["product_prediction"] == ["Polyketide", "NRP"]
assert dict_repr["mibig_bgc_class"] is None
assert dict_repr["description"] == "Sample description"
assert dict_repr["strain_id"] == "sample_strain"
assert dict_repr["antismash_id"] is None
assert dict_repr["antismash_region"] is None

bgc.add_parent(GCF("1"))
bgc.mibig_bgc_class = [
"NRP",
]
bgc.antismash_id = "ABC_0001"
bgc.antismash_region = 1
dict_repr = bgc.to_dict()
assert dict_repr["GCF_id"] == [
"1",
]
assert dict_repr["GCF_bigscape_class"] == list()
assert dict_repr["mibig_bgc_class"] == [
"NRP",
]
assert dict_repr["antismash_id"] == "ABC_0001"
assert dict_repr["antismash_region"] == 1
Loading