Repo organization and folders structure

[lucapinello]

Hi guys,

Given that a subteam is already working on the code-base and other people are working on notebooks it is a good moment to reach consensus on how to structure the repo.

I propose the following structure

dna-diffusion
docs
tests
notebooks

Within dna-diffusion we can have these subfolders:
data
models
metrics
utils

We can discuss more at the next Wed meeting on zoom but please write below your thoughts!

[IhabBendidi]

Sounds great ! Using the same structure, I thought of adding a few folders. For example for the general structure, we can have

dna-diffusion
docs
tests
notebooks
scripts
model-zoo

• "model-zoo" would host scripts to download all successful pretrained models that we have made, if any other person wants to use them or improve on them, or just to leave them for the community
• "scripts" would contain all the necessary scripts with all necessary hyperparameters to reproduce any single successful experiment

Within dna-diffusion, I propose the following subfolders :

data
models
losses
utils

• I took off "metrics" as a folder because in wip: dataloader first draft #17 it was discussed that having one single file for all metrics, "metrics.py" would be much better for code organization.
• Losses would contaain all the operations necessary for our loss functions

[mateibejan1]

P.S.: Additional "logs" and "datasets" directories will be created during train/test time.

[mateibejan1]

Hey @ssenan, pinging you about the dataloaders example. You can look at StableDiffusion, which split their dataloaders in to multiple classes, each for every dataset they use. This approach is called separation of concerns in engineering and the scope is to:

1. Simplify changes. Longer files are harder to keep track of, even through commit changes.
2. Isolate bugs.

[ssenan]

Awesome, I'll have a look into it and update the dataloader accordingly. Appreciate the tips

[LucasSilvaFerreira]

Hi! I created some restructuration in the main DNA-diffusion branch. This is the initial suggestion. @matei Bejan and others, please feel free to suggest a different structure, I know that your refactoring can modify this structure. This is just the initial idea, don't if you need to change it.
I added some simple information about the directories but it should be intuitive. Please respect the structure case by adding new files/dirs to the repository.