Redesign registry; Allow publishing to registry (#86)

<!-- related issues, e.g. "will close #123" -->

## Description
<!-- describe in a few words the changes -->

This changes registry to from *one* file that is *manually updated* to
*multiple* files that are *auto-generated*. Now, the registry should be
only updated with the new `nupm publish` command.

See the [doc
page](https://github.com/nushell/nupm/blob/bde509cfd9a53939c7a4df01564136e1f9c0b2ee/docs/design/registry.md)
for more details and the `registry/` directory for the new proposed
structure.

### Misc changes

* The structure of the registry is also different. The git/local package
type is now encoded in a flat table instead of being a separate record.
See `registry/` and `tests/packages/registry/` for examples.
* The `path` field can be `null`, not requiring the "."

### TODO

- [x] Add package file hashing to stop redownloading them all the time

---------

Co-authored-by: Antoine Stevan <[email protected]>
Co-authored-by: amtoine <[email protected]>

docs/design/registry.md

@@ -0,0 +1,103 @@
+# Registry
+
+Registry is a collection of .nuon files that tell nupm where to look for packages when doing tasks like `nupm search` or `nupm install`.
+
+Two types of files compose a registry:
+* The "main" registry file containing the list of "registry package files"
+* Registry package files containing the details of each package.
+
+These files **should not** be edited manually. They are intended to be auto-generated and updated with `nupm publish` only. They also shouldn't contain any newlines to avoid potential problems with file hashes between Windows and non-Windows platforms.
+
+## "Main" registry file
+
+Table with one package per row and the following columns:
+* `name`: Name of the package
+* `path`: Path to the "registry package file", relative* to the main registry file. When looking up a package, nupm joins this path to the path/URL of the "main" registry file and fetches it. Both local paths and URLs are handled the same way.
+* `hash`: Hash of the "registry package file" to avoid re-downloading them all the time.
+
+The file is sorted by `name`. No duplicate package names allowed.
+
+## "Registry package file"
+
+These files contain the actual information about the package that is used do fetch and install the package. Multiple versions of the same package are supported. It has exactly the following columns:
+* `name`: Name of the package
+* `version`: Version of the package
+* `path`: Path where to look for nupm.nuon (relative to the package root*, in the case of git packages, or the main registry file, if local package)
+* `type`: Type of the package. Currently only "git" and "local"
+* `info`: Package-specific info based on `type`. It can be one of the following:
+  * `null` if `type` is "local"
+  * `record<url: string, revision: string>` if `type` is "git"
+
+This file is sorted by `version`. No duplicate versions allowed.
+
+_*absolute paths work, but are discouraged, only to be used for local testing etc._
+
+## Example registry structure
+
+_See the new `registry/` directory, the following example slightly differs from it._
+
+```
+./registry
+  +-- registry.nuon
+  +-- amtoine
+      +-- nu-git-manager.nuon
+      +-- nu-git-manager-sugar.nuon
+```
+
+```nushell
+> open registry/registry.nuon
+ #           name                         path                     hash
+───────────────────────────────────────────────────────────────────────────
+ 0   nu-git-manager         amtoine/nu-git-manager.nuon         md5-4aaae15412fb84233fcb19716f6b7e89
+ 1   nu-git-manager-sugar   amtoine/nu-git-manager-sugar.nuon   md5-d0c7641c0b369e7c944cc668741734d9
+
+> open amtoine/nu-git-manager.nuon | table -e
+ #        name        version          path           type                            info
+────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ 0   nu-git-manager   0.1.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.1.0
+ 1   nu-git-manager   0.2.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.2.0
+ 2   nu-git-manager   0.3.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.3.0
+ 3   nu-git-manager   0.4.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.4.0
+ 4   nu-git-manager   0.5.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.5.0
+ 5   nu-git-manager   0.6.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.6.0
+ 6   nu-git-manager   0.7.0     pkgs/nu-git-manager   git     url        https://github.com/amtoine/nu-git-manager
+                                                              revision   0.7.0
+
+> open amtoine/nu-git-manager-sugar.nuon | table -e
+ #           name           version             path              type                            info
+────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
+ 0   nu-git-manager-sugar   0.1.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.1.0
+ 1   nu-git-manager-sugar   0.2.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.2.0
+ 2   nu-git-manager-sugar   0.3.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.3.0
+ 3   nu-git-manager-sugar   0.4.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.4.0
+ 4   nu-git-manager-sugar   0.5.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.5.0
+ 5   nu-git-manager-sugar   0.6.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.6.0
+ 6   nu-git-manager-sugar   0.7.0     pkgs/nu-git-manager-sugar   git     url        https://github.com/amtoine/nu-git-manager
+                                                                          revision   0.7.0
+```
+
+## Publishing a package
+
+It is possible to only publish to a registry stored on your file system because we don't have a web service or anything like that.
+
+The intented workflow for publishing a package is:
+1. Check out the git repository with the registry
+2. `cd` into the package you want to publish
+3. Run `nupm publish chosen_registry` to preview the changes
+4. Repeat 3 by adjusting the `nupm publish` flags until you have the desired output
+5. Run the final command with the `--save` flag which will save the registry files
+6. Commit the changes to the registry, create a PR upstream, etc.
+
+The reason for steps 3. and 4. is that `nupm publish` tries to guess some values to make publishing less tedious. For example, if you're in a git repository, nupm tries to get the URL of the "origin", or the first available remote by default. This should be a sane default for most packages and frees you from having to pass the `--info` flag every time. The guess can be wrong, however, that's why you should check the output of step 3 and make the desired changes before saving the changes with `--save`.

nupm.nuon

@@ -1,7 +1,7 @@
 {
     name: nupm
     type: module
-    version: "0.1.0"
+    version: "0.2.0"
     description: "Nushell package manager"
     license: "LICENSE"
 }

nupm/install.nu

@@ -3,7 +3,7 @@ use std log
 use utils/completions.nu complete-registries
 use utils/dirs.nu [ nupm-home-prompt cache-dir module-dir script-dir tmp-dir ]
 use utils/log.nu throw-error
-use utils/misc.nu check-cols
+use utils/misc.nu [check-cols hash-fn url]
 use utils/package.nu open-package-file
 use utils/registry.nu search-package
 use utils/version.nu filter-by-version
@@ -130,10 +130,9 @@ def download-pkg [
     pkg: record<
         name: string,
         version: string,
-        url: string,
-        revision: string,
         path: string,
         type: string,
+        info: any,
     >
 ]: nothing -> path {
     # TODO: Add some kind of hashing to check that files really match
@@ -149,33 +148,37 @@ def download-pkg [
     mkdir $git_dir
     cd $git_dir
 
-    let repo_name = $pkg.url | url parse | get path | path parse | get stem
-    let url_hash = $pkg.url | hash md5 # in case of git repo name collision
-    let clone_dir = $'($repo_name)-($url_hash)-($pkg.revision)'
+    let repo_name = $pkg.info.url | url stem
+    let url_hash = $pkg.info.url | hash-fn # in case of git repo name collision
+    let clone_dir = $'($repo_name)-($url_hash)-($pkg.info.revision)'
 
-    let pkg_dir = $env.PWD | path join $clone_dir $pkg.path
+    let pkg_dir = if $pkg.path == null {
+        $env.PWD | path join $clone_dir
+    } else {
+        $env.PWD | path join $clone_dir ($pkg.path | path dirname)
+    }
 
     if ($pkg_dir | path exists) {
         print $'Package ($pkg.name) found in cache'
         return $pkg_dir
     }
 
     try {
-        git clone $pkg.url $clone_dir
+        git clone $pkg.info.url $clone_dir
     } catch {
-        throw-error $'Error cloning repository ($pkg.url)'
+        throw-error $'Error cloning repository ($pkg.info.url)'
     }
 
     cd $clone_dir
 
     try {
-        git checkout $pkg.revision
+        git checkout $pkg.info.revision
     } catch {
-        throw-error $'Error checking out revision ($pkg.revision)'
+        throw-error $'Error checking out revision ($pkg.info.revision)'
     }
 
     if not ($pkg_dir | path exists) {
-        throw-error $'Path ($pkg.path) does not exist'
+        throw-error $'Path ($pkg_dir) does not exist'
     }
 
     $pkg_dir
@@ -187,10 +190,7 @@ def fetch-package [
     --registry: string  # Which registry to use
     --version: string  # Package version to install (string or null)
 ]: nothing -> path {
-    let regs = (search-package $package
-        --registry $registry
-        --version $version
-        --exact-match)
+    let regs = search-package $package --registry $registry --exact-match
 
     if ($regs | is-empty) {
         throw-error $'Package ($package) not found in any registry'
@@ -216,7 +216,11 @@ def fetch-package [
     } else {
         # local package path is relative to the registry file (absolute paths
         # are discouraged but work)
-        $reg.path | path dirname | path join $pkg.path
+        if $pkg.path == null {
+            $reg.registry_path | path dirname
+        } else {
+            $reg.registry_path | path dirname | path join $pkg.path
+        }
     }
 }
 

nupm/mod.nu

@@ -4,9 +4,10 @@ use utils/dirs.nu [
 ]
 
 export module install.nu
-export module test.nu
+export module publish.nu
 export module search.nu
 export module status.nu
+export module test.nu
 
 export-env {
     # Ensure that $env.NUPM_HOME is always set when running nupm. Any missing