Super-fy the zq command doc (#5457)

brimdata · Nov 9, 2024 · efa3300 · efa3300
1 parent eaef611
commit efa3300
Show file tree

Hide file tree

Showing 16 changed files with 173 additions and 185 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,7 +7,7 @@
 * Update the [`grok` function docs](docs/language/functions/grok.md) with additional examples and guidance (#5243)
 * Update the [Lateral Subquery docs](docs/language/lateral-subqueries.md) with an emphasis on when primitive values or arrays are returned by [Lateral Expressions](docs/language/lateral-subqueries.md#lateral-expressions) (#5264)
 * The terms "pipeline" and "branch" are now used throughout the [Zed docs](docs/README.md) instead of "dataflow" and "leg" (#5272)
-* Add docs for [`lake` output format](docs/commands/zq.md#zed-lake-metadata-output) and [`zed ls`](docs/commands/zed.md#ls) (#5187)
+* Add docs for [`lake` output format](docs/commands/super.md#superdb-data-lake-metadata-output) and [`zed ls`](docs/commands/zed.md#ls) (#5187)
 * Add docs for the [`top` operator](docs/language/operators/top.md) (#5276)
 * Add [`fluentd` integration docs](docs/integrations/fluentd.md) (#5190, #5195)
 * Add a [`strftime` function](docs/language/functions/strftime.md) to format `time` values into strings (#5197, #5204)
@@ -435,7 +435,7 @@ questions.
 * Rework dataflow model and Zed compiler optimizations (#2669)
 * Add initial `explode` operator that can break values from complex fields out into separate records (#2673)
 * Fix an issue where including a particular `time`-typed field in a shaper script caused errors with shaping other fields (#2685)
-* Silently discard duplicate fields when reading NDJSON records, which works around [Suricata bug 4016](https://redmine.openinfosecfoundation.org/issues/4106) (#2691)
+* Silently discard duplicate fields when reading NDJSON records, which works around Suricata bug 4016 (#2691)
 * Fix an issue where ZSON type values were output without parentheses (#2700)
 * Swallow single-backslash-escaped `/` when reading NDJSON, which allows for reading default Suricata EVE output (#2697)
 * Improve the error message shown when no Zed lake root is specified (#2701, #2739)

diff --git a/docs/README.md b/docs/README.md
@@ -13,7 +13,7 @@ without giving up JSON's uncanny ability to represent eclectic data.
 ## Getting Started
 
 Trying out SuperDB is easy: just [install](install.md) the command-line tool
-[`super`](commands/zq.md) and run through the [tutorial](tutorials/zq.md).
+[`super`](commands/super.md) and run through the [tutorial](tutorials/zq.md).
 
 Compared to putting JSON data in a relational column, the
 [super-structured data model](formats/zed.md) makes it really easy to

diff --git a/docs/commands/README.md b/docs/commands/README.md
@@ -1,18 +1,14 @@
 # Command Tooling
 
-The Zed system is managed and queried with the [`zed` command](zed.md),
-which is organized into numerous subcommands like the familiar command patterns
-of `docker` or `kubectrl`.
-Built-in help for the `zed` command and all of its subcommands is always
-accessible with the `-h` flag.
+The [`super` command](super.md) is used to execute command-line queries on
+inputs from files, HTTP URLs, or [S3](../integrations/amazon-s3.md).
 
-The [`zq` command](zq.md) offers a convenient slice of `zed` for running
-stand-alone, command-line queries on inputs from files, HTTP URLs, or [S3](../integrations/amazon-s3.md).
-`zq` is like [`jq`](https://stedolan.github.io/jq/) but is easier and faster, utilizes the richer
-Zed data model, and interoperates with a number of other formats beyond JSON.
-If you don't need a Zed lake, you can install just the
-slimmer `zq` command which omits lake support and dev tools.
+The [`super db` sub-commands](zed.md) are for creating, configuring, ingesting
+into, querying, and orchestrating SuperDB data lakes. These sub-commands are
+organized into further subcommands like the familiar command patterns
+of `docker` or `kubectl`.
 
-`zq` is always installed alongside `zed`.  You might find yourself mixing and
-matching `zed` lake queries with `zq` local queries and stitching them
-all together with Unix pipelines.
+All operations with these commands utilize the [super data model](../formats/README.md)
+and provide querying via [SuperSQL](../language/README.md).
+
+Built-in help for `super` and all sub-commands is always accessible with the `-h` flag.
diff --git a/docs/commands/zq.md → docs/commands/super.md b/docs/commands/zq.md → docs/commands/super.md
diff --git a/docs/commands/zed.md b/docs/commands/zed.md
@@ -18,7 +18,7 @@ sidebar_label: zed
 <p id="status"></p>
 
 :::tip Status
-While [`zq`](zq.md) and the [Zed formats](../formats/README.md)
+While [`super`](super.md) and the [Zed formats](../formats/README.md)
 are production quality, the Zed lake is still fairly early in development
 and alpha quality.
 That said, Zed lakes can be utilized quite effectively at small scale,
@@ -341,9 +341,9 @@ for that sub-command.
 sub-command and so forth.
 
 By default, commands that display lake metadata (e.g., [`log`](#log) or
-[`ls`](#ls)) use the human-readable [lake metadata output](zq.md#zed-lake-metadata-output)
+[`ls`](#ls)) use the human-readable [lake metadata output](super.md#superdb-data-lake-metadata-output)
 format.  However, the `-f` option can be used to specify any supported
-[output format](zq.md#output-formats).
+[output format](super.md#output-formats).
 
 ### Auth
 ```
@@ -460,8 +460,8 @@ schema-agnostic fashion.  Data of any _shape_ can be stored in any pool
 and arbitrary data _shapes_ can coexist side by side.
 
 As with `zq`,
-the [input arguments](zq.md#usage) can be in
-any [supported format](zq.md#input-formats) and
+the [input arguments](super.md#usage) can be in
+any [supported format](super.md#input-formats) and
 the input format is auto-detected if `-i` is not provided.  Likewise,
 the inputs may be URLs, in which case, the `load` command streams
 the data from a Web server or [S3](../integrations/amazon-s3.md) and into the lake.
@@ -538,7 +538,7 @@ commit history.
 
 Since commit objects are stored as Zed, the metadata can easily be
 queried by running the `log -f bsup` to retrieve the log in ZNG format,
-for example, and using [`zq`](zq.md) to pull the metadata out
+for example, and using [`super`](super.md) to pull the metadata out
 as in:
 ```
 zed log -f bsup | zq 'has(meta) | yield {id,meta}' -

diff --git a/docs/formats/zjson.md b/docs/formats/zjson.md
@@ -269,7 +269,7 @@ and an array of union of string, and float64 --- might have a value that looks l
 
 A ZJSON file is composed of ZJSON objects formatted as
 [newline delimited JSON (NDJSON)](https://en.wikipedia.org/wiki/JSON_streaming#NDJSON).
-e.g., the [zq](../commands/zq.md) CLI command
+e.g., the [super](../commands/super.md) CLI command
 writes its ZJSON output as lines of NDJSON.
 
 ## 4. Example

diff --git a/docs/integrations/fluentd.md b/docs/integrations/fluentd.md
@@ -394,7 +394,7 @@ the size of the commit objects to which they're initially stored.
 
 2. **ZNG format** - In the [shaping example](#shaping-example) shown above, we
 used the [Super JSON format](../formats/jsup.md) format for the shaped data output from
-[`zq`](../commands/zq.md). This text format is typically used in contexts
+[`super`](../commands/super.md). This text format is typically used in contexts
 where human readability is required. Due to its compact nature,
 [Super Binary](../formats/bsup.md) format would have been preferred, but in our research
 we found Fluentd consistently steered us toward using only text formats.

diff --git a/docs/integrations/zeek/data-type-compatibility.md b/docs/integrations/zeek/data-type-compatibility.md
@@ -64,7 +64,7 @@ there is no authoritative specification of the Zeek TSV log format.
 ## Example
 
 The following example shows a TSV log that includes each Zeek data type, how
-it's output as Super JSON by [`zq`](../../commands/zq.md), and then how it's written back out again as a Zeek
+it's output as Super JSON by [`super`](../../commands/super.md), and then how it's written back out again as a Zeek
 log. You may find it helpful to refer to this example when reading the
 [type-specific details](#type-specific-details).
 

diff --git a/docs/integrations/zeek/reading-zeek-log-formats.md b/docs/integrations/zeek/reading-zeek-log-formats.md
@@ -14,7 +14,7 @@ the Zed [command line tools](../../commands/README.md).
 [Zeek TSV](https://docs.zeek.org/en/master/log-formats.html#zeek-tsv-format-logs)
 is Zeek's default output format for logs. This format can be read automatically
 (i.e., no `-i` command line flag is necessary to indicate the input format)
-with the Zed tools such as [`zq`](../../commands/zq.md).
+with the Zed tools such as [`super`](../../commands/super.md).
 
 The following example shows a TSV [`conn.log`](https://docs.zeek.org/en/master/logs/conn.html) being read via `zq` and
 output as [Super JSON](../../formats/jsup.md).

diff --git a/docs/lake/api.md b/docs/lake/api.md
@@ -526,7 +526,7 @@ service will expect ZSON as the payload format.
 An exception to this is when [loading data](#load-data) and Content-Type is not
 specified. In this case the service will attempt to introspect the data and may
 determine the type automatically. The
-[input formats](../commands/zq.md#input-formats) table describes which
+[input formats](../commands/super.md#input-formats) table describes which
 formats may be successfully auto-detected.
 
 ### Response Payloads

diff --git a/docs/language/operators/file.md b/docs/language/operators/file.md
@@ -8,5 +8,5 @@
 
 :::tip Note
 The `file` shorthand is exclusively for working with inputs to
-[`zq`](../../commands/zq.md) and is not available for use with [Zed lakes](../../commands/zed.md).
+[`super`](../../commands/super.md) and is not available for use with [Zed lakes](../../commands/zed.md).
 :::
diff --git a/docs/language/operators/from.md b/docs/language/operators/from.md
@@ -29,13 +29,13 @@ their data to its output.  A data source can be
 * the [`pass` operator](pass.md), to treat the upstream pipeline branch as a source.
 
 :::tip Note
-File paths and URIs may be followed by an optional [format](../../commands/zq.md#input-formats) specifier.
+File paths and URIs may be followed by an optional [format](../../commands/super.md#input-formats) specifier.
 :::
 
 Sourcing data from pools is only possible when querying a lake, such as
 via the [`super db` command](../../commands/zed.md) or
 [SuperDB lake API](../../lake/api.md). Sourcing data from files is only possible
-with the [`super` command](../../commands/zq.md).
+with the [`super` command](../../commands/super.md).
 
 When a single pool name is specified without `@`-referencing a commit or ID, or
 when using a pool pattern, the tip of the `main` branch of each pool is

diff --git a/docs/language/operators/load.md b/docs/language/operators/load.md
@@ -11,7 +11,7 @@ load <pool>[@<branch>] [author <author>] [message <message>] [meta <meta>]
 :::tip Note
 The `load` operator is exclusively for working with pools in a
 [SuperDB data lake](../../commands/zed.md) and is not available for use in
-[`super`](../../commands/zq.md).
+[`super`](../../commands/super.md).
 :::
 
 ### Description

diff --git a/docs/language/overview.md b/docs/language/overview.md
@@ -51,7 +51,7 @@ much as a modern SQL engine optimizes a declarative SQL query.
 ## Search and Analytics
 
 Zed is also intended to provide a seamless transition from a simple search experience
-(e.g., typed into a search bar or as the query argument of the [`zq`](../commands/zq.md) command-line
+(e.g., typed into a search bar or as the query argument of the [`super`](../commands/super.md) command-line
 tool) to more a complex analytics experience composed of complex joins and aggregations
 where the Zed language source text would typically be authored in a editor and
 managed under source-code control.

diff --git a/docs/tutorials/zed.md b/docs/tutorials/zed.md
@@ -85,7 +85,7 @@ Our data has been committed. The `-use prs` argument in `zed load` tells
 
 With our data now loaded let's run a quick `count()` query to verify that we have
 the expected data. To do this we'll use the `zed query` command. To those
-familiar with [`zq`](../commands/zq.md), `zed query` operates similarly except
+familiar with [`super`](../commands/super.md), `zed query` operates similarly except
 it doesn't accept file input arguments since it queries pools.
 
 ```bash

diff --git a/docs/tutorials/zq.md b/docs/tutorials/zq.md
@@ -9,7 +9,7 @@ This tour provides new users of `super` an overview of the tool and
 the [SuperPipe language](../language/README.md)
 by walking through a number of examples on the command-line.
 This should get you started without having to read through all the gory details
-of the [SuperPipe language](../language/README.md) or [`super` command-line usage](../commands/zq.md).
+of the [SuperPipe language](../language/README.md) or [`super` command-line usage](../commands/super.md).
 
 We'll start with some simple one-liners on the command line where we feed
 some data to `super` with `echo` and specify `-` for `super` input to indicate
@@ -37,7 +37,7 @@ doing interesting things on that input, and emitting results, of course, as JSON
 
 `jq` is awesome and powerful, but its syntax and computational model can
 sometimes be daunting and difficult.  We tried to make `zq` really easy and intuitive,
-and it is usually faster, sometimes [much faster](../commands/zq.md#performance),
+and it is usually faster, sometimes [much faster](../commands/super.md#performance),
 than `jq`.
 
 To this end, if you want full JSON compatibility without having to delve into the
@@ -166,7 +166,7 @@ go from ZSON to an efficient binary row format ([Super Binary](../formats/bsup.m
 and columnar ([Super Columnar](../formats/csup.md)) --- and vice versa ---
 with complete fidelity and no loss of information.  In this tour,
 we'll stick to ZSON (though for large data sets,
-[ZNG is much faster](../commands/zq.md#performance)).
+[Super Binary is much faster](../commands/super.md#performance)).
 
 The first thing you'll notice about ZSON is that you don't need
 quotations around field names.  We can see this by taking some JSON