merge v1.29.0 into master branch

F5Networks · May 31, 2022 · cc1366b · cc1366b
1 parent f10f0e5
commit cc1366b
Show file tree

Hide file tree

Showing 56 changed files with 4,304 additions and 153 deletions.
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
@@ -269,8 +269,8 @@ build_docs:
         -   rm -rf docs/_templates
         - fi
         - make html
-        - vale --glob='*.{md,rst}' .
-        #  - make linkcheck
+        - vale --glob='*.rst' .
+        - make linkcheck
     tags:
         - docker-executor
     artifacts:
@@ -479,6 +479,7 @@ pages:
         - develop
         - doc-release-branch
         - joes-new-ts-wip
+        - debs-doc-wip
 
 # Publish docs to clouddocs.f5.com
 publish_docs_to_production:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,17 @@
 # Changelog
 Changes to this project are documented in this file. More detail and links can be found in the Telemetry Streaming [Document Revision History](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html).
 
+## 1.29.0
+### Added
+- TS-568: Added support for SNMP endpoints in Telemetry_Endpoints
+- TS-233: [GitHub #186](https://github.com/F5Networks/f5-telemetry-streaming/issues/186): Added outputMode parameter to Generic HTTP consumer to support raw data output
+### Fixed
+- TS-660 - Metric Consumers should not fail when 'null' found in data
+- TS-69 - Prometheus consumer should ignore NaN values
+### Changed
+- AUTOTOOL-3077: Update npm packages (fixed medium and low priority vulnerabilities in dependencies - kafka-node)
+### Removed
+
 ## 1.28.0
 ### Added
 ### Fixed

diff --git a/SUPPORT.md b/SUPPORT.md
@@ -18,10 +18,10 @@ Currently supported versions:
 | Software Version | Release Type  | First Customer Ship | End of Support  |
 |------------------|---------------|---------------------|-----------------|
 | TS 1.20.1        | LTS           |  30-Jun-2021        | 30-Jun-2022     |
-| TS 1.26.0        | Feature       |  25-Jan-2022        | 25-Apr-2022     |
 | TS 1.27.0        | Feature       |  08-Mar-2022        | 08-Jun-2022     |
 | TS 1.27.1        | LTS           |  19-Apr-2022        | 19-Apr-2023     |
 | TS 1.28.0        | Feature       |  19-Apr-2022        | 19-Jul-2022     |
+| TS 1.29.0        | Feature       |  31-May-2022        | 31-Aug-2022     |
 
 Versions no longer supported:
 
@@ -52,5 +52,6 @@ Versions no longer supported:
 | TS 1.23.0        | Feature       |  21-Sep-2021        | 21-Dec-2021     |
 | TS 1.24.0        | Feature       |  02-Nov-2021        | 02-Feb-2022     |
 | TS 1.25.0        | Feature       |  14-Dec-2021        | 14-Mar-2022     |
+| TS 1.26.0        | Feature       |  25-Jan-2022        | 25-Apr-2022     |
 
 See the [Release notes](https://github.com/F5Networks/f5-telemetry-streaming/releases) and [Telemetry Streaming documentation](https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/revision-history.html) for new features and issues resolved for each release.
diff --git a/contributing/README.md b/contributing/README.md
@@ -108,7 +108,7 @@ How does the project handle a typical `POST` request?
             "trace": false,
             "format": "default"
         },
-        "schemaVersion": "1.28.0"
+        "schemaVersion": "1.29.0"
     }
 }
 ```

diff --git a/contributing/process_release.md b/contributing/process_release.md
@@ -64,6 +64,7 @@
     * 1.26.0 - 17.8 MB
     * 1.27.0 - 17.7 MB
     * 1.28.0 - 17.7 MB
+    * 1.29.0 - 17.8 MB
   * Install build to BIG-IP, navigate to folder `/var/config/rest/iapps/f5-telemetry/` and check following:
     * Run `du -sh` and check that folder's size (shouldn't be much greater than previous versions):
       * 1.4.0 - 65 MB
@@ -91,6 +92,7 @@
       * 1.26.0 - 127 MB
       * 1.27.0 - 127 MB
       * 1.28.0 - 127 MB
+      * 1.29.0 - 129 MB
     * Check `node_modules` folder - if you see `eslint`, `mocha` or something else from [package.json](package.json) `devDependencies` section - something wrong with build process. Probably some `npm` flags are work as not expected and it MUST BE FIXED before publishing.
 * Ensure that all tests (unit tests and functional tests passed)
 * Optional: Ensure that your local tags match remote. If not, remove all and re-fetch:

diff --git a/docs/conf.py b/docs/conf.py
@@ -79,7 +79,7 @@
 # The short X.Y version.
 version = u''
 # The full version, including alpha/beta/rc tags.
-release = u'1.28.0'
+release = u'1.29.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/docs/custom-endpoints.rst b/docs/custom-endpoints.rst
@@ -17,21 +17,25 @@ The Telemetry_Endpoints class is where you define your endpoints and their paths
 
 
 
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-| Parameter          | Required?  |  Description/Notes                                                                                      |
-+====================+============+=========================================================================================================+
-| class              |   Yes      |  Telemetry_Endpoints                                                                                    |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-| basePath           |   No       |  Optional base path value to prepend to each individual endpoint path you specify in "items"            |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-| enable             |   No       |  Whether you want to enable this class. The default is **true**.                                        |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-| items              |   Yes      |  Object with each property an endpoint with their own properties.                                       |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-|    \- name         |   No       |  Optional name for the item                                                                             |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
-|    \- path         |   Yes      |  Path to query data from                                                                                |
-+--------------------+------------+---------------------------------------------------------------------------------------------------------+
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+| Parameter          | Required?  |  Description/Notes                                                                                |
++====================+============+===================================================================================================+
+| class              |   Yes      |  Telemetry_Endpoints                                                                              |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+| basePath           |   No       |  Optional base path value to prepend to each individual endpoint path you specify in "items"      |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+| enable             |   No       |  Whether you want to enable this class. The default is **true**.                                  |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+| items              |   Yes      |  Object with each property an endpoint with their own properties.                                 |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+|    \- name         |   No       |  Optional name for the item                                                                       |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+|    \- path         |   Yes      |  Path to query data from                                                                          |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
+|                    |            |  Which protocol to use to retrieve custom endpoint data.                                          |
+|    \- protocol     |   No       |  **http:** Retrieves data using http requests from the specified iControlREST endpoints.          |
+|                    |            |  **snmp:** Retrieves SNMP from "snmpwalking" the specified SNMP OIDs.                             |
++--------------------+------------+---------------------------------------------------------------------------------------------------+
 
 
 For example, your declaration could add the following snippet, which contains endpoints for profiles, and for total connections for a virtual:

diff --git a/docs/declarations.rst b/docs/declarations.rst
@@ -347,6 +347,27 @@ With the **addTags** property, the StatsD consumer also sends numeric metrics bu
 |
 
 
+.. _snmpep:
+
+Querying SNMP using a custom endpoint
+-------------------------------------
+.. sidebar:: :fonticon:`fa fa-info-circle fa-lg` Version Notice:
+
+   Support for SNMP for custom endpoints is available in TS v1.29 and later
+
+
+This example shows how you can query SNMP using a custom endpoint in a Telemetry Streaming declaration using version 1.29 and later. 
+See :doc:`custom-endpoints` for more information on this feature.
+
+
+.. literalinclude:: ../examples/declarations/system_custom_endpoints_with_snmp.json
+    :language: json
+
+
+
+|
+
+
 
 
 .. |proxy| raw:: html

diff --git a/docs/event-listener.rst b/docs/event-listener.rst
@@ -133,7 +133,7 @@ If you are using a standard BIG-IP system (one that does not have restrictions o
 
    .. code-block:: bash
 
-        create ltm virtual telemetry_local destination 255.255.255.254:6514 rules { telemetry_local_rule }
+        create ltm virtual telemetry_local destination 255.255.255.254:6514 rules { telemetry_local_rule } profiles replace-all-with { f5-tcp-progressive } source-address-translation { type automap } persist replace-all-with { source_addr { default yes }}
 
 
 #. Create the pool.

diff --git a/docs/revision-history.rst b/docs/revision-history.rst
@@ -11,8 +11,20 @@ Document Revision History
         - Description
         - Date
 
+      * - 1.29.0
+        - **Updated the documentation for Telemetry Streaming v1.29.0. This release contains the following changes:** |br| * Added support for querying SNMP using a custom endpoint (see :ref:`SNMP example<snmpep>`) |br| * Added outputMode parameter to Generic HTTP consumer to support raw data output (see :ref:`Generic HTTP<http-ref>`) |br| |br| Issues Resolved: |br| * Metric Consumers should not fail when 'null' found in data |br| * Prometheus consumer should ignore NaN values
+        - 5-31-22
+
+      * - 1.28.0
+        - Updated the documentation for Telemetry Streaming v1.28.0. This release contains the following changes: |br| * Increased number of messages in a single PutLogEvents command for AWS Logs consumer and added retry logic for InvalidSequenceTokenException, `GitHub #191 <https://github.com/F5Networks/f5-telemetry-streaming/issues/191>`_ |br| * Updated the path to stored declarations, and noted that qkview now has access to the Declaration History file (see :ref:`Stored Declarations<save>`)
+        - 4-19-22
+
+      * - 1.27.1
+        - Released Telemetry Streaming v1.27.1 as a LTS (Long Term Support) version. See the Telemetry Streaming |supportmd| for information about the Telemetry Streaming support policy.
+        - 4-19-22
+
       * - 1.27.0
-        - Updated the documentation for Telemetry Streaming v1.27.0. This release contains the following changes: |br| * Telemetry Streaming now stores up to 30 of the most recent declarations (see :ref:`Stored Declarations<save>`) |br| * Verified support for ElasticSearch 8 (see :ref:`ElasticSearch<elasticsearch-ref>`) |br| * Removed the experimental label from the OpenTelemetry Exporter consumer (see :ref:`OpenTelemetry<opent>`) |br| * Updated the DataDog consumer to use the v2 logs API |br| * TS now closes the tracer by timeout when inactive |br| |br| Issues Resolved: |br| * Added additional sanitation to StatsD metric and tag naming, `GitHub #184 <https://github.com/F5Networks/f5-telemetry-streaming/issues/184>`_ |br| * RegEx catastrophic backtracking on attempt to mask secrets in JSON data |br| * Added info about installed ASM attack signatures to output (see :ref:`System Information<sysinfo>`), `GitHub #174 <https://github.com/F5Networks/f5-telemetry-streaming/issues/174>`_)
+        - Updated the documentation for Telemetry Streaming v1.27.0. This release contains the following changes: |br| * Telemetry Streaming now stores up to 30 of the most recent declarations (see :ref:`Stored Declarations<save>`) |br| * Verified support for ElasticSearch 8 (see :ref:`ElasticSearch<elasticsearch-ref>`) |br| * Removed the experimental label from the OpenTelemetry Exporter consumer (see :ref:`OpenTelemetry<opent>`) |br| * Updated the DataDog consumer to use the v2 logs API |br| * TS now closes the tracer by timeout when inactive |br| |br| Issues Resolved: |br| * Added additional sanitation to StatsD metric and tag naming, `GitHub #184 <https://github.com/F5Networks/f5-telemetry-streaming/issues/184>`_ |br| * RegEx catastrophic backtracking on attempt to mask secrets in JSON data |br| * Added info about installed ASM attack signatures to output (see :ref:`System Information<sysinfo>`, `GitHub #174 <https://github.com/F5Networks/f5-telemetry-streaming/issues/174>`_)
         - 3-8-22
 
       * - 1.26.0

diff --git a/docs/setting-up-consumer.rst b/docs/setting-up-consumer.rst
@@ -42,7 +42,7 @@ The following items have been added to the Splunk consumer since it was introduc
       * - 1.19
         - **compressionType**
         - Sets the type of compression.  Be sure to see :ref:`Memory usage spikes<splunkmem>` in the Troubleshooting section for information on the **compressionType** property. When set to **none**, this property stops TS from compressing data before sending it to Splunk, which can help reduce memory usage.
-        - 
+
 
 **IMPORTANT**: The following declaration includes the additional properties shown in the table. If you attempt to use this declaration on a previous version, it will fail. On previous versions, remove the highlighted line(s), and the comma from the previous line. 
 
@@ -566,8 +566,8 @@ The following items have been added to the StatsD consumer since it was introduc
       * - 1.25
         - **convertBooleansToMetrics**
         - This property allows you to choose whether or not to convert boolean values to metrics (true becomes 1, false (default0) becomes 0). |br| By default, Telemetry Streaming uses Boolean values as tag values that are attached to individual metrics. If **convertBooleansToMetrics** is set to **true**, any Boolean values are instead converted to numeric values, which are then sent to the consumer(s) as a metric. |br| Note: Telemetry Streaming does not send a Boolean as both a tag and a metric; a Boolean value is sent to the consumer(s) as either a tag or as a metric.
-        - 
-
+
+        
 **IMPORTANT**: The following declaration includes the additional properties shown in the table. If you attempt to use this declaration on a previous version, it will fail. On previous versions, remove the highlighted line(s), and the comma from the previous line. 
 
 
@@ -623,6 +623,10 @@ The following items have been added to the Generic HTTP consumer since it was in
         - **rootCertificate**
         - The Certificate Authority root certificate, used to validate the client certificate. Certificate verification can be disabled by setting allowSelfSignedCert=true. Must be formatted as a 1-line string, with literal new line characters.
 
+      * - 1.29
+        - **outputMode**
+        - Possible values: **raw**, **processed**.  OutputMode provides the options to send data with the Generic HTTP consumer in an "as-is" (**raw**) format instead of the generated JSON payload (**processed**) allowing the data to be sent in Line Protocol format to a raw event listener and have it forwarded through the Generic HTTP consumer. 
+
 |
 
 **IMPORTANT**: The following declaration includes the additional properties shown in the table. If you attempt to use this declaration on a previous version, it will fail. On previous versions, remove the highlighted line(s), and the comma from the previous line. 
@@ -631,6 +635,7 @@ Example Declaration:
 
 .. literalinclude:: ../examples/declarations/generic_http.json
     :language: json
+    :emphasize-lines: 24
 
 
 |
@@ -921,7 +926,7 @@ OpenTelemetry Exporter
 
 .. sidebar:: :fonticon:`fa fa-info-circle fa-lg` Version Notice:
 
-    The OpenTelemetry Exporter consumer was introduced as an EXPERIMENTAL consumer in TS 1.23, and is supported in DO 1.28 and later.
+    The OpenTelemetry Exporter consumer was introduced as an EXPERIMENTAL consumer in TS 1.23, and is supported in TS 1.27 and later.
 
 The OpenTelemetry Exporter Consumer exports telemetry data to an OpenTelemetry Collector, or OpenTelemetry Protocol compatible API.
 

diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
@@ -108,7 +108,9 @@ Specific troubleshooting entries
 
 I need to access declarations I previously sent to Telemetry Streaming
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-As of Telemetry Streaming 1.27, TS stores up to 30 recent declarations at **/shared/tmp/telemetry/declarationHistory**. 
+In Telemetry Streaming 1.27, TS stored up to 30 recent declarations at **/shared/tmp/telemetry/declarationHistory**. 
+
+In Telemetry Streaming 1.28 and later, TS stores up to 30 recent declarations at **/var/log/restnoded/telemetryDeclarationHistory**.  These stored declarations are now accessible by F5's **qkview** utility for use by F5 Technical Support if necessary.  For more information on the qkview utility, see |qkv|.
 
 This includes declarations submitted upon following events:
 
@@ -457,4 +459,8 @@ Telemetry Streaming 1.19 and later includes the **compressionType** property in
 
 .. |issues| raw:: html
 
-   <a href="https://github.com/F5Networks/f5-telemetry-streaming/issues" target="_blank">Issue on GitHub</a>
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/issues" target="_blank">Issue on GitHub</a>
+
+.. |qkv| raw:: html
+
+   <a href="https://github.com/F5Networks/f5-telemetry-streaming/issues" target="_blank">qkview on AskF5</a>
diff --git a/examples/declarations/generic_http.json b/examples/declarations/generic_http.json
@@ -20,6 +20,7 @@
         ],
         "passphrase": {
             "cipherText": "apikey"
-        }
+        },
+        "outputMode": "processed"
     }
-}
+}