Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It is not possible to set 'smlAutoChildTableNameDelimiter' nor 'smlChildTableName' when trying to insert schemaless OpenTSDBJsonFormatProtocol using the nodeJs connector #28657

Open
patmcquade opened this issue Nov 5, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@patmcquade
Copy link

patmcquade commented Nov 5, 2024

Bug Description
When setting smlAutoChildTableNameDelimiter to any value and then trying to insert schemaless data using the InfluxDB Line Protocol via the @tdengine/websocket node package, it results in an error. Using smlChildTableName has no effect, and the table names are the same as when omitting to set this value.

All of the above works when using InfluxDB Line Protocol instead of OpenTSDB JSON protocol but i actually would prefer using the OpenTSDB JSON protocol

I'm using the kubernetes Deployment from https://docs.tdengine.com/operation/deployment/#kubernetes. This has a few problems to start when using as described with tdengine 3.3.3.0 which i could resolve. I had to replace the taoskeeper.toml and set the TAOS_ADAPTER_NODE_EXPORTER_URLS env variable as both had at buildkitsandbox somewhere instead of 127.0.0.1.

To Reproduce
Steps to reproduce the behavior:

  1. set smlAutoChildTableNameDelimiter to any value i tried it with '_'
  2. run the example https://docs.tdengine.com/reference/connectors/node/#schemaless-writing removing/commenting out the telnet and influxparts
  3. The creation of the db works but writing the data fails

  1. set smlChildTableName to 'tname'
  2. run the example https://docs.tdengine.com/reference/connectors/node/#schemaless-writing removing/commenting out the telnet and influxparts and edit the influxdata var to
    let jsonData = ["{\"metric\": \"metric_json\",\"tname\":\"some_tname\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"];
  3. the generated table does not have the name some_tname as defined above but the autogenerated name

Expected Behavior
It should work as described in the documentation https://docs.tdengine.com/reference/schemaless/#main-processing-logic-for-schemaless-writing, which is the case with the InfluxDB Line Protocol

Screenshots
Error when using smlAutoChildTableNameDelimiter

schemalessInsertError

Environment
statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tdengine
  namespace: test
  labels:
    app: taosd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: taosd
  template:
    metadata:
      labels:
        app: taosd
    spec:
      containers:
        - name: tdengine
          image: tdengine/tdengine:3.3.3.0
          ports:
            - name: tcp0
              containerPort: 6030
              protocol: TCP
            - name: tcp1
              containerPort: 6041
              hostPort: 36041
              protocol: TCP
          envFrom:
            - configMapRef:
                name: tdengine-config
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: SERVICE_NAME
              value: tdengine
            - name: STS_NAME
              value: tdengine
            - name: STS_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: TAOS_SERVER_PORT
              value: '6030'
            - name: TAOS_FIRST_EP
              value: $(STS_NAME)-0.$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local:$(TAOS_SERVER_PORT)
            - name: TAOS_FQDN
              value: $(POD_NAME).$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local
            - name: TAOS_MONITOR_FQDN
              value: 'NONE'
            - name: TAOS_SML_AUTO_CHILD_TABLE_NAME_DELIMITER
              value: '_'
            #            - name: TAOS_SML_CHILD_TABLE_NAME
            #              value: 'tname'
            - name: TAOS_ADAPTER_NODE_EXPORTER_URLS
              value: '["http://localhost:9100"]'
          resources: { }
          volumeMounts:
            - name: tdengine-taosdata
              mountPath: /var/lib/taos
            - name: tdengine-taoslog
              mountPath: /var/log/taos
            - name: taos-keeper-toml
              mountPath: /etc/taos/taoskeeper.toml
              subPath: taoskeeper.toml
              readOnly: true
          livenessProbe:
            exec:
              command:
                - taos-check
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            exec:
              command:
                - taos-check
            initialDelaySeconds: 5
            timeoutSeconds: 5000
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      volumes:
        - name: taos-keeper-toml
          configMap:
            name: taos-keeper-toml-cm
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: tdengine-taosdata
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        volumeMode: Filesystem
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: tdengine-taoslog
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  serviceName: tdengine
  updateStrategy:
    type: RollingUpdate
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: tdengine
  namespace: test
spec:
  ports:
    - name: tcp0
      protocol: TCP
      port: 6030
    - name: tcp1
      protocol: TCP
      port: 36041
      targetPort: 6041

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: tdengine-config
  namespace: test
data:
  CLUSTER: '1'
  TAOS_REPLICA: '1'

taos-keeper-toml-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: taos-keeper-toml-cm
  namespace: test
data:
  taoskeeper.toml: |
    instanceId = 64
    
    # Listen port, default is 6043
    port = 6043
    
    # go pool size
    gopoolsize = 50000
    
    # interval for metrics
    RotationInterval = "15s"
    
    [tdengine]
    host = "127.0.0.1"
    port = 6041
    username = "root"
    password = "taosdata"
    usessl = false
    
    [metrics]
    # metrics prefix in metrics names.
    prefix = "taos"
    
    # export some tables that are not super table
    tables = []
    
    # database for storing metrics data
    [metrics.database]
    name = "log"
    # database options for db storing metrics data
    [metrics.database.options]
    vgroups = 1
    buffer = 64
    keep = 90
    cachemodel = "both"
    
    [environment]
    # Whether running in cgroup.
    incgroup = false
    
    [log]
    # The directory where log files are stored.
    # path = "/var/log/taos"
    level = "info"
    # Number of log file rotations before deletion.
    rotationCount = 30
    # The number of days to retain log files.
    keepDays = 30
    # The maximum size of a log file before rotation.
    rotationSize = "1GB"
    # If set to true, log files will be compressed.
    compress = false
    # Minimum disk space to reserve. Log files will not be written if disk space falls below this limit.
    reservedDiskSize = "1GB"
@patmcquade patmcquade added the bug Something isn't working label Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant