Skip to content

Latest commit

 

History

History
111 lines (88 loc) · 2.21 KB

README.md

File metadata and controls

111 lines (88 loc) · 2.21 KB
Raw

SyntheticDatasetGenerator

Dependencies

pip install -r requirements.txt
  • python >=3.6
  • liquibase in the path (in case documentation generation needed)
  • postgresql database (in case both documentation & data loaded needed)

## Run

  1. Push your base file into input
  2. Fill config/sdgen.yaml
  3. Run make clean run ddl

Install

make

Test

make test

Principle

Goal

  • input:
    • a config file in yaml
    • csv files already existing (eg: terminologies)
  • output:
    • one ore multiple data csv reproductible

Csv Format (input & output)

Csv must respect:

  • encoding: utf-8
  • separator: ";"
  • quote: False
  • header: None
  • date format: "%Y-%M-%d"
  • datetime format: "%Y-%M-%d %H:%m:%s"
  • strings must not contain the separator (";")

Table

  • Primary Keys: each table must have an integer primary key
  • Foreign Keys: they must refer a primary key.

Yaml Configuration

  • Table
    • existing table must have a "input" defined
    • generated table must not have "input" defined
  • Fields Class
    • sequence: an autoincrement from 1 to "table.tableSize"
    • simple (percentNull)
      • integer (begin, end)
      • bigint (begin, end)
      • date (begin, end)
      • time (begin, end)
      • varchar (begin, end)
      • regexp (pattern)
      • real (begin, end, decimal)
    • lookup (fk, table, field)

# Data Type

  • DBC Type Java Type
  • CHAR String
  • VARCHAR String
  • LONGVARCHAR String
  • NUMERIC java.math.BigDecimal
  • DECIMAL java.math.BigDecimal
  • BIT boolean
  • BOOLEAN boolean
  • TINYINT byte
  • SMALLINT short
  • INTEGER int
  • BIGINT long
  • REAL float
  • FLOAT double
  • DOUBLE double
  • BINARY byte[]
  • VARBINARY byte[]
  • LONGVARBINARY byte[]
  • DATE java.sql.Date
  • TIME java.sql.Time
  • TIMESTAMP java.sql.Timestamp
  • CLOB Clob
  • BLOB Blob
  • ARRAY Array
  • DISTINCT mapping of underlying type
  • STRUCT Struct
  • REF Ref
  • DATALINK java.net.URL
  • JAVA_OBJECT underlying Java class

Generate schema

make load doc