Skip to content

Latest commit

 

History

History
2290 lines (1111 loc) · 111 KB

CHANGELOG.md

File metadata and controls

2290 lines (1111 loc) · 111 KB
Raw

CHANGELOG

v0.26.0 (2023-08-01)

Ci

  • ci: Update semantic release config (d7f778b)

Documentation

  • docs: Add currently supported Mappings to documentation intro (ada73aa)

Feature

  • feat: Add mapping for iNaturalist gbif data

This mapping adds a mapping to lbsntransform for the iNaturalist gbif CSV data format [1]. Some opinionated decisions are made in the mapping, (1) species names are mapped to lbsn post_filter; (2) kingdom is mapped to topic_group, and (3) occurrences that match known emoji are added to the lbsn emoji list. \n[1]: https://www.gbif.org/dataset/50c9509d-22c7-4a22-a47d-8c48425ef4a7 (02368df)

Fix

  • fix: If source_web is True, input_path expects a List[str] (a1339b1)

  • fix: --use_csv_dictreader not working when input source is url (2f04251)

Refactor

  • refactor: Improve datetime parsing (2449044)

  • refactor: Use kwargs for repeated attrs (8265985)

Style

  • style: Minor code formatting (244b9d6)

Unknown

  • Merge branch 'feat-inaturalist' into 'master'

fix: If source_web is True, input_path expects a List[str]

See merge request lbsn/lbsntransform!18 (b56ec2c)

  • Merge branch 'feat-inaturalist' into 'master'

feat: Add mapping for iNaturalist gbif data

See merge request lbsn/lbsntransform!17 (9d3f2d7)

  • tests: Add iNaturalist gbif integration test (23cc9f2)

  • mappings: Use lower case for topic_group attribute (a008360)

v0.25.1 (2023-05-16)

Fix

  • fix: TypeError for unhashable type: 'CompositeKey' (lazy-initialized) (68fd9c5)

Unknown

  • mappings: Capture media-oembed NoneType (40ec1a0)

  • mappings: Reddit capture NoneType for media_type (b1412d5)

  • mappings: Fix wrong referencedPost_pkey in comment processing (2410fae)

  • mappings: Fix reddit mapping submitting both Post and PostReaction for the same reference (b996489)

v0.25.0 (2023-05-15)

Chore

  • chore: Use commit as version source for python-semantic-release (0c49de7)

Ci

  • ci: Activate tests on all branches (a32b3f1)

  • ci: Remove duplicate install (8300c73)

  • ci: Fix gitlab docker runner connection to lbsn-rawdb-test (docker network) (b8e0fbe)

  • ci: Fix nltk not installed (9e681e0)

  • ci: Fix argdown (c559e58)

  • ci: Disable releasing on development branches (5e38bdf)

  • ci: Split release stage into test and artifacts, add pip caching (ba0ef47)

Feature

  • feat: Add mapping for Reddit comments and submissions (a665e9d)

  • feat: Add new topic_group and post_downvotes attribute (12c6bbb)

  • feat: Allow handing pre-defined BaseConfig to main() (14549d9)

Fix

  • fix: Report package version without package name (94081e0)

Refactor

  • refactor: Significant refactor for Hashtag, @-Mention and UserMention in HelperFunctions (ead9afb)

  • refactor: Remove complicated ProtoCompositeContainer check, replace with simple length-check (33251be)

  • refactor: Update standard lbsnmapping - simplify header (7bf1d55)

  • refactor: Simplify handling of unused args and kwargs in mappings (8bec334)

  • refactor: Handling of mapping-arguments as kwargs and args (04241da)

Style

  • style: Refactor code style (53214be)

Unknown

  • Merge branch 'reddit-mapping' into 'master'

feat: Add Reddit mapping

See merge request lbsn/lbsntransform!16 (5d65ffc)

  • tests: Add basic tests for user-mentions and hashtag-extractors (3d128d2)

  • todo: Add hint to list-append for Twitter mapping (e6bccc9)

  • deps: Update lbsnstructure dependency to 1.2.0 (cde7c49)

  • Merge branch 'ci-test' into 'master'

Ci test

See merge request lbsn/lbsntransform!15 (ba22067)

  • Merge branch 'tests' into 'master'

Tests

See merge request lbsn/lbsntransform!14 (878ab08)

  • tests: Add YFCC integration test and update CI (312b958)

  • dev: Pytest testpaths (ee564cd)

  • dev: Add pytest to depdendencies (fb21071)

  • 0.24.2 (b5ed646)

v0.24.2 (2023-05-09)

Chore

  • chore: Update black formatting with config.py exclude (cc4aa3b)

Documentation

  • docs: Update to reflect pyproject.toml only based setup (6bce8d9)

  • docs: Fix formatting in cli-commands page (c482347)

Fix

  • fix: Update yml dependencies to use version range for protobuf, too (bfad7b2)

  • fix: Migrate from setup.cfg to pyproject.toml

fixes: #7 (7ee7598)

Unknown

  • Fix pipeline badge URL in README.md (ded258e)

  • Source path update to version.py (825008d)

  • ci - use version from file

ERROR: Job failed: exit code 1 (?) (21c4f3c)

  • Fix argparse indentation error (8ba2828)

  • Cleanup step after semantic-version fix (804d464)

v0.24.1 (2023-05-05)

Unknown

  • Use tags as singe-point-of-truth for version source (62fa9b3)

v0.24.0 (2023-05-05)

Ci

  • ci: Fix semantic-release version flow (d4d62a5)

  • ci: Use version from git tags (ef7be73)

  • ci: Update documentation deployment from external to GL pages (34352d9)

Documentation

  • docs: Add information regarding semantic release versioning (eb667d4)

  • docs: Update installation instructions based on new dependencies and deprecated setup.py (416c869)

  • docs: Add a short description of the file structure and layout (d56041d)

Feature

  • feat: Migrate from setup.py to pyproject.toml (7de0de0)

Refactor

  • refactor: Update mappings to match new lbsnstructure spec (701f2eb)

  • refactor: Follow the standard src-layout for setuptools (0e6d96a)

Unknown

  • Working merge for file-based semantic versioning using setup.cfg and pyproject.toml (3d1b724)

  • Fix build (e095227)

  • dependencies: Pin lbsnstructure>=1.0.3 with protobuf 4.19.0 compiled (fc99f2e)

  • dependencies: Pin shapely<2.0.0 until GEOS.WKTWriter is available (847ab81)

  • Fix docs wrong folder (de3515a)

  • Fix paths in ci (4858388)

  • Fix paths (a7e2d7d)

  • Fix cp: can't create 'public/...' (18271f3)

  • Fix paths in documentation deploy (cbae069)

v0.23.0 (2022-11-24)

Chore

  • chore: Also update pinning of protobuf in setup.py (1bc1974)

  • chore: Update dependency pinnings and dev environment (aaf70f5)

  • chore: Pin emoji package to >=2.0.0 (fa8a129)

Documentation

  • docs: Fix typo (7e43ef4)

  • docs: Fix typo in --commit_volume cli docs (70d96b9)

Feature

  • feat: Add experimental temporal._month_hashtag_latlng base processing (60da223)

Fix

  • fix: NLTK stopwords not available notice (c395bcb)

  • fix: EMOJI_UNICODE deprecated in emoji>=2.0.0

Use the official function to extract distinct unicode emoji from string, available from emoji 2.0.0 on (3097a42)

Unknown

  • Merge branch 'composite-feature-multi' into 'master'

Composite feature multi base

See merge request lbsn/lbsntransform!13 (a612b4e)

  • Update CHANGELOG.md (990ca8f)

  • Merge branch 'master_gh' into master_new (1ea3d66)

  • Fix typo (a995b6f)

  • Update setup.py

Fix: Vulnerability in protobuf dependency (dcf7a45)

v0.22.1 (2022-11-23)

Chore

  • chore: Update dependency pinnings and dev environment (4717c7a)

  • chore: Pin emoji package to >=2.0.0 (f4a39ec)

  • chore: Add instagram mapping example (985ee26)

Documentation

  • docs: Fix typo (63ba7e4)

  • docs: Fix typo in --commit_volume cli docs (c36d9aa)

  • docs: Better explain the use of (e43afdd)

  • docs: Clarify to use all lbsn objects by default when mapping from lbsn raw (202fd75)

  • docs: Fix internal link (67f4c90)

  • docs: Improve description of --include_lbsn_bases and --commit_volume args (b7697a3)

  • docs: Add header information on command line interface page (9602c47)

  • docs: Add changelog to documentation (766a40d)

Fix

  • fix: EMOJI_UNICODE deprecated in emoji>=2.0.0

Use the official function to extract distinct unicode emoji from string, available from emoji 2.0.0 on (599158a)

  • fix: Typo (5145455)

  • fix: --skip_until_file not implemented (18e7665)

  • fix: Fix TypeError NoneType for skipped_geo reporting (fe67231)

  • fix: Catch empty values in lbsn database record arrays (6ad2d82)

Performance

  • perf: Clear key hashes on finalize_output() (864336d)

Refactor

  • refactor: Add docstring and remove remnant (36a4f2d)

  • refactor: Same indent for all doctsrings in argparse (8fa91fb)

Style

Unknown

  • Fix emoji extract missing commits (1c309a3)

  • Merge branch 'fix-emoji-extract' into 'master'

fix: EMOJI_UNICODE deprecated in emoji>=2.0.0

See merge request lbsn/lbsntransform!12 (12e0d0f)

  • Update setup.py

Fix: Vulnerability in protobuf dependency (dbf224e)

v0.22.0 (2022-04-08)

Documentation

  • docs: Fix typo (4034867)

  • docs: Add internal links to Use Cases (da93d5c)

  • docs: Add twitterparser example to use lbsntransform as a package (2a68f08)

Feature

  • feat: Allow --commit_volume to be overriden

To prevent deadlocks on concurrent lbsntransform writes (e.g. multiple processes running at the same time (50c4bf8)

Fix

  • fix: Skip all base records with any empty key

Primary Key constraints automatically add NOT NULL constraint in postgres, thus empty keys are not allowed and can cause errors on incomping records, where information is missing (e.g. empty post_create_date for 'date' base) (e3caac1)

  • fix: Skip empty keys for temporal hll structures (263774c)

  • fix: Compatibility for 'carousel' types from previous lbsnstructure (caf21c3)

v0.21.3 (2022-03-18)

Documentation

  • docs: Add note to activate optional nltk stopwords filter feature (015511e)

  • docs: Deprecate cx_freeze setup (4700821)

Fix

  • fix: Pin google.protobuf to latest release 3.19.4 or earlier (f591487)

Unknown

  • --amend (8d810c6)

  • ScalarMapContainer not found in protobuf dependency (Windows only) (db98ae7)

v0.21.2 (2022-03-18)

Documentation

  • docs: Add hint towards Linux installation (ce2c75b)

Unknown

  • Revert "fix: ScalarMapContainer not found in protobuf dependency (Windows only)"

This reverts commit 7d469f5fa2e2ec93a0f02d5ae0c08384cf5c3ec0. (f847758)

v0.21.1 (2022-03-17)

Ci

  • ci: Fix argdown doc generator (f40b3d3)

Fix

  • fix: ScalarMapContainer not found in protobuf dependency (Windows only) (7d469f5)

  • fix: Deactivate currently not supported CSV output (92f887d)

Unknown

  • Merge branch 'fix-composite-container' into 'master'

Fix composite container

See merge request lbsn/lbsntransform!11 (63351b7)

  • Merge branch 'ci-test' into 'master'

ci: Fix argdown doc generator

See merge request lbsn/lbsntransform!10 (a30a068)

v0.21.0 (2022-03-15)

Chore

  • chore: Force update badges (7779255)

Documentation

  • docs: Remove duplicate mappings_path from example (854aad5)

  • docs: Fix internal links in Use Cases (61f772a)

Feature

  • feat: Allow lbsntransform args to be predefined by another package (c587bbf)

Unknown

  • Merge branch 'feat-predefined-args' into 'master'

feat: Allow lbsntransform args to be predefined by another package

See merge request lbsn/lbsntransform!9 (6a770f1)

v0.20.0 (2021-05-11)

Refactor

  • refactor: Reflect hlldb design decision to rename date_hll to pud_hll (Picture or Post User Days) (4f6d578)

v0.19.0 (2021-05-11)

Feature

  • feat: Add _month_latlng and _month_hashtag composite bases (71bbc22)

v0.18.3 (2021-05-05)

Chore

  • chore: Fix code block display issue in docs (13b2efc)

  • chore: Fix argparse script (f8d48ff)

Fix

  • fix: csv.reader bug that got introduced in #2a79f01c (49243c5)

v0.18.2 (2021-04-26)

Documentation

  • docs: Improve description on how hmac defaults are used (aa5e265)

Fix

  • fix: On empty hmac, do not override if crypt.salt is set in hll worker db (24f0a4d)

Unknown

  • Merge branch 'master' of github.com:Sieboldianus/lbsntransform (ff79126)

v0.18.1 (2021-04-19)

Documentation

  • docs: Fix badge links not updating on GH

According to #1065 (8a425ed)

Fix

  • fix: Error on empty HMAC (8d29832)

  • fix: Grapheme clusters not found in newest emoji.UNICODE_EMOJI (emoji >= v.1.0.1) (cfd6f28)

Unknown

  • Merge branch 'master' of github.com:Sieboldianus/lbsntransform (994e5ad)

v0.18.0 (2021-04-16)

Documentation

  • docs: Improve description of --transfer_count (9847f5e)

  • docs: Fix docstring typo (7d13423)

Feature

  • feat: Add function to extract @-Mentions from string (714d5c9)

  • feat: Add option to switch to csv.DictReader() (2a79f01)

v0.17.0 (2021-04-15)

Documentation

  • docs: Fix list formatting (2751d74)

  • docs: Fix typo (e000386)

  • docs: Update Windows install instructions (9d0cb6e)

  • docs: Fix api link (b31b1ce)

  • docs: Fix api link (1745f39)

  • docs: Add last git revision date to pages (b254df7)

Feature

  • feat: Properly intergrate hmac hashing and warn user on empty key (4cecd96)

v0.16.1 (2021-03-13)

Chore

  • chore: Add Dockerfile and update docs (2f9afae)

  • chore: fix ci yaml (11e8a45)

Documentation

  • docs: Fix Readme image link on pypi (ec352ca)

  • docs: Major overhaul of CLI argument formatting (d12b4c1)

  • docs: Improve formatting (c766d8e)

  • docs: Add docker mount note (dcd98a2)

  • docs: Add note towards Docker input from bind mounts (a1e1564)

  • docs: Fix formatting issue in --override_lbsn_query_schema (ac26a2f)

  • docs: fix typo (42fe31f)

  • docs: Add example to read from live lbsn db and to live hll db (b07a7c2)

  • docs: correct order of --editable --no-deps for quirky pip (a768da1)

Refactor

  • refactor: Docs structure and formatting (1449aca)

  • refactor: CLI arg docstring formatting (aedd00c)

  • refactor: Remove empty default passwords (f4f2e15)

  • refactor: Use absolute imports instead of relative (ab8db19)

Unknown

  • Minor cleanup (fbdc69b)

  • Restructure markdown formatting (7ee2298)

  • Minor formatting fix (0cfa7a3)

  • Fix code blocks in argument docs (ec5d7a0)

  • Fix link (d1de7e3)

  • Fix typo (74c6a11)

  • Minor formatting fix (ff8cffb)

  • Minor formatting fix (e6c73fc)

  • Fix linK (e054f01)

  • Minor formatting fix (a1791ae)

  • Minor formatting fix (fcdac9a)

  • Minor formatting fix (61f286f)

  • Merge branch 'dev' (9481e09)

  • Improve docs format of headings (eaabc88)

  • Fix markdown formatting (22aac5c)

  • Minor formatting (9bd065b)

  • Fix csv not available for syntax highlight (59eb48b)

  • Merge branch 'dev' (7ff450e)

  • Merge branch 'master' of gitlab.vgiscience.de:lbsn/lbsntransform (2f312ea)

  • Merge branch 're-master' into 'master'

it works

See merge request lbsn/lbsntransform!8 (5f9ec06)

  • it works (78e784e)

  • Merge branch 'docker-image' into 'master'

chore: fix ci yaml

See merge request lbsn/lbsntransform!7 (3ff4150)

  • Merge branch 'docker-image' into 'master'

build a docker image and push it into the registry

See merge request lbsn/lbsntransform!6 (64668ed)

v0.16.0 (2021-01-14)

Chore

  • chore: Get version from version.py (bb201b4)

Documentation

  • docs: Update linux install command (bc35783)

  • docs: Update and cleanup Readme (f16cf2c)

  • docs: Re-order recommended pip setup instructions (19e4575)

Feature

  • feat: Add --dry-run option (335631b)

Fix

  • fix: Disable CSV output until further notice (2014ce5)

Unknown

v0.15.0 (2021-01-09)

Feature

  • feat: add method for hashtag extraction from string (134119c)

Fix

  • fix: improved exception reporting for malformed records (b7b83e2)

Style

  • style: minor formatting (6adfa2b)

Unknown

  • Revert "fix: Catch any geos.WKTReader() exceptions"

This reverts commit 14f7721902cd8a42304f6a80ff6e8d53263684a1. (b8cdb99)

v0.14.1 (2021-01-06)

Ci

  • ci: Remove pypi search during version badge generation (06d90ad)

Documentation

  • docs: Fix admonition formatting (c33b8c1)

  • docs: update conda install instructions (4f0f96b)

  • docs: fix links and rehrase sections (76875bf)

Fix

  • fix: Catch any geos.WKTReader() exceptions (14f7721)

  • fix: Windows lbsntransform.tools module not found. (1643112)

Unknown

v0.14.0 (2020-12-11)

Documentation

  • docs: Fix linebreak conversion on python to markdown arg-docstring conversion (ec14cc0)

Feature

  • feat: Dynamic load of mapping modules

Remove obsolete mapping modules

Minor refactor mapping load

fix yml formatting errors

Minor formatting in docs (4168509)

  • feat: Dynamic load of mapping modules (09de72f)

Unknown

  • Minor formatting in docs (e719714)

  • fix yml formatting errors (1d3831c)

  • Minor refactor mapping load (108b97d)

  • Remove obsolete mapping modules (93d7759)

  • Move example mappings to resources folder (91f31f1)

  • Update docs with more examples (5b32cb2)

  • Compatibility for 1.4.0 LBSN protobuf structure version (51af268)

  • Add python and bash to highlightjs config of mkdocs (902127c)

  • Update links in docs (0897bdd)

  • fixup: invalid linebreak after markdown link format in docs (59184dc)

  • fixup: invalid linebreak after markdown link format in docs (632a8e7)

  • fixup: additional linebreak after colon in markdown lists (f0e4da2)

  • fixup: use space character in *.md file (44fa92e)

  • Merge branch 'dev' (ed27a32)

  • Add a quite recognizable hint to the docs. (48a85d5)

v0.13.0 (2020-05-12)

Chore

  • chore: MANIFEST.in recursive-include of submodules (80d4f65)

  • chore: update manifest, clean setup.py (2b74ef3)

Ci

  • ci: fix missing stopwords in pdoc3 doc generation (606ab7f)

  • ci: transition from gitlab only syntax to new if syntax (fd9c011)

Documentation

  • docs: add submodule docstrings (ed9b754)

  • docs: include main in api-docs, exclude empty submodule inits (a1d3e00)

  • docs: Update installation instructions (0a90472)

  • docs: Add instructions for using conda package manager (bd58d2b)

Feature

  • feat: add topical._hashtag_latlng and social.community base (d20e766)

  • feat: optional schema name override in cli (7c05673)

Fix

  • fix: emoji extracted from body do not include grapheme clusters (1ddf046)

  • fix: correct measure of userdays (hll) (36d6fa2)

Refactor

  • refactor: rename module input_data to input (0635767)

Unknown

  • Update pdoc format to process submodules (75dd5d9)

  • Update Flickr YFCC place match (4e17002)

  • Use sane_lists extensioN (eb022fa)

  • Use highlightjs for fenced code blocks (adf0180)

  • Explicitly highlight code-blocks in config (b5fb5ca)

  • Fix site_url parameter for mkdocs (ab634d7)

  • Update mkdocs extensions (6a06292)

  • Minor rephrasing (79da4fc)

v0.12.2 (2020-03-03)

Ci

Documentation

  • docs: add instructions for installing lbsntransform in Linux (6ab02ca)

Fix

  • fix: Module not found when installed with pip in Linux (718ae78)

  • fix: geos not found when installed from conda-forge (1c4adcd)

Refactor

  • refactor: use input_data instead of reserved class name 'input' (8cbd6ee)

Unknown

v0.12.1 (2020-02-11)

Chore

  • chore: add newline (argdown parse) (f9e70c3)

Ci

  • ci: fix argdown process (c7dd9b8)

Documentation

  • docs: clarify --startwith_db_rownumber (8a0d44b)

Fix

  • fix: assertion error on records-shard merge

This issue was difficult to debug. Usually, GROUP BY would also return sorted data because PG sorts data by default when using GROUP By. Except that for small arrays, it doesn't sort. Adding excplicit sort order to make_shard_sql fixes #2 (71cf51f)

  • fix: performance issue with large hll upserts (bac72b3)

  • fix: TypeError for live db query with skip_records (b0216a4)

  • fix: update lbsnstructure min version (371e35d)

  • fix: add skipping of records based on count (383a79b)

  • fix: --startwith_db_rownumber flag ignored on LBSN input (4dfa1f3)

  • fix: cli output reporting (line ending) (7ac3c01)

  • fix: add exception handling for hll db queries (3ecf721)

Refactor

  • refactor: use type alias for LBSNObjects (2a8284e)

Unknown

v0.12.0 (2020-01-22)

Feature

  • feat: add _emoji_latlng base (40634b0)

  • feat: add cli option to selectively include hll bases (bf02a53)

  • feat: allow lbsn input filtering per type (5ce9d6f)

  • feat: add composite base example (_latlng_term) (9f4b8af)

Fix

  • fix: add origin to lbsn mapping input scheme (be7ea94)

  • fix: correct lbsn reference for LBSN mapping (dcd096e)

Refactor

  • refactor: type hints for lbsn bases (use aliases) (4ad00d1)

  • refactor: type hinting (1522b3d)

Unknown

v0.11.0 (2019-12-20)

Documentation

  • docs: Update sequence chart (a891edb)

  • docs: add mkdocs Documentation, Examples and Quick Start (c7d3d47)

Feature

  • feat: Add lbsn to lbsn mapping (647fe8b)

  • feat: allow zipping of local inputs (6ebedf1)

  • feat: allow skipping until record x (3dc34d7)

Fix

  • fix: connection stream abort handling (f8a5d1b)

  • fix: yfcc parse error (0acc54d)

  • fix: yfcc100m parsing error

(NoneType, but expected one of: int, long) (e2f6ae3)

  • fix: reading stream error

Error while reading records: <class 'TypeError'> unsupported operand type(s) for +: 'NoneType' and 'list' (fe868bc)

Refactor

  • refactor: minor code updates (92b6571)

Unknown

  • Merge branch 'feat-live-import-squashed' into 'master'

Feat live import squashed

See merge request lbsn/lbsntransform!4 (23a66eb)

v0.10.2 (2019-11-21)

Refactor

  • refactor: syntax improvements (2e30326)

  • refactor: improve syntax formatting (697dddb)

Unknown

  • Remove docker login from ci (eb6e028)

  • Use full path to registry image in ci (37d5d40)

  • Add dockerlogin to gitlab-ci (57802e7)

  • Merge branch 'dev' of gitlab.vgiscience.de:lbsn/lbsntransform into dev (7827825)

  • Use local gitlab registry python-ci image (a408e47)

v0.10.1 (2019-11-20)

Fix

  • fix: invalid call to logger (ae16fd7)

Refactor

  • refactor: remove debug code (be93f9b)

v0.10.0 (2019-11-19)

Documentation

Feature

  • feat: lbsn raw to hll structure transformation

First part of hll feature implementation

Add hllworker and update bases

refactor: imports and import formatting

hll_add_agg implementation

Merge hll_add_agg results back to records

refactor: use class inheritance for hll bases

refactor: move try..catch to contextmanager

refactor: reduce code duplication in hll.Base classes

refactor: formatting to code conventions

Code separation improvements

feat: allow zipping of multiple web sources

fix: updates

fix: NUL character exception in values

fixup: nul characters

fixup: NUL character

refactor: cleanup & code duplication reduction (db1c24e)

Refactor

  • refactor: untangle classes in helper_functions (3a139fa)

  • refactor: major project structure revision

fix: null geom check (b408a6b)

  • refactor: cleanup cli args and config formatting (104c53c)

  • refactor: cleanup cli args and config formatting (c2f6537)

  • refactor: use variable for null_geom (49d859a)

Unknown

  • fix readme (765453f)

  • fix links in readme (24da601)

  • Merge branch 'dev-hll' into dev (fa903b4)

  • Pin lbsnstructure in chore (5525be0)

  • Merge branch 'master' of github.com:Sieboldianus/lbsntransform (60ca66e)

v0.9.1 (2019-10-23)

Fix

  • fix: pin lbsnstructure version (c74644c)

v0.9.0 (2019-10-23)

Feature

  • feat: migrations to lbsnstructure v1.3.0 (acbed1b)

v0.8.3 (2019-10-21)

Fix

  • fix: missing coalesce for place_description (c6457d5)

  • fix: catch empty proto_map (23559d5)

Unknown

  • substitute description from about on empty (0bd58aa)

v0.8.2 (2019-09-17)

Fix

  • fix: add support for postgres hstore (8c99a19)

Unknown

  • Update place attributes (3d1af08)

  • initial facebook place graph mapping (cbcd1d8)

v0.8.1 (2019-09-17)

Chore

  • chore: enable file name report log (88862dd)

  • chore: use main conda image for gitlab (0a91790)

  • chore: fix conda not available in latest continuum image (9587363)

  • chore: update conda container (b66c148)

  • chore: update gitlab-ci conda config (0c79a37)

Fix

  • fix: correct close of log file (c75d115)

  • fix: return records for json.load wrapper (472e3d2)

  • fix: catch jsonDecodeError on read; wrap exceptions (64888cf)

  • fix: improve uncought exception reporting (22cc420)

  • fix: CSV iterator (c372431)

  • fix: flickr mapping return pipe (393d708)

  • fix: handling of empty language in twitter json parse (bee811c)

Refactor

  • refactor: separate logging formatter for JSon exceptions (6db6b2c)

Unknown

  • append log, if already exists (320c4cf)

  • improve logging statistics (7391712)

  • concatenate log files per day (d7685e5)

  • remove trailing input() in cli-mode (9481dc1)

  • fixup: close log handlers (78756c2)

  • fixup: use correct logging reference (f7ee594)

  • fixup error reporting (6564ca9)

  • Merge branch 'master' into dev (beaf45b)

v0.8.0 (2019-08-22)

Chore

  • chore: disable no member pylint (02bd777)

Feature

  • feat: add option to process line separated json (6300838)

Fix

  • fix: reporting of count_glob and identified records (7feb38a)

  • fix: RepeatedCompositeField Error in Windows and MacOS (dc87e87)

  • fix: Twitter tweet parsing structure update (7d19357)

  • fix: wrong reporting of processed records (fcf0dd8)

  • fix: record pipeline to return only single records (8bd895a)

  • fix: database rollback integrity error (969b482)

  • fix: use pathlib for os independent path handling (0d12c9e)

  • fix: recognize transfer_limit (1fd9bde)

  • fix: add exception handling for database out of space (977517e)

Refactor

  • refactor: extract functions and follow type conventions (25df83f)

  • refactor: update to latest lbsnstructure (95c80e6)

Unknown

  • Merge branch 'dev' (a1e0fea)

  • Update config docs on revursive load info (fdcf138)

  • Update for ignoreing scripts/ (a849427)

v0.7.3 (2019-07-12)

Documentation

  • docs(readme): update command line args info (62a04cf)

Fix

  • fix: do not overwrite with Null-Island Geometry (65788a6)

  • fix: typo (8fc1769)

Unknown

v0.7.2 (2019-07-12)

Fix

  • fix: pipeline handle for different input queries (fbbfa1f)

  • fix: pipeline generators for local file loop (4f62a70)

  • fix: reporting for local input loop count (0cc4a2e)

  • fix: json local input array parse (928952d)

  • fix: store final remaining records (f789a1e)

Refactor

  • refactor: simplify main loop (6b3912f)

  • refactor: rename protected dict (c421f7b)

Unknown

  • remove file reporting (0f28368)

  • remove orphan counters (7ea800b)

  • add timestamp to protobuf func (c674eab)

v0.7.1 (2019-06-11)

Fix

  • fix: on update do not overwrite with default (9d3de9d)

  • fix: store origin_id before any insert (55df1b1)

v0.7.0 (2019-06-10)

Chore

  • chore: add requests and update dependencies (9a9089f)

Documentation

  • docs: update readme link to protobuf spec (8d72df0)

Feature

  • feat: add yfcc100m place data mapping (2245fc3)

Fix

  • fix: remove wrong place id column, update photo id (fbca35d)

Refactor

  • refactor: clean up imports (f88084e)

Unknown

  • Update handlers for default values and null geometry (f1582a9)

v0.6.0 (2019-06-03)

Chore

  • chore: add gitlab ci for basic test and badges (5d04980)

Documentation

  • docs: update readme badges (7797196)

Feature

  • feat: allow streaming web input source (64ad913)

Fix

  • fix: add bitarray for gitlab ci dev yml (5baf21c)

Unknown

v0.5.0 (2019-06-03)

v0.4.0 (2019-06-03)

Chore

  • chore: disable pylint messages (c337eaa)

  • chore: file mode changes WSL (735887c)

  • chore: add environment_dev.yml for dev deps (12466ef)

Feature

  • feat: Flickr import revise functions (4d07673)

  • feat: add importer class for YFCC100M dataset (f138f90)

  • feat: add option to specify postgres port (293e3cf)

Fix

  • fix: pathlib glob update (4b979d5)

  • fix: remove lbsntransform from dev yaml (51b5ec7)

  • fix: bug (d8f711c)

  • fix: use list for lbsn_records (7957c39)

  • fix: correct field for post_guid and add place_guid (bfc4c21)

  • fix: missing input args (f0c3337)

Refactor

  • refactor: clean up comments (cee0027)

  • refactor: major refactor for pipe processing (f99397b)

  • refactor: code conventions (485010f)

  • refactor: add ip-port split function (2678467)

  • refactor: add docstrings (706d379)

  • refactor: use pathlib for cross-system compatibility (b7265d6)

  • refactor: basic code improvements (66fa778)

  • refactor(cx_setup): code conventions (b7e32ef)

Style

  • style: move imports to head of file (ef3a6fb)

Unknown

  • fix increase csv field size limit (0f19001)

  • update comment (9a7be6f)

  • Remove re package as it is not needed (see regex) (fe2b1e0)

v0.3.21 (2019-01-11)

Fix

  • fix: input cursor ref without connection (6b01fc5)

v0.3.20 (2019-01-11)

Fix

  • fix: remove hardcoding of exclude city and country geoaccuracy posts
  • this is now possible through the use of min_geoaccuracy input arg
  • includes a number of style improvements (5956da2)

Style

  • style: minor code style updates (pylint) (04e797f)

v0.3.19 (2019-01-09)

Fix

  • fix: include srid for WKT bug
  • related to how shapely handles WKT by default
  • also fixed a number of issues and updated style conventions (fe0ffaa)

Unknown

  • bug found in null geometry (eaf85ba)

v0.3.18 (2019-01-06)

Fix

v0.3.17 (2019-01-06)

Fix

v0.3.16 (2019-01-06)

Fix

v0.3.15 (2019-01-06)

Fix

v0.3.14 (2019-01-06)

Fix

v0.3.13 (2019-01-06)

Fix

v0.3.12 (2019-01-06)

Fix

v0.3.11 (2019-01-06)

Fix

  • fix(cx_setup): use correct target name (e7ac3fd)

v0.3.10 (2019-01-06)

Fix

  • fix(cx_setup): added correct version ref (a18bee4)

v0.3.9 (2019-01-06)

Fix

  • fix(shared_structure): auto versioning (9265caf)

v0.3.8 (2019-01-06)

Fix

  • fix(shared_structure): auto versioning (19d7128)

v0.3.7 (2019-01-05)

Fix

v0.3.6 (2019-01-05)

Fix

v0.3.5 (2019-01-05)

Fix

v0.3.4 (2019-01-05)

Fix

v0.3.3 (2019-01-05)

Fix

v0.3.2 (2019-01-05)

Fix

  • fix: update autpo version (f1dd51d)

Unknown

  • fix (shared_structure): added docstrings (e26bc35)

  • feat (shared_structure): allow initializing empty structures

  • e.g. used in Tag Maps package
  • this update also changes class references, due to proper Capital Letters (d4fdb01)

v0.3.1 (2019-01-04)

Fix

v0.3.0 (2019-01-04)

Feature

  • feat: add auto changelog (926cb20)

Unknown

  • Feat: add auto changelog (2eb510d)

  • doc: added docstring to setup.py (d1d7e89)

v0.2.0 (2019-01-04)

Feature

  • feat: implement semantic versioning (41447eb)

v0.1.22 (2019-01-04)

Fix

Unknown

v0.1.21 (2019-01-04)

Fix

v0.1.20 (2019-01-04)

Fix

v0.1.19 (2019-01-04)

Fix

v0.1.18 (2019-01-04)

Fix

v0.1.17 (2019-01-04)

Fix

v0.1.16 (2019-01-04)

Fix

v0.1.15 (2019-01-04)

Fix

v0.1.14 (2019-01-04)

Fix

v0.1.13 (2019-01-04)

Fix

  • fix: setup.py version file ref (979d6e7)

v0.1.12 (2019-01-04)

Fix

  • fix: versioning publish 4 (888a584)

v0.1.11 (2019-01-04)

Fix

  • fix: versioning publish 3 (a3c0118)

v0.1.10 (2019-01-04)

Fix

  • fix: versioning publish 2 (ac0bbe3)

v0.1.9 (2019-01-04)

Fix

  • fix: versioning publish (fb52e02)

v0.1.8 (2019-01-04)

Fix

v0.1.7 (2019-01-04)

Unknown

v0.1.603 (2019-01-04)

Fix

  • fix: semantic-release versioning (f29b9dc)

v0.1.601 (2019-01-04)

Feature

  • feat: add semantic-release version control

Fix

  • fix: increase version number in main.py too (f88b408)

v0.1.600 (2019-01-03)

Unknown

  • Increased version to 0.1.600 (36b0758)

  • Minor bugfixes due to code refactor

  • added docstrings to modules (52608e4)

  • Refactored main, config; tested & bugfixes (da20640)

  • Additional code refactor for matching conventions

  • submit_data class
  • pep8 conformity (e27786a)

  • refactored main to pep8 (0679c00)

  • more code refactorings according to pep8 (3f1645f)

  • Refactored to snake style additional (1a87553)

  • refactored HF and twitter mapping to snake style code convention (fed1a4b)

  • Minor refactoring based on VSCode Move (e34af34)

  • Fixed reference bug for local file input (94e5362)

  • Fixed 2 issues with parsing incomplete twitter jsons

  • no country code
  • no bounding box (bd6ab20)

  • Updated gitignore, changed Line endings CRLF to LF (b702b5d)

  • Increased version, fixed merge conflicts (6390146)

v0.1.521 (2018-12-23)

Unknown

  • Fixed dev merge (6bc62a1)

  • Added min geoaccuracy functions

  • needs to be checked with Flickr mapping
  • main() needs to be cleaned up, better organisation of reporting statistics (f62bade)

v1.5.20 (2018-12-19)

Unknown

v0.1.518 (2018-12-19)

Unknown

  • Increased version (e8e3ee8)

  • Merge branch 'dev'

Fixed proto-composite-bug (05628c4)

  • Fixed protocoll buffers bug (Repeated Composite Container (5097d6b)

  • Merge branch 'dev'

Hotfix ignore reactions (16a3bc8)

  • hotfix-ignore-reactions (f6e640d)

v0.1.517 (2018-12-18)

Unknown

  • Merged new features for Twitter filter (02c29a8)

  • Added several features for twitter input

  • ignore input sources feature
  • ignore non geotagged option
  • ignore reactions for transfer (92d261a)
  • Added pg application name reporting (5214cc7)

v0.1.516 (2018-12-10)

Unknown

  • Fixed setup.py to include package data; increase version (3590c56)

v0.1.515 (2018-12-06)

Unknown

  • Increase Minor Version Hotfix (9b0f0d3)

  • include VERSION to your MANIFEST.in (3eced07)

v0.1.514 (2018-12-06)

Unknown

v0.1.513 (2018-12-06)

Unknown

v0.1.512 (2018-12-05)

Unknown

  • Minor code optimization, tests (5aa9a1d)

  • Added classes to include in manifest.in instead of setup.py don't use both! (0b73a5c)

v0.1.511 (2018-12-05)

Unknown

  • Fixed setup.py to include classes for pip install (439eeb2)

  • Added new classes to init (39beb4d)

v0.1.510 (2018-12-05)

Unknown

  • Increased version to 0.1.510, updated readme, prepared for pypi (606a2ea)

  • Merge branch 'hot-fix-flickr' into 'master'

Hot Fix for Flickr mapping, tested

See merge request lbsn/lbsntransform!3 (b841817)

  • Hot Fix for Flickr mapping, tested (1346179)

  • Merge Conflicts field mapping flickr (7467151)

  • Merge branch 'flickr-mapping' into 'master'

Flickr mapping (tested function)

See merge request lbsn/lbsntransform!1 (3e6f6ea)

  • Flickr mapping (tested function) (d789978)

  • Cleaned up Flickr Mapping, ready for master (186a832)

  • Fixed some minor bugs after testing on lbsn_test (e7751d9)

  • First stable Flickr mapping (ff61db4)

  • Updated Flickr post mapping.

  • also replaced line endings CRLF to LF only as to better cross os collaboration (5ec719d)

  • Initial Flickr mapping structure (a1f6e54)

  • Initial Flickr Input (bd77cb0)

  • Maintenance work and code style optimization (7fe87a3)

  • Maintenance work and code style optimization (a851b1a)

  • Local input bug fix (d5e6f8c)

  • updated readme (9cb8adf)

  • Cleaned input args (4730eb7)

v0.1.5 (2018-12-03)

Unknown

  • Increased version to 0.1.5 (8c7a407)

  • Fixed single file output bug (c170da5)

  • Refactor CSV and LBSN db into separate classes

  • formatted csv methods to pep guidelines
  • added additional class for proto-lbsn-db mapping
  • some additional cleanups (a431370)

  • updated backup gitignore (7a85383)

  • gitignore fix for vs .sln (3424514)

  • Added support for additional mapping modules (extend Flickr) (49972ab)

  • Updated Timestamp to include native protobuf package; added lbsntransform to PyPi

  • minor modifications to readme and setup.py (7fa200f)

v0.1.4 (2018-07-26)

Unknown

  • Merge branch 'master' of gitlab.vgiscience.de:lbsn/lbsn-twitter-json-mapping (cb3e5f5)

  • Remove remote VS (e75b81c)

  • Remove remote VS (722b4ea)

  • Updated gitignore (c44825f)

  • Updated naming structure to Pep 8 conventions (f066437)

  • Removed VS solution file (8335d4b)

  • Added script execution function through main.py for main package (0dce759)

  • Added script execution function through main.py for main package (c46068a)

  • Merge branch 'master' into refactor-structure (65fe4ee)

  • Added files to gitignore (a645946)

  • Added scripts to git ignore (64ea3a0)

  • Moved closer to python structure convention (2478044)

  • Started to refactor structure (ee15196)

  • Moved Code Files in Subfolder according to Minimal Guide (a82ae46)

  • Updated formatting to python conventions (e15d2ed)

  • Improved use of python code style conventions for main module (8d017c8)

  • Added build/setup. Fixed loop in merge (c360ae6)

  • Removed files from git (5430048)

  • Removed files from git (422cda7)

  • updated git ignore (f46ecdf)

  • Added setup.py, built test wheel (4233a90)

  • Moved to Visual Studio for Dev; Added Files to gitignore (5e50a8b)

  • Added Split Output for large Input Files. Needs testing. (56164f2)

  • Increased version from 0.1.3 to 0.1.4 (76832f9)

  • Working Update of CSV Output/ Write to File (e67c152)

  • Fixed Headers written bug (83a582f)

  • Added sorting & merging for output CSVs (bb9d6d8)

  • Small refactoring for SQL inserts; fixed bug in transferlimit loop (6770f61)

  • Significant refactoring to implement parallel CSV output for faster /Copy Import

  • also refactored ugly select function and dict procedures, now procedural (13ae6ac)

  • Added CSV Output for later COPY FROM file import (00482f7)

  • Added missing 0 (f6570d8)

  • No Transferlimit if transferlimit = 0 (06197dc)

  • Changes to submit routine for new lbsn structure (f515857)

  • Added count affected monitoring (f1a8e99)

  • Fixed fieldMapping Greatest and mergeArray (P)SQLs (deeb854)

  • Changed user submission to coalesce because of overwritten by NULL bug (4067ed6)

  • Added mapping of full relations (m-to-m relationships)

  • for user_groups follows, user_groups_member, user_mentions, user_friend, user_connected (63ce830)
  • Added Relation-Structure to transfer script; Added friends & follows mapping
  • many-to-many relationships can now fully be mapped (e.g. isFriend, isFollwer etc.) (1ee9a26)
  • Fixed memory leak; added memory_leak detection function

  • Fixed a bug that would result in empty object assigned to dict (Passing of Vars) (98a9af7)

  • Tested live & local Input and fixed several smaller bugs included time monitoring class (1562954)

  • Added Geocoding Option for Text Location Strings (ae10cb8)

  • Fixed missing Language for user exception (573fb9a)

  • Fixed bug in Record Merge that would overwrite values with default (44c7897)

  • Added User Groups Structure (9d86bc5)

  • Fixed Nul Error on prepare psycopg2 (e319d58)

  • Added Batching of Insertions for significant speed increase to output db (2d686a9)

  • Fixed bug in Exception Handling of Submit Routine (bb0e56f)

  • Removed comments (60f6918)

  • Removed comments (5256a91)

  • Improved handling of missing UserGuids for some Post (0b82afb)

  • Added Input and processing for stackedJson (no nesting) (3f3d6a6)

  • Fixed bugs in lbsn_dbRetrieve due to CSV/JSON update (75b20f7)

  • Bugfix for retweet parsing (206b65e)

  • Added option to read from local .json/.csv (fc5156a)

  • Improved reporting. Changed 'Reply' to 'Comment' (449ec50)

  • Improved DB Input and Output Loop procedure for increased speed

  • less take on Output DB due to more processing in-memory (b914900)

  • Moved config to separate file (56b810a)

  • Minor refactoring of main (3f9d78c)

  • Removed useless finished from break loop (7d3347d)

  • Removed duplicate = (465140c)

  • Bugfix for assignment of place names and place name alternatives (3bedb7c)

  • Added missing conn.rollback() on exception (bd7c22a)

  • Added function to automatically add languages on insertion exception (e19e254)

  • Significant refactoring of classes.fieldMapping. Mapping of tweet to lbsn Structure should be more logical now (c252419)

  • Removed debug comments (d78f571)

  • Improved Reporting and DB Loop procedures (91a8e2e)

  • Added transfer functions for user, post and post_reaction (5fd7d4f)

  • Added Submit Functions for Place and City (df119bc)

  • Added enhanced Array_Merge for name_alternatives column; added args for startInputID to endInputID (e3cb761)

  • Added bytefiles to git ignore (5d40b08)

  • Removed bytecode from git (7e1b1ce)

  • Added Github boilerplate gitignore (45a2c4c)

  • Added submit entry function for lbsnCountry; tested on lbsn_test (42561bb)

  • Added Author, Version, License; added submitData class (395c2b0)

  • Cleaned up comments (fbc6279)

  • reset default values for args (c885bc0)

  • Added output pw to example config; Added DBRowNumber reporting in main (ead1cbd)

  • Fixed missing Country, Place and City refs in lbsnPost Mapping (85c44d4)

  • Fixed bug in name_alternatives assignment duplicating main 'name' for places (ff1371b)

  • Small refactoring (90095a7)

  • Added basic deep comüpare routine for 2 comparing and merging all values of all fields of two records (98c8e99)

  • Better Count of records, fixed referencing issues (bcfd1de)

  • BugFix in DB Loop Records (81dcc58)

  • Added Composite Structure for storing individual records (c561d01)

  • completed mapping of twitter json to protobuf lbsn structure (2e15c83)

  • Readme formatting (dcadc3d)

  • Readme formatting (2d19778)

  • Readme formatting (24090a0)

  • First version of complete Twitter Tweet mapping to lbsn-protobuf (500d751)

  • Added missing linebreaks in README (f2b1079)

  • Added missing linebreaks in README (6920180)

  • Added missing linebreaks in README (6da73be)

  • Added missing linebreaks in README (0c1a786)

  • Added README (96695a9)

  • Added gitignore; implemented lbsn protobuf structure; restructured code to protobuf defenitions - everything up to postReaction done (2bb2c2d)

  • Added LBSN ProtoBuf Structure 0.1.5 (25bbb75)

  • Initial Commit (9c792fb)