Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Geospatial Data Type and GIS Function Support for milvus #37124

Closed
wants to merge 2 commits into from
Closed

feat: Geospatial Data Type and GIS Function Support for milvus #37124

wants to merge 2 commits into from

Conversation

tasty-gumi
Copy link
Contributor

@tasty-gumi tasty-gumi commented Oct 25, 2024 •
edited
Loading

issue:#27576

Main Goals

  1. Create and describe collections with geospatial fields, enabling both client and server to recognize and process geo fields.
  2. Insert geospatial data as payload values in the insert binlog, and print the values for verification.
  3. Load segments containing geospatial data into memory.
  4. Ensure query outputs can display geospatial data.
  5. Support filtering on GIS functions for geospatial columns.

Solution

  1. Add Type: Modify the Milvus core by adding a Geospatial type in both the C++ and Go code layers, defining the Geospatial data structure and the corresponding interfaces.
  2. Dependency Libraries: Introduce necessary geospatial data processing libraries. In the C++ source code, use Conan package management to include the GDAL library. In the Go source code, add the go-geom library to the go.mod file.
  3. Protocol Interface: Revise the Milvus protocol to provide mechanisms for Geospatial message serialization and deserialization.
  4. Data Pipeline: Facilitate interaction between the client and proxy using the WKT format for geospatial data. The proxy will convert all data into WKB format for downstream processing, providing column data interfaces, segment encapsulation, segment loading, payload writing, and cache block management.
  5. Query Operators: Implement simple display and support for filter queries. Initially, focus on filtering based on spatial relationships for a single column of geospatial literal values, providing parsing and execution for query expressions.
  6. Index Construction: Consider building an H3 index, utilizing the C interface provided by the H3 system.
  7. Client Modification: Enable the client to handle user input for geospatial data and facilitate end-to-end testing.Check the modification in pymilvus.

new pr of Geospatial data type , delete incomplete H3 Index development and useless generated files.

@tasty-gumi
initial code for geo
9f36a9b 
add geospatial interface in src common

change type define and add segcore support

add storage & chunkdata support

feature: go package storage & proxy & typeutil support geospatial type in internal and typeutil in pkg

Signed-off-by: tasty-gumi <[email protected]>

add geospatial interface in src common

change type define and add segcore support

change: use wkb only in core

Signed-off-by: tasty-gumi <[email protected]>

fix:the geospatial only use std::string as FieldDataImpl template paramters && add geospatial data generation && pass chunk ,growing , sealed test

fix : merge confilcts after rebase ,test nullable not pass due to upstream

feat:basic GIS Function expr and visitor impl and GIS proto support && add:storage test of geo data

Signed-off-by: tasty-gumi <[email protected]>

feat:add proxy validate (pass httpserver test) && plan parser of geospatialfunction

fix:sealedseg && go tidy

fix:go mod

feat:can produce wkt result for pymilvus client

feat: add parser and query operator for geos filed && print geos binlog as wkt

fix:fielddataimpl interface
Signed-off-by: tasty-gumi <[email protected]>

fix: some format of code && segmentfault debug for rebase

Signed-off-by: tasty-gumi <[email protected]>

add: import util test for parquet and mix compaction test

Signed-off-by: tasty-gumi <[email protected]>
@tasty-gumi
fix: delete useless file and fix error for rebase
b2cd628 
Signed-off-by: tasty-gumi <[email protected]>
@sre-ci-robot sre-ci-robot added area/dependency Pull requests that update a dependency file area/test sig/testing test/integration integration test size/XXL Denotes a PR that changes 1000+ lines. labels Oct 25, 2024
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tasty-gumi
To complete the pull request process, please assign wxyucs after the PR has been reviewed.
You can assign the PR to them by writing /assign @wxyucs in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Oct 25, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify mergify bot mentioned this pull request Oct 25, 2024
Merged
@tasty-gumi tasty-gumi closed this Oct 25, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 25, 2024

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@cydrain cydrain Awaiting requested review from cydrain

Assignees
No one assigned
Labels
area/dependency Pull requests that update a dependency file area/test dco-passed DCO check passed. kind/feature Issues related to feature request from users sig/testing size/XXL Denotes a PR that changes 1000+ lines. test/integration integration test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tasty-gumi @sre-ci-robot