[Bug]: inverted index does not support string longer than 65530

[sunby]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

If you insert strings longer than 65530, milvus will not return warnings or errors but tantivy's log will print warning.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[zhuwenxing]

insert strings longer than 65530

Is this referring to the length of the entire string or the length of a single word?

[sunby]

insert strings longer than 65530

Is this referring to the length of the entire string or the length of a single word?

it's a const variable MAX_TOKEN_LEN in tantivy, so I think it's a word.

[sunby]

I will do some tests to check what's the influence for query using inverted index.

[sunby]

/assign

[xiaofan-luan]

how can a token be that long?

We do need to tune the max varchar length field. is there anything stop us from increasing varchar to 256K or 1M?

[yanliang567]

/assign @sunby
/unassign

[sunby]

how can a token be that long?

We do need to tune the max varchar length field. is there anything stop us from increasing varchar to 256K or 1M?

We tested inverted index with 65535 length string and this warning occured.

[sunby]

I write an unit test to verify it. And strings longer than 65530 can not be searched because they are dropped in tantivy.

https://github.com/quickwit-oss/tantivy/blob/c71ea7b2effe6494c1a2b9e234db71f89255c06a/src/postings/postings_writer.rs#L139-L148

[sunby]

how can a token be that long?

We do need to tune the max varchar length field. is there anything stop us from increasing varchar to 256K or 1M?

We use "raw" tokenizer which means no tokenizer in tantivy.

[xiaofan-luan]

how can a token be that long?
We do need to tune the max varchar length field. is there anything stop us from increasing varchar to 256K or 1M?

We use "raw" tokenizer which means no tokenizer in tantivy.

This seems to be a non blocker issue.

Is there a blocking issue if we want to grow the size of varchar to 256k? like we use some smaller bits for a size