[Bug]: [benchmark][standalone] search raises error partition not loaded after partition loaded, and search raises error collection not loaded after partition released in multi-partition scene
#37849
Assignees
Labels
kind/bug
Issues or changes related a bug
test/benchmark
benchmark test
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Milestone
Comments
wangting0128
added
kind/bug
|
/assign @czs007 |
yanliang567
added
triage/accepted
|
It is currently suspected that the collection object does not exist when QN try to load the partition, rendering the call to add partitions ineffective. |
|
The reference count of the collection on QN is constantly fluctuating, during which there is a period when the collection does not exist. |
|
The problem here is not proxy cache. it's querycoord trying to add extra check on GetShardLeadersWithChannels. To me, this is check is fully unnecessary. Querycoord should just return shard delegator's location to proxy and proxy should send request to the right delegator and let delegator itself should be responsible to hold request if they are not ready. Try to think in a more succinct way of who should be responsible of doing what is very important. Right now we add too many temporary logic to fix corner cases and make the system very hard to understand. |
From the error message, it seems that this is not related to querynode but more like a issue of querycoord GetShardLeaders function. |
same root cause，different errorargo task: multi-vector-corn-1-1732197600 server: client log: test steps: |
same root cause，different errorargo task：multi-vector-corn-1-1732197600 server: client log: test steps: |
|
@czs007 @weiliu1031 please help check ~ |
Maybe the root cause is the sameargo task: fouramf-bitmap-scenes-5xlvn server: client log: test steps: |
reproduceargo task: fouramf-bitmap-scenes-5xlvn server: client log: test steps: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
argo task: fouramf-bitmap-scenes-h5n9j/
test case name: test_bitmap_locust_dml_partitions_standalone
server:
search_failed.log
expect：
step 3: search not failed
step 6: search raises error
partition not loaded{pod=~"fouramf-bitmap-scenes-h5n9j-6-milvus-standalone-6db4bb95cdnq5lk"} |~ "e6d76ba73466916c6d9d58288528e6e8|scene_test_partition_MmORR0U8"Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS :purpose: `primary key: INT64`, divided into 10 partitions 1. building `BITMAP` index on all supported 12 scalar fields 2. 2 fields of different vector types 3. load and search partial partitions & DML(upsert) requests :test steps: 1. create collection with fields: 'float_vector': 128dim 'float_vector_1': 768dim 'id': primary key type is INT64 all scalar fields: varchar max_length=100, array max_capacity=13 2. build indexes: IVF_SQ8: 'float_vector' HNSW: 'float_vector_1' BITMAP: all scalar fields 3. insert 5 million data 4. flush collection 5. build indexes again using the same params 6. load collection 7. concurrent request: - scene_insert_partition (partition: create->insert->flush->release->drop) - scene_test_partition (partition: create->insert->flush->index again->load->search->release->search failed->drop) <- search raises error after loading - scene_test_partition_hybrid_search (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop) - release_partitions: 10 prepared partitions - upsert: batch=1Milvus Log
No response
Anything else?
test result:
The text was updated successfully, but these errors were encountered: