Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: panic: runtime error: invalid memory address or nil pointer dereference #31925

Closed
1 task done
haliliceylan opened this issue Apr 5, 2024 · 28 comments
Closed
1 task done
Assignees
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@haliliceylan
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: v2.4.0-rc.1-dev 
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): (default - not changed)
- SDK version(e.g. pymilvus v2.0.0rc2): no sdk
- OS(Ubuntu or CentOS): MacOS 13.6.4 - Docker Server 25.0.5
- CPU/Memory: M2 Pro, 8 GB Ram (ARM)
- GPU: -
- Others: -

Current Behavior

I was using the milvusdb/milvus:v2.3.2 container as a standalone setup locally, and it was working perfectly. However, when I decided to upgrade to version 2.4.0, I encountered some issues. I followed the installation instructions for the standalone version from the official Milvus website at https://milvus.io/docs/install_standalone-docker.md. Unfortunately, I received a 'nil pointer dereference' error during the process. I suspect this error might be related to the embedded etcd component.

Expected Behavior

It should work with default settings.

Steps To Reproduce

1. Open https://milvus.io/docs/install_standalone-docker.md
2. Run wget https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh
3. Run bash standalone_embed.sh start
4. You will see constant message "Wait for Milvus Starting..."
5. Check docker container status and logs.

Milvus Log

WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240405 11:31:40.932515 1 hook.cpp:194] [SERVER][bitset_hook][milvus] bitset hook simd type: REF
I20240405 11:31:40.932624 1 hook.cpp:262] [SERVER][all_boolean_hook][milvus] AllFalse/AllTrue hook simd type: NEON
I20240405 11:31:40.932633 1 hook.cpp:280] [SERVER][invert_boolean_hook][milvus] InvertBoolean hook simd type: NEON
I20240405 11:31:40.932637 1 hook.cpp:308] [SERVER][logical_boolean_hook][milvus] InvertBoolean hook simd type: NEON
I20240405 11:31:40.932641 1 hook.cpp:242] [SERVER][find_term_hook][milvus] find term hook simd type: REF
I20240405 11:31:40.932646 1 hook.cpp:336] [SERVER][equal_val_hook][milvus] equal val hook simd type: REF
I20240405 11:31:40.932648 1 hook.cpp:357] [SERVER][less_val_hook][milvus] less than val hook simd type:REF
I20240405 11:31:40.932652 1 hook.cpp:378] [SERVER][greater_val_hook][milvus] greater than val hook simd type: REF
I20240405 11:31:40.932657 1 hook.cpp:399] [SERVER][less_equal_val_hook][milvus] less equal than val hook simd type: REF
I20240405 11:31:40.932667 1 hook.cpp:420] [SERVER][greater_equal_val_hook][milvus] greater equal than val hook simd type: REF
I20240405 11:31:40.932670 1 hook.cpp:441] [SERVER][not_equal_val_hook][milvus] not equal val hook simd type: REF
I20240405 11:31:40.932673 1 hook.cpp:462] [SERVER][equal_col_hook][milvus] equal column hook simd type:REF
I20240405 11:31:40.932677 1 hook.cpp:483] [SERVER][less_col_hook][milvus] less than column hook simd type:REF
I20240405 11:31:40.932680 1 hook.cpp:504] [SERVER][greater_col_hook][milvus] greater than column hook simd type:REF
I20240405 11:31:40.932684 1 hook.cpp:525] [SERVER][less_equal_col_hook][milvus] less equal than column hook simd type: REF
I20240405 11:31:40.932687 1 hook.cpp:546] [SERVER][greater_equal_col_hook][milvus] greater equal than column hook simd type:REF
I20240405 11:31:40.932690 1 hook.cpp:567] [SERVER][not_equal_col_hook][milvus] not equal column hook simd type: REF
2024/04/05 11:31:41 maxprocs: Leaving GOMAXPROCS=10: CPU quota undefined

__  _________ _   ____  ______    

/ |/ / / /| | / / / / / __/
/ /|
/ // // /| |/ / // /\ \
// /////_/__/

Welcome to use Milvus!
Version: v2.4.0-rc.1-dev
Built: Wed Mar 20 11:08:35 UTC 2024
GitCommit: f4449d4
GoVersion: go version go1.22.1 linux/arm64

TotalMem: 8318390272
UsedMem: 25427968

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2024/04/05 11:31:41.012 +00:00] [INFO] [roles/roles.go:304] ["starting running Milvus components"]
[2024/04/05 11:31:41.012 +00:00] [INFO] [roles/roles.go:167] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/04/05 11:31:41.015 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/04/05 11:31:41.015 +00:00] [DEBUG] [server/global_rmq.go:61] ["Close Rocksmq!"]
[2024/04/05 11:31:41.016 +00:00] [INFO] [roles/roles.go:280] ["All cleanup done, handleSignals goroutine quit"]
[2024/04/05 11:31:41.016 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x2706e74]

goroutine 1 gp=0x40000021c0 m=8 mp=0x400061a008 [running]:
panic({0x49b4380?, 0x7632d90?})
/usr/local/go/src/runtime/panic.go:779 +0x140 fp=0x40011cd430 sp=0x40011cd380 pc=0x1b7d3d0
runtime.panicmem(...)
/usr/local/go/src/runtime/panic.go:261
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:881 +0x308 fp=0x40011cd490 sp=0x40011cd430 pc=0x1b99b58
github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer.func1()
/root/milvus/pkg/util/etcd/etcd_server.go:49 +0xb4 fp=0x40011cd650 sp=0x40011cd4a0 pc=0x2706e74
sync.(*Once).doSlow(0x40011cd728?, 0x450a54c?)
/usr/local/go/src/sync/once.go:74 +0x100 fp=0x40011cd6b0 sp=0x40011cd650 pc=0x1be3770
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer(0x20?, {0x4001099dd0?, 0x4f867a8?}, {0x4001092ed0?, 0x40011cd7b8?}, {0x4000c47058?, 0x40011cd7b8?}, {0x4000c47048?, 0x40011cd7b8?})
/root/milvus/pkg/util/etcd/etcd_server.go:37 +0x9c fp=0x40011cd730 sp=0x40011cd6b0 pc=0x2706d3c
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0x40001185f0)
/root/milvus/cmd/roles/roles.go:333 +0x344 fp=0x40011cdbd0 sp=0x40011cd730 pc=0x450a574
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x781e3c0, {0x4000124960, 0x3, 0x3}, 0x4000ae07e0)
/root/milvus/cmd/milvus/run.go:47 +0x254 fp=0x40011cdc90 sp=0x40011cdbd0 pc=0x4513c94
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0x4000124960, 0x3, 0x3})
/root/milvus/cmd/milvus/milvus.go:60 +0x22c fp=0x40011cdd00 sp=0x40011cdc90 pc=0x45139bc
main.main()
/root/milvus/cmd/main.go:95 +0x370 fp=0x40011cdf40 sp=0x40011cdd00 pc=0x45179a0
runtime.main()
/usr/local/go/src/runtime/proc.go:271 +0x28c fp=0x40011cdfd0 sp=0x40011cdf40 pc=0x1b8172c
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40011cdfd0 sp=0x40011cdfd0 pc=0x1bbdb74

goroutine 18 gp=0x4000102380 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000080790 sp=0x4000080770 pc=0x1b81b88
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
/usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0x40000807d0 sp=0x4000080790 pc=0x1b819e8
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40000807d0 sp=0x40000807d0 pc=0x1bbdb74
created by runtime.init.6 in goroutine 1
/usr/local/go/src/runtime/proc.go:314 +0x24

goroutine 19 gp=0x4000102540 m=nil [GC sweep wait]:
runtime.gopark(0x7794c01?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000080f60 sp=0x4000080f40 pc=0x1b81b88
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0x4000110000)
/usr/local/go/src/runtime/mgcsweep.go:318 +0x108 fp=0x4000080fb0 sp=0x4000080f60 pc=0x1b68d88
runtime.gcenable.gowrap1()
/usr/local/go/src/runtime/mgc.go:203 +0x28 fp=0x4000080fd0 sp=0x4000080fb0 pc=0x1b5cd18
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000080fd0 sp=0x4000080fd0 pc=0x1bbdb74
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:203 +0x6c

goroutine 20 gp=0x4000102700 m=nil [GC scavenge wait]:
runtime.gopark(0x7799b80?, 0x55487c8?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000081760 sp=0x4000081740 pc=0x1b81b88
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x7799b80)
/usr/local/go/src/runtime/mgcscavenge.go:425 +0x5c fp=0x4000081790 sp=0x4000081760 pc=0x1b665ec
runtime.bgscavenge(0x4000110000)
/usr/local/go/src/runtime/mgcscavenge.go:658 +0xac fp=0x40000817b0 sp=0x4000081790 pc=0x1b66bbc
runtime.gcenable.gowrap2()
/usr/local/go/src/runtime/mgc.go:204 +0x28 fp=0x40000817d0 sp=0x40000817b0 pc=0x1b5ccb8
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40000817d0 sp=0x40000817d0 pc=0x1bbdb74
created by runtime.gcenable in goroutine 1
/usr/local/go/src/runtime/mgc.go:204 +0xac

goroutine 21 gp=0x4000102c40 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x50de410?, 0xa0?, 0xe3?, 0x2000000020?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000084580 sp=0x4000084560 pc=0x1b81b88
runtime.runfinq()
/usr/local/go/src/runtime/mfinal.go:194 +0x108 fp=0x40000847d0 sp=0x4000084580 pc=0x1b5bd68
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40000847d0 sp=0x40000847d0 pc=0x1bbdb74
created by runtime.createfing in goroutine 1
/usr/local/go/src/runtime/mfinal.go:164 +0x80

goroutine 22 gp=0x4000102e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000081f30 sp=0x4000081f10 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x4000081fd0 sp=0x4000081f30 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000081fd0 sp=0x4000081fd0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 34 gp=0x400061c380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400062e730 sp=0x400062e710 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x400062e7d0 sp=0x400062e730 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062e7d0 sp=0x400062e7d0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 35 gp=0x400061c540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400062ef30 sp=0x400062ef10 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x400062efd0 sp=0x400062ef30 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062efd0 sp=0x400062efd0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 36 gp=0x400061c700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400062f730 sp=0x400062f710 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x400062f7d0 sp=0x400062f730 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062f7d0 sp=0x400062f7d0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 37 gp=0x400061c8c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400062ff30 sp=0x400062ff10 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x400062ffd0 sp=0x400062ff30 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062ffd0 sp=0x400062ffd0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 38 gp=0x400061ca80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000630730 sp=0x4000630710 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x40006307d0 sp=0x4000630730 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40006307d0 sp=0x40006307d0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 39 gp=0x400061cc40 m=nil [GC worker (idle)]:
runtime.gopark(0x38c19618f0cb?, 0x1?, 0xa?, 0xe1?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000630f30 sp=0x4000630f10 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x4000630fd0 sp=0x4000630f30 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000630fd0 sp=0x4000630fd0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 40 gp=0x400061ce00 m=nil [GC worker (idle)]:
runtime.gopark(0x7821900?, 0x1?, 0xe8?, 0x87?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000631730 sp=0x4000631710 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x40006317d0 sp=0x4000631730 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40006317d0 sp=0x40006317d0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 41 gp=0x400061cfc0 m=nil [GC worker (idle)]:
runtime.gopark(0x38c196173ceb?, 0x1?, 0xef?, 0xee?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000631f30 sp=0x4000631f10 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x4000631fd0 sp=0x4000631f30 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000631fd0 sp=0x4000631fd0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 42 gp=0x400061d180 m=nil [GC worker (idle)]:
runtime.gopark(0x38c194d0dc4b?, 0x0?, 0x0?, 0x0?, 0x0?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400062a730 sp=0x400062a710 pc=0x1b81b88
runtime.gcBgMarkWorker()
/usr/local/go/src/runtime/mgc.go:1310 +0xd8 fp=0x400062a7d0 sp=0x400062a730 pc=0x1b5ee08
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062a7d0 sp=0x400062a7d0 pc=0x1bbdb74
created by runtime.gcBgMarkStartWorkers in goroutine 1
/usr/local/go/src/runtime/mgc.go:1234 +0x28

goroutine 23 gp=0x4000ab6000 m=nil [select]:
runtime.gopark(0x4000087f58?, 0x2?, 0x18?, 0x7e?, 0x4000087f24?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000087dd0 sp=0x4000087db0 pc=0x1b81b88
runtime.selectgo(0x4000087f58, 0x4000087f20, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x608 fp=0x4000087ee0 sp=0x4000087dd0 pc=0x1b95638
github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0x4000ab21e0, {0x5596578, 0x4000aa2050})
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:83 +0xb8 fp=0x4000087fa0 sp=0x4000087ee0 pc=0x3443e08
github.com/panjf2000/ants/v2.(*Pool).goPurge.gowrap1()
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0x30 fp=0x4000087fd0 sp=0x4000087fa0 pc=0x3444340
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000087fd0 sp=0x4000087fd0 pc=0x1bbdb74
created by github.com/panjf2000/ants/v2.(*Pool).goPurge in goroutine 1
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0xc8

goroutine 24 gp=0x4000ab61c0 m=nil [select]:
runtime.gopark(0x4000099f58?, 0x2?, 0x18?, 0x9e?, 0x4000099f2c?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000099dd0 sp=0x4000099db0 pc=0x1b81b88
runtime.selectgo(0x4000099f58, 0x4000099f28, 0x0?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x608 fp=0x4000099ee0 sp=0x4000099dd0 pc=0x1b95638
github.com/panjf2000/ants/v2.(*Pool).ticktock(0x4000ab21e0, {0x5596578, 0x4000aa20a0})
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:125 +0xf8 fp=0x4000099fa0 sp=0x4000099ee0 pc=0x3444128
github.com/panjf2000/ants/v2.(*Pool).goTicktock.gowrap1()
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x30 fp=0x4000099fd0 sp=0x4000099fa0 pc=0x34444d0
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000099fd0 sp=0x4000099fd0 pc=0x1bbdb74
created by github.com/panjf2000/ants/v2.(*Pool).goTicktock in goroutine 1
/root/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x100

goroutine 167 gp=0x4000fac380 m=7 mp=0x4000101008 [syscall]:
runtime.notetsleepg(0x781fe00, 0xffffffffffffffff)
/usr/local/go/src/runtime/lock_futex.go:246 +0x34 fp=0x400062b790 sp=0x400062b760 pc=0x1b4d174
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:152 +0x30 fp=0x400062b7b0 sp=0x400062b790 pc=0x1bb9600
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x1c fp=0x400062b7d0 sp=0x400062b7b0 pc=0x1c88b7c
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400062b7d0 sp=0x400062b7d0 pc=0x1bbdb74
created by os/signal.Notify.func1.1 in goroutine 1
/usr/local/go/src/os/signal/signal.go:151 +0x28

goroutine 166 gp=0x4000ab7c00 m=nil [select, locked to thread]:
runtime.gopark(0x4000086fa0?, 0x2?, 0x88?, 0x6e?, 0x4000086f8c?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x4000086e30 sp=0x4000086e10 pc=0x1b81b88
runtime.selectgo(0x4000086fa0, 0x4000086f88, 0x0?, 0x0, 0x4000ae0501?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x608 fp=0x4000086f40 sp=0x4000086e30 pc=0x1b95638
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal_unix.go:1034 +0x198 fp=0x4000086fd0 sp=0x4000086f40 pc=0x1bb2c28
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000086fd0 sp=0x4000086fd0 pc=0x1bbdb74
created by runtime.ensureSigM in goroutine 1
/usr/local/go/src/runtime/signal_unix.go:1017 +0xd8

goroutine 169 gp=0x4000f9bdc0 m=nil [select]:
runtime.gopark(0x400105ef48?, 0x2?, 0xa0?, 0x7d?, 0x400105ee24?)
/usr/local/go/src/runtime/proc.go:402 +0xc8 fp=0x400105ecd0 sp=0x400105ecb0 pc=0x1b81b88
runtime.selectgo(0x400105ef48, 0x400105ee20, 0x40012cc140?, 0x0, 0x0?, 0x1)
/usr/local/go/src/runtime/select.go:327 +0x608 fp=0x400105ede0 sp=0x400105ecd0 pc=0x1b95638
github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0x40001187d0, {0x4f867da, 0xa})
/root/milvus/pkg/config/refresher.go:69 +0x180 fp=0x400105efa0 sp=0x400105ede0 pc=0x2781410
github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.gowrap1()
/root/milvus/pkg/config/refresher.go:51 +0x30 fp=0x400105efd0 sp=0x400105efa0 pc=0x2781190
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400105efd0 sp=0x400105efd0 pc=0x1bbdb74
created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1 in goroutine 1
/root/milvus/pkg/config/refresher.go:51 +0x98

goroutine 170 gp=0x4001646000 m=nil [runnable]:
github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.gowrap1()
/root/milvus/pkg/config/refresher.go:51 fp=0x40000837d0 sp=0x40000837d0 pc=0x2781160
runtime.goexit({})
/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x40000837d0 sp=0x40000837d0 pc=0x1bbdb74
created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1 in goroutine 1
/root/milvus/pkg/config/refresher.go:51 +0x98

Anything else?

I suspect its about embeded etcd server bc of this log line : github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer(0x20?, {0x4001099dd0?, 0x4f867a8?}, {0x4001092ed0?

Also, since I am testing it locally, there is no data persistence, so I do not expect there to be data incompatibility between versions.

@haliliceylan haliliceylan added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 5, 2024
@haliliceylan
Copy link
Author

The problem stemmed from the milvus directory; the directory "/var/lib/milvus" did not exist, which led to the error. By manually creating the directory with the command "mkdir -p /var/lib/milvus/etcd," I was able to resolve the issue. Although I did not require data persistence and hence was not using volumes, this solution worked, and Milvus is now functioning correctly.

Thank you.

@pooya-mohammadi
Copy link

@haliliceylan I tried your approach however it didn't work?
Which version did you try?

@xiaofan-luan
Copy link
Collaborator

@haliliceylan I tried your approach however it didn't work? Which version did you try?

what is your milvus release and how did you deploy it? any config you changed when you delopy?

@pooya-mohammadi
Copy link

Thank you for responding @xiaofan-luan

curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
sudo bash standalone_embed.sh start

I removed volumes and also created the one mentioned by @haliliceylan
still the same error

@xiaofan-luan
Copy link
Collaborator

what is your milvus.yaml? and you get same error [2024/04/05 11:31:41.016 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
panic: runtime error: invalid memory address or nil pointer dereference?

@xiaofan-luan
Copy link
Collaborator

if it's same line, it probably means you don't put config file ,milvus.yaml into the right place

@pooya-mohammadi
Copy link

@xiaofan-luan I don't have a milvus.yaml file that's why I remove the volumes for them

@pooya-mohammadi
Copy link

@xiaofan-luan

panic: runtime error: invalid memory address or nil pointer dereference

After running:

docker run         --name milvus-standalone         --security-opt seccomp:unconfined         -e ETCD_USE_EMBED=true         -e ETCD_DATA_DIR=/var/lib/milvus/etcd         -e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml         -e COMMON_STORAGETYPE=local         -p 19530:19530         -p 9091:9091         -p 2379:2379         --health-cmd="curl -f http://localhost:9091/healthz"         --health-interval=30s         --health-start-period=90s         --health-timeout=20s         --health-retries=3         --rm         milvusdb/milvus:v2.4.5         milvus run standalone

@pooya-mohammadi
Copy link

@xiaofan-luan same error for this input as well:

docker run \
        --name milvus-standalone \
        --security-opt seccomp:unconfined \
        -e ETCD_USE_EMBED=true \
        -e ETCD_DATA_DIR=/var/lib/milvus/etcd \
        -e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml \
        -e COMMON_STORAGETYPE=local \
        -p 19530:19530 \
        -p 9091:9091 \
        -p 2379:2379 \
        -v $(pwd)/volumes/milvus:/var/lib/milvus \
        -v $(pwd)/embedEtcd.yaml:/milvus/configs/embedEtcd.yaml \
        -v $(pwd)/user.yaml:/milvus/configs/user.yaml \
        --health-cmd="curl -f http://localhost:9091/healthz" \
        --health-interval=30s \
        --health-start-period=90s \
        --health-timeout=20s \
        --health-retries=3 \
        --rm \
        milvusdb/milvus:v2.4.5 \
        milvus run standalone

@xiaofan-luan
Copy link
Collaborator

@xiaofan-luan

panic: runtime error: invalid memory address or nil pointer dereference

After running:

docker run         --name milvus-standalone         --security-opt seccomp:unconfined         -e ETCD_USE_EMBED=true         -e ETCD_DATA_DIR=/var/lib/milvus/etcd         -e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml         -e COMMON_STORAGETYPE=local         -p 19530:19530         -p 9091:9091         -p 2379:2379         --health-cmd="curl -f http://localhost:9091/healthz"         --health-interval=30s         --health-start-period=90s         --health-timeout=20s         --health-retries=3         --rm         milvusdb/milvus:v2.4.5         milvus run standalone

can you make sure your docker all three files
-v $(pwd)/volumes/milvus:/var/lib/milvus
-v $(pwd)/embedEtcd.yaml:/milvus/configs/embedEtcd.yaml
-v $(pwd)/user.yaml:/milvus/configs/user.yaml \

@xiaofan-luan
Copy link
Collaborator

I guess the directory you run this scripts maybe you don't have write access or waht.
And please use docker logs -f dockerid to get full logs

@pooya-mohammadi
Copy link

sudo docker run \
        --name milvus-standalone \
        --security-opt seccomp:unconfined \
        -e ETCD_USE_EMBED=true \
        -e ETCD_DATA_DIR=/var/lib/milvus/etcd \
        -e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml \
        -e COMMON_STORAGETYPE=local \
        -p 19530:19530 \
        -p 9091:9091 \
        -p 2379:2379 \
        -v $(pwd)/volumes/milvus:/var/lib/milvus \
        -v $(pwd)/embedEtcd.yaml:/milvus/configs/embedEtcd.yaml \
        -v $(pwd)/user.yaml:/milvus/configs/user.yaml \
        --health-cmd="curl -f http://localhost:9091/healthz" \
        --health-interval=30s \
        --health-start-period=90s \
        --health-timeout=20s \
        --health-retries=3 \
        --rm \
        milvusdb/milvus:v2.4.5 \
        milvus run standalone
[sudo] password for aicvi: 
Sorry, try again.
[sudo] password for aicvi: 
[2024/10/09 14:38:52.424 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2024/10/09 14:38:52 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined

    __  _________ _   ____  ______    
   /  |/  /  _/ /| | / / / / / __/    
  / /|_/ // // /_| |/ / /_/ /\ \    
 /_/  /_/___/____/___/\____/___/     

Welcome to use Milvus!
Version:   v2.4.5
Built:     Wed Jun 19 02:47:56 UTC 2024
GitCommit: 60695bdb
GoVersion: go version go1.20.7 linux/amd64

TotalMem: 134758846464
UsedMem: 29884416

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2024/10/09 14:38:52.426 +00:00] [INFO] [roles/roles.go:307] ["starting running Milvus components"]
[2024/10/09 14:38:52.426 +00:00] [INFO] [roles/roles.go:170] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/10/09 14:38:52.432 +00:00] [INFO] [config/manager.go:301] ["Get configuration by items failed"] [error="Read config failed: /milvus/configs/user.yaml: read /milvus/configs/user.yaml: is a directory"]
[2024/10/09 14:38:52.432 +00:00] [WARN] [paramtable/base_table.go:175] ["init baseTable with file failed"] [configFile="[milvus.yaml,default.yaml,user.yaml]"] [error="failed to load FileSource cause: 5265616420636f6e666967206661696c65643a202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2072656164202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2069732061206469726563746f7279"]
[2024/10/09 14:38:52.434 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/10/09 14:38:52.434 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/10/09 14:38:52.434 +00:00] [DEBUG] [nmq/nmq_server.go:106] ["Closing Natsmq!"]
[2024/10/09 14:38:52.435 +00:00] [INFO] [roles/roles.go:283] ["All cleanup done, handleSignals goroutine quit"]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x28dd5c1]

goroutine 1 [running]:
panic({0x4f041a0, 0x7af1fb0})
	/usr/local/go/src/runtime/panic.go:987 +0x3bb fp=0xc0005a54b8 sp=0xc0005a53f8 pc=0x1c2df5b
runtime.panicmem(...)
	/usr/local/go/src/runtime/panic.go:260
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:841 +0x37d fp=0xc0005a5518 sp=0xc0005a54b8 pc=0x1c4625d
github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer.func1()
	/go/src/github.com/milvus-io/milvus/pkg/util/etcd/etcd_server.go:49 +0xe1 fp=0xc0005a56c8 sp=0xc0005a5518 pc=0x28dd5c1
sync.(*Once).doSlow(0xc0005a5798?, 0x2aa94c6?)
	/usr/local/go/src/sync/once.go:74 +0xc2 fp=0xc0005a5728 sp=0xc0005a56c8 pc=0x1c89162
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
github.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer(0x80?, {0xc00005ae10?, 0x5553363?}, {0xc00057e5b8?, 0xc00141f808?}, {0x554a595?, 0xc0000061a0?}, {0x55406a6?, 0xc0000061a0?})
	/go/src/github.com/milvus-io/milvus/pkg/util/etcd/etcd_server.go:37 +0xcc fp=0xc0005a57a8 sp=0xc0005a5728 pc=0x28dd44c
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0003fe4b0)
	/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:336 +0x3ea fp=0xc0005a5c78 sp=0xc0005a57a8 pc=0x49dd96a
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0x0?, {0xc00019a150?, 0x3, 0x3}, 0xc0004416e0)
	/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47 +0x2e5 fp=0xc0005a5d48 sp=0xc0005a5c78 pc=0x49e9e05
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc00019a150?, 0x3, 0x3})
	/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x20e fp=0xc0005a5dc0 sp=0xc0005a5d48 pc=0x49e9a8e
main.main()
	/go/src/github.com/milvus-io/milvus/cmd/main.go:95 +0x3e5 fp=0xc0005a5f80 sp=0xc0005a5dc0 pc=0x49ee7c5
runtime.main()
	/usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0005a5fe0 sp=0xc0005a5f80 pc=0x1c310a7
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0005a5fe8 sp=0xc0005a5fe0 pc=0x1c67f01

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bcfb0 sp=0xc0000bcf90 pc=0x1c314d6
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc0000bcfe0 sp=0xc0000bcfb0 pc=0x1c31310
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bcfe8 sp=0xc0000bcfe0 pc=0x1c67f01
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bd780 sp=0xc0000bd760 pc=0x1c314d6
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000bd7c8 sp=0xc0000bd780 pc=0x1c19b9e
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000bd7e0 sp=0xc0000bd7c8 pc=0x1c0eba6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bd7e8 sp=0xc0000bd7e0 pc=0x1c67f01
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000e6000?, 0x5b2d200?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bdf70 sp=0xc0000bdf50 pc=0x1c314d6
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x7c3d960)
	/usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc0000bdfa0 sp=0xc0000bdf70 pc=0x1c179d3
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc0000bdfc8 sp=0xc0000bdfa0 pc=0x1c17fe5
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc0000bdfe0 sp=0xc0000bdfc8 pc=0x1c0eb46
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bdfe8 sp=0xc0000bdfe0 pc=0x1c67f01
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 18 [finalizer wait]:
runtime.gopark(0x0?, 0x56b52b8?, 0xc0?, 0x42?, 0x2000000020?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bc628 sp=0xc0000bc608 pc=0x1c314d6
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000bc7e0 sp=0xc0000bc628 pc=0x1c0db67
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bc7e8 sp=0xc0000bc7e0 pc=0x1c67f01
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000b8750 sp=0xc0000b8730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000b87e0 sp=0xc0000b8750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b87e8 sp=0xc0000b87e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000be750 sp=0xc0000be730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000be7e0 sp=0xc0000be750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000be7e8 sp=0xc0000be7e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000114750 sp=0xc000114730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001147e0 sp=0xc000114750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001147e8 sp=0xc0001147e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000b8f50 sp=0xc0000b8f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000b8fe0 sp=0xc0000b8f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b8fe8 sp=0xc0000b8fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000114f50 sp=0xc000114f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000114fe0 sp=0xc000114f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000114fe8 sp=0xc000114fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000115750 sp=0xc000115730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001157e0 sp=0xc000115750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001157e8 sp=0xc0001157e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000115f50 sp=0xc000115f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000115fe0 sp=0xc000115f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000116750 sp=0xc000116730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001167e0 sp=0xc000116750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001167e8 sp=0xc0001167e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000110750 sp=0xc000110730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001107e0 sp=0xc000110750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001107e8 sp=0xc0001107e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000110f50 sp=0xc000110f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000110fe0 sp=0xc000110f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000110fe8 sp=0xc000110fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000111750 sp=0xc000111730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001117e0 sp=0xc000111750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001117e8 sp=0xc0001117e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000111f50 sp=0xc000111f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000111fe0 sp=0xc000111f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000112750 sp=0xc000112730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001127e0 sp=0xc000112750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001127e8 sp=0xc0001127e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000112f50 sp=0xc000112f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000112fe0 sp=0xc000112f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000112fe8 sp=0xc000112fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000113750 sp=0xc000113730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001137e0 sp=0xc000113750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001137e8 sp=0xc0001137e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bef50 sp=0xc0000bef30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000befe0 sp=0xc0000bef50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000befe8 sp=0xc0000befe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000113f50 sp=0xc000113f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000113fe0 sp=0xc000113f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x4ebbcc3d1acf?, 0x3?, 0xa6?, 0x2c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bf750 sp=0xc0000bf730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bf7e0 sp=0xc0000bf750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bf7e8 sp=0xc0000bf7e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x7c8fa00?, 0x1?, 0x5b?, 0xa6?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000bff50 sp=0xc0000bff30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000bffe0 sp=0xc0000bff50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000bffe8 sp=0xc0000bffe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x7c8fa00?, 0x1?, 0x5f?, 0x5a?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000f0750 sp=0xc0000f0730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000f07e0 sp=0xc0000f0750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000f07e8 sp=0xc0000f07e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x4ebbcc35ae0c?, 0x1?, 0xa3?, 0x2e?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000116f50 sp=0xc000116f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000116fe0 sp=0xc000116f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 58 [GC worker (idle)]:
runtime.gopark(0x7c8fa00?, 0x1?, 0x75?, 0x1?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000ec750 sp=0xc0000ec730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000ec7e0 sp=0xc0000ec750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000ec7e8 sp=0xc0000ec7e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x4ebbcc35ab0d?, 0x3?, 0x3d?, 0x15?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000117750 sp=0xc000117730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001177e0 sp=0xc000117750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001177e8 sp=0xc0001177e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x4ebbcc3d36a3?, 0x1?, 0xfd?, 0xcc?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000117f50 sp=0xc000117f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000117fe0 sp=0xc000117f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 42 [GC worker (idle)]:
runtime.gopark(0x7c8fa00?, 0x1?, 0x45?, 0x7c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000128750 sp=0xc000128730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001287e0 sp=0xc000128750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001287e8 sp=0xc0001287e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 43 [GC worker (idle)]:
runtime.gopark(0x4ebbcc35ad91?, 0x3?, 0x78?, 0x1a?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000128f50 sp=0xc000128f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000128fe0 sp=0xc000128f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000128fe8 sp=0xc000128fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 44 [GC worker (idle)]:
runtime.gopark(0x4ebbcc3d22c8?, 0x1?, 0xea?, 0x98?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000129750 sp=0xc000129730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0001297e0 sp=0xc000129750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0001297e8 sp=0xc0001297e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 45 [GC worker (idle)]:
runtime.gopark(0x4ebbcc3d1fd9?, 0x1?, 0x3d?, 0x1c?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 46 [GC worker (idle)]:
runtime.gopark(0x7c8fa00?, 0x1?, 0xb9?, 0x41?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00012a750 sp=0xc00012a730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00012a7e0 sp=0xc00012a750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00012a7e8 sp=0xc00012a7e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 47 [GC worker (idle)]:
runtime.gopark(0x4ebbcc35a71c?, 0x1?, 0x30?, 0x57?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00012af50 sp=0xc00012af30 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00012afe0 sp=0xc00012af50 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00012afe8 sp=0xc00012afe0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x4ebbcc35aae1?, 0x1?, 0x42?, 0x6?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000b9750 sp=0xc0000b9730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000b97e0 sp=0xc0000b9750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000b97e8 sp=0xc0000b97e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 48 [GC worker (idle)]:
runtime.gopark(0x4ebbcc37e6c7?, 0x1?, 0x7f?, 0xb4?, 0x0?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00012b750 sp=0xc00012b730 pc=0x1c314d6
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00012b7e0 sp=0xc00012b750 pc=0x1c10911
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00012b7e8 sp=0xc00012b7e0 pc=0x1c67f01
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 186 [select]:
runtime.gopark(0xc0000cde78?, 0x2?, 0x0?, 0x0?, 0xc0000cde34?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000cdcb8 sp=0xc0000cdc98 pc=0x1c314d6
runtime.selectgo(0xc0000cde78, 0xc0000cde30, 0xc0014c6040?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000cddf8 sp=0xc0000cdcb8 pc=0x1c41ffe
github.com/milvus-io/milvus/pkg/config.(*refresher).refreshPeriodically(0xc001505270, {0x5552cbf, 0xa})
	/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:69 +0x1fd fp=0xc0000cdfb8 sp=0xc0000cddf8 pc=0x2968f3d
github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1.1()
	/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:51 +0x2e fp=0xc0000cdfe0 sp=0xc0000cdfb8 pc=0x2968c2e
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000cdfe8 sp=0xc0000cdfe0 pc=0x1c67f01
created by github.com/milvus-io/milvus/pkg/config.(*refresher).start.func1
	/go/src/github.com/milvus-io/milvus/pkg/config/refresher.go:51 +0xae

goroutine 97 [select]:
runtime.gopark(0xc0000eef80?, 0x2?, 0x28?, 0xd?, 0xc0000eef54?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000eedc0 sp=0xc0000eeda0 pc=0x1c314d6
runtime.selectgo(0xc0000eef80, 0xc0000eef50, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000eef00 sp=0xc0000eedc0 pc=0x1c41ffe
github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc000464a00, {0x5b7fc10, 0xc00002ccd0})
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:83 +0x10a fp=0xc0000eefb8 sp=0xc0000eef00 pc=0x36a202a
github.com/panjf2000/ants/v2.(*Pool).goPurge.func1()
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0x2e fp=0xc0000eefe0 sp=0xc0000eefb8 pc=0x36a262e
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000eefe8 sp=0xc0000eefe0 pc=0x1c67f01
created by github.com/panjf2000/ants/v2.(*Pool).goPurge
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:147 +0xe5

goroutine 162 [select]:
runtime.gopark(0xc0000f0f80?, 0x2?, 0x28?, 0xd?, 0xc0000f0f44?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000f0dc8 sp=0xc0000f0da8 pc=0x1c314d6
runtime.selectgo(0xc0000f0f80, 0xc0000f0f40, 0x0?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000f0f08 sp=0xc0000f0dc8 pc=0x1c41ffe
github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc000464a00, {0x5b7fc10, 0xc00002cd20})
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:125 +0x145 fp=0xc0000f0fb8 sp=0xc0000f0f08 pc=0x36a23c5
github.com/panjf2000/ants/v2.(*Pool).goTicktock.func1()
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x2e fp=0xc0000f0fe0 sp=0xc0000f0fb8 pc=0x36a27ce
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000f0fe8 sp=0xc0000f0fe0 pc=0x1c67f01
created by github.com/panjf2000/ants/v2.(*Pool).goTicktock
	/go/pkg/mod/github.com/panjf2000/ants/[email protected]/pool.go:154 +0x115

goroutine 195 [select, locked to thread]:
runtime.gopark(0xc0000f17a8?, 0x2?, 0x52?, 0x18?, 0xc0000f17a4?)
	/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000f1618 sp=0xc0000f15f8 pc=0x1c314d6
runtime.selectgo(0xc0000f17a8, 0xc0000f17a0, 0x0?, 0x0, 0xc0000f1701?, 0x1)
	/usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000f1758 sp=0xc0000f1618 pc=0x1c41ffe
runtime.ensureSigM.func1()
	/usr/local/go/src/runtime/signal_unix.go:1004 +0x1b0 fp=0xc0000f17e0 sp=0xc0000f1758 pc=0x1c5d230
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000f17e8 sp=0xc0000f17e0 pc=0x1c67f01
created by runtime.ensureSigM
	/usr/local/go/src/runtime/signal_unix.go:987 +0xbd

goroutine 167 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
	/usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0000f1fa0 sp=0xc0000f1f68 pc=0x1bff6b4
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0000f1fc0 sp=0xc0000f1fa0 pc=0x1c63caf
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0000f1fe0 sp=0xc0000f1fc0 pc=0x1d340b9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000f1fe8 sp=0xc0000f1fe0 pc=0x1c67f01
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x2a

I did it with sudo. Same error. I do have right access it's in my home directory
@xiaofan-luan

@xiaofan-luan
Copy link
Collaborator

are u using Natsmq? ["init baseTable with file failed"] [configFile="[milvus.yaml,default.yaml,user.yaml]"] seems that the config file does not exist
we no longer support natsmq

@pooya-mohammadi
Copy link

@xiaofan-luan I don't know what is Natsmq!
Can you tell how can I disable it or check whether I'm using it or not?

@xiaofan-luan
Copy link
Collaborator

xiaofan-luan commented Oct 10, 2024

@xiaofan-luan I don't know what is Natsmq! Can you tell how can I disable it or check whether I'm using it or not?

can you explain how did you bring up your cluster?

did you follow this https://milvus.io/docs/install_standalone-docker.md?

could you check you have write access on /var/lib/milvus/etcd ?

@xiaofan-luan
Copy link
Collaborator

you probably want to make sure you have write access on your local $(pwd)/volumes directory.
becasue all etcd, milvus data will be written to this directory and it is mount to /var/lib/milvus/etcd

@pooya-mohammadi
Copy link

sudo mkdir -p /var/lib/milvus/etcd
$ ls /var/lib/milvus/etcd/

This is empty

@pooya-mohammadi
Copy link

@xiaofan-luan I've tried different ways:

$ curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh

# Start the Docker container
$ bash standalone_embed.sh start

The output is like the following:

Wait for Milvus Starting...

@xiaofan-luan
Copy link
Collaborator

this inside docker /var/lib/milvus/etcd/

on your host you should search for volume directory

@xiaofan-luan
Copy link
Collaborator

/var/lib/milvus/etcd/ is mount to $(pwd)/volumes

@pooya-mohammadi
Copy link

ls $(pwd)/volumes
milvus

I have access to it as well

@xiaofan-luan
Copy link
Collaborator

@zwd1208 @LoveEachDay any other comment?

@pingliu
Copy link
Contributor

pingliu commented Oct 11, 2024

Because you have already started the v2.4.0-rc.1-dev version of the container before, directly executing bash standalone_embed.sh start will start the existing version of the container, and v2.4.0-rc.1-dev has a known startup error issue.

You can use the following command to clear the information of the old version, and then start it again.

bash standalone_embed.sh stop
bash standalone_embed.sh delete
bash standalone_embed.sh start

@xiaofan-luan
Copy link
Collaborator

bash standalone_embed.sh delete

we can't do bash standalone_embed.sh delete right? this will delete your data.

@pingliu
Copy link
Contributor

pingliu commented Oct 11, 2024

bash standalone_embed.sh delete

we can't do bash standalone_embed.sh delete right? this will delete your data.

yes, it will delete data.

@pingliu
Copy link
Contributor

pingliu commented Oct 11, 2024

if you cannot delete data, please switch to the directory where the volume data is located and use the following command

sudo docker stop milvus-standalone
sudo docker rm milvus-standalone
bash standalone_embed.sh start

@pingliu
Copy link
Contributor

pingliu commented Oct 11, 2024

sudo docker run \
        --name milvus-standalone \
        --security-opt seccomp:unconfined \
        -e ETCD_USE_EMBED=true \
        -e ETCD_DATA_DIR=/var/lib/milvus/etcd \
        -e ETCD_CONFIG_PATH=/milvus/configs/embedEtcd.yaml \
        -e COMMON_STORAGETYPE=local \
        -p 19530:19530 \
        -p 9091:9091 \
        -p 2379:2379 \
        -v $(pwd)/volumes/milvus:/var/lib/milvus \
        -v $(pwd)/embedEtcd.yaml:/milvus/configs/embedEtcd.yaml \
        -v $(pwd)/user.yaml:/milvus/configs/user.yaml \
        --health-cmd="curl -f http://localhost:9091/healthz" \
        --health-interval=30s \
        --health-start-period=90s \
        --health-timeout=20s \
        --health-retries=3 \
        --rm \
        milvusdb/milvus:v2.4.5 \
        milvus run standalone
__  _________ _   ____  ______    

/ |/ / / /| | / / / / / __/
/ /|
/ // // /| |/ / // /\ \
// /////_/__/

Welcome to use Milvus!
Version: v2.4.5

[2024/10/09 14:38:52.432 +00:00] [INFO] [config/manager.go:301] ["Get configuration by items failed"] [error="Read config failed: /milvus/configs/user.yaml: read /milvus/configs/user.yaml: is a directory"]
[2024/10/09 14:38:52.432 +00:00] [WARN] [paramtable/base_table.go:175] ["init baseTable with file failed"] [configFile="[milvus.yaml,default.yaml,user.yaml]"] [error="failed to load FileSource cause: 5265616420636f6e666967206661696c65643a202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2072656164202f6d696c7675732f636f6e666967732f757365722e79616d6c3a2069732061206469726563746f7279"]
[2024/10/09 14:38:52.434 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/10/09 14:38:52.434 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/10/09 14:38:52.434 +00:00] [DEBUG] [nmq/nmq_server.go:106] ["Closing Natsmq!"]
[2024/10/09 14:38:52.435 +00:00] [INFO] [roles/roles.go:283] ["All cleanup done, handleSignals goroutine quit"]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x28dd5c1]

This error is because the embedEtcd.yaml and user.yaml files cannot be found.

@pooya-mohammadi
Copy link

pooya-mohammadi commented Oct 11, 2024

@pingliu Thank you, Removing the container and starting with the bash file worked.

sudo docker stop milvus-standalone
sudo docker rm milvus-standalone
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
bash standalone_embed.sh start

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Development

No branches or pull requests

5 participants