Add remove ids for indexrefine #1650

Job	Run time
Format	33s
Linux arm64 packages	0s
Linux x86_64 GPU packages (CUDA 11.4.4)	0s
Linux x86_64 (cmake)	15m 36s
Linux x86_64 GPU packages (CUDA 12.1.1)	0s
Linux x86_64 GPU w/ RAFT packages (CUDA 11.8.0)	0s
Linux x86_64 GPU w/ RAFT packages (CUDA 12.1.1)	0s
Linux x86_64 packages	0s
OSX arm64 packages	0s
Windows x86_64 packages	0s
Linux arm64 SVE (cmake)	14m 28s
Linux x86_64 AVX2 (cmake)	15m 16s
Linux arm64 (conda)	11m 34s
Linux x86_64 (conda)	16m 24s
Linux x86_64 GPU (cmake)	36m 24s
Linux x86_64 GPU w/ RAFT (cmake)	1h 33m 40s
Linux x86_64 AVX512 (cmake)	21m 34s
Linux x86_64 GPU w/ ROCm (cmake)	30m 23s
Windows x86_64 (conda)	35m 6s
	4h 50m 58s

Provide feedback