One node doesnt get balanced properly

[salarali]

Bug Report

I have this issue sometimes that 1 node fails to be balanced properly. It happens randomly and I see no error in logs.

The connection count remains very low for a long time.
How should I go about debugging this? I checked the server and server is up and running.

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your version? (Required)

1.2.0

[salarali]

It could be related that after some time, all session migrations stop all of a sudden.

[djshow832]

1. Add from={the backend instance} in the PromQL in the Grafana panel to filter the migrations from this backend. E.g. sum(increase(tiproxy_balance_migrate_total{k8s_cluster="$k8s_cluster", tidb_cluster="$tidb_cluster", instance=~"$instance", from="127.0.0.1:3080"}[1m])) by (reason)
2. Check the CPU, memory, 4000 and 10080 ports, and network of this backend. I guess the reason comes from memory. Check if the memory usage is too high or if it increases too sharply.
3. Search the log in TiProxy for update backend to see the status change of backends. E.g. [update backend] [namespace=default] [backend_addr=[10.90.2.153:4000](http://10.90.2.153:4000/)] [prev=healthy] [cur="down, err: http status 500: connect status port failed"] indicates that the backend 10.90.2.153:4000 is currently down because the 10080 status cannot be connected.

I hope that will help.

[salarali]

So I see that the connections got moved away from it because of memory factor, but after that, no logs related to that backend at all. No update backend. The node is healthy as well. Memory use is also very low, no reason why it should not get more connections moved over to it.

[djshow832]

Can you query the metrics that TiProxy has read?

1. Find the TiProxy owner by Grafana panel Server -> Owner, get the instance with label metric_reader.
2. Query the metrics from the owner by HTTP API: curl 127.0.0.1:3080/api/backend/metrics. The result contains the current metrics of all backends. It may be hard to read. You can paste it here.

[salarali]

Let me check those out.

I also added logs here:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L311

	if balanceCount > 0 {
		router.logger.Info("backends to balance", zap.String("from", busiestBackend.Addr()), zap.String("to", idlestBackend.Addr()), zap.String("reason", reason))
	}

And I am getting these logs there:

Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.159 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.169 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.179 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.189 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.199 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.209 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.219 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.229 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]
Aug 28 03:27:56 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:27:56.239 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.89.188:5678] [reason=status]

172.30.72.136 is dead and gone now. I guess that is the reason balancing just stops after a while.

[djshow832]

Please try to connect to the 4000 and 10080 ports. curl 127.0.0.1:10080/status.
BTW, you don't have to add logs, you can just turn the log level to debug.

[salarali]

Curl commands from the tiproxy node:

curl 172.30.72.136:1008/status
curl: (7) Failed to connect to 172.30.72.136 port 1008 after 3131 ms: Couldn't connect to server

This is the dead node that is keeps trying to move connections away from. I use port 1008 instead of 10080.

Metrics from tiproxy:

{ "cpu": { "172.30.65.197:1008": { "Step1History": [ [ 1724815804.77, "3163.686875" ], [ 1724815809.775, "3167.0746875" ], [ 1724815814.799, "3170.431875" ], [ 1724815819.795, "3173.8028125" ], [ 1724815824.769, "3177.0978125" ], [ 1724815829.769, "3179.7228125" ], [ 1724815834.77, "3182.265" ], [ 1724815839.803, "3185.47" ], [ 1724815844.77, "3188.686875" ], [ 1724815849.77, "3191.1971875" ], [ 1724815854.773, "3192.9121875" ], [ 1724815859.768, "3193.3196875" ] ], "Step2History": [ [ 1724815804.77, "0.6347676070631327" ], [ 1724815809.775, "0.6768856143856362" ], [ 1724815814.799, "0.6682299960191553" ], [ 1724815819.795, "0.6747272818254312" ], [ 1724815824.769, "0.6624447125050408" ], [ 1724815829.769, "0.525" ], [ 1724815834.77, "0.5083358328333787" ], [ 1724815839.803, "0.6367971388833552" ], [ 1724815844.77, "0.6476494866116514" ], [ 1724815849.77, "0.502062500000011" ], [ 1724815854.773, "0.3427943234059855" ], [ 1724815859.768, "0.08158158158154152" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815804.77, "3697.093125" ], [ 1724815809.77, "3697.4859375" ], [ 1724815814.771, "3697.8125" ], [ 1724815819.78, "3697.905" ], [ 1724815824.775, "3698.13875" ], [ 1724815829.77, "3699.020625" ], [ 1724815834.772, "3700.964375" ], [ 1724815839.772, "3703.625" ], [ 1724815844.77, "3706.604375" ], [ 1724815849.793, "3709.37625" ], [ 1724815854.787, "3713.2875" ], [ 1724815859.789, "3717.2003125" ] ], "Step2History": [ [ 1724815804.77, "0.1392499999999927" ], [ 1724815809.77, "0.07856250000004365" ], [ 1724815814.771, "0.06529944011195941" ], [ 1724815819.78, "0.018466759832341802" ], [ 1724815824.775, "0.0467967967967713" ], [ 1724815829.77, "0.17655155155155883" ], [ 1724815834.772, "0.3885945621751118" ], [ 1724815839.772, "0.5321249999999964" ], [ 1724815844.77, "0.5961134453781294" ], [ 1724815849.793, "0.5518365518614193" ], [ 1724815854.787, "0.7831898277933739" ], [ 1724815859.789, "0.782249600159976" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815804.765, "872.634375" ], [ 1724815809.765, "872.6390625" ], [ 1724815814.766, "872.644375" ], [ 1724815819.765, "872.649375" ], [ 1724815824.765, "872.6546875" ], [ 1724815829.765, "872.6596875" ], [ 1724815834.766, "872.665" ], [ 1724815839.765, "872.67" ], [ 1724815844.765, "872.675625" ], [ 1724815849.765, "872.6803125" ], [ 1724815854.765, "872.685625" ], [ 1724815859.765, "872.6909375" ] ], "Step2History": [ [ 1724815804.765, "0.0011874999999918145" ], [ 1724815809.765, "0.0009375000000090949" ], [ 1724815814.766, "0.0010622875424805898" ], [ 1724815819.765, "0.001000200040007092" ], [ 1724815824.765, "0.0010625000000118235" ], [ 1724815829.765, "0.0009999999999990906" ], [ 1724815834.766, "0.0010622875424805898" ], [ 1724815839.765, "0.001000200040007092" ], [ 1724815844.765, "0.001125000000001819" ], [ 1724815849.765, "0.0009375000000090949" ], [ 1724815854.765, "0.001062499999989086" ], [ 1724815859.765, "0.0010625000000118235" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815804.783, "2135.5996875" ], [ 1724815809.768, "2137.6321875" ], [ 1724815814.77, "2139.039375" ], [ 1724815819.768, "2140.269375" ], [ 1724815824.767, "2141.935625" ], [ 1724815829.767, "2143.73125" ], [ 1724815834.768, "2146.1565625" ], [ 1724815839.769, "2149.355625" ], [ 1724815844.767, "2152.294375" ], [ 1724815849.772, "2155.663125" ], [ 1724815854.782, "2159.8134375" ], [ 1724815859.781, "2164.080625" ] ], "Step2History": [ [ 1724815804.783, "0.7957718388512637" ], [ 1724815809.768, "0.4077231695084854" ], [ 1724815814.77, "0.2813249700119879" ], [ 1724815819.768, "0.24609843937575393" ], [ 1724815824.767, "0.3333166633327102" ], [ 1724815829.767, "0.35912499999994907" ], [ 1724815834.768, "0.4849655068986239" ], [ 1724815839.769, "0.6396845630874444" ], [ 1724815844.767, "0.587985194077591" ], [ 1724815849.772, "0.6730769230769412" ], [ 1724815854.782, "0.82840568862274" ], [ 1724815859.781, "0.8536082216443471" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815804.783, "5181.1353125" ], [ 1724815809.787, "5185.5109375" ], [ 1724815814.786, "5189.7796875" ], [ 1724815819.773, "5193.738125" ], [ 1724815824.773, "5196.954375" ], [ 1724815829.779, "5198.663125" ], [ 1724815834.77, "5198.7990625" ], [ 1724815839.77, "5198.9175" ], [ 1724815844.77, "5199.1275" ], [ 1724815849.769, "5199.29375" ], [ 1724815854.774, "5199.314375" ], [ 1724815859.77, "5199.3678125" ] ], "Step2History": [ [ 1724815804.783, "0.8873850229952772" ], [ 1724815809.787, "0.8744254596324106" ], [ 1724815814.786, "0.8539207841568678" ], [ 1724815819.773, "0.7937512532583699" ], [ 1724815824.773, "0.64325000000008" ], [ 1724815829.779, "0.3413403915301202" ], [ 1724815834.77, "0.027236525746379865" ], [ 1724815839.77, "0.023687499999869032" ], [ 1724815844.77, "0.042000000000007275" ], [ 1724815849.769, "0.03325665133030972" ], [ 1724815854.774, "0.004120879120900927" ], [ 1724815859.77, "0.010696056845425407" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815804.79, "4308.91625" ], [ 1724815809.773, "4313.6228125" ], [ 1724815814.772, "4318.255" ], [ 1724815819.772, "4322.8815625" ], [ 1724815824.772, "4327.45125" ], [ 1724815829.775, "4332.0190625" ], [ 1724815834.78, "4336.53625" ], [ 1724815839.776, "4341.2296875" ], [ 1724815844.771, "4345.9265625" ], [ 1724815849.773, "4350.518125" ], [ 1724815854.771, "4354.0234375" ], [ 1724815859.782, "4358.444375" ] ], "Step2History": [ [ 1724815804.79, "0.9370639824596517" ], [ 1724815809.773, "0.9445238811959791" ], [ 1724815814.772, "0.9266228245649786" ], [ 1724815819.772, "0.9253124999999273" ], [ 1724815824.772, "0.9139375000000655" ], [ 1724815829.775, "0.913014691185187" ], [ 1724815834.78, "0.9025349650350741" ], [ 1724815839.776, "0.9394390512410072" ], [ 1724815844.771, "0.9403153153152425" ], [ 1724815849.773, "0.9179453218712079" ], [ 1724815854.771, "0.7013430372149659" ], [ 1724815859.782, "0.8822465575733459" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815799.933, "3745.8446875" ], [ 1724815804.806, "3750.320625" ], [ 1724815809.779, "3754.791875" ], [ 1724815814.813, "3759.3171875" ], [ 1724815819.783, "3763.7778125" ], [ 1724815824.812, "3768.3053125" ], [ 1724815829.78, "3772.7946875" ], [ 1724815834.795, "3777.3184375" ], [ 1724815839.889, "3781.8484375" ], [ 1724815844.778, "3786.3090625" ], [ 1724815850.034, "3790.824375" ], [ 1724815854.893, "3795.2865625" ], [ 1724815859.853, "3799.848125" ] ], "Step2History": [ [ 1724815799.933, "0.9056960764788144" ], [ 1724815804.806, "0.9185178534783239" ], [ 1724815809.779, "0.8991051679067071" ], [ 1724815814.813, "0.8989496424314516" ], [ 1724815819.783, "0.897510060362206" ], [ 1724815824.812, "0.9002783853649127" ], [ 1724815829.78, "0.9036584138485617" ], [ 1724815834.795, "0.9020438683948736" ], [ 1724815839.889, "0.8892815076560159" ], [ 1724815844.778, "0.9123798322765726" ], [ 1724815850.034, "0.8590777207001833" ], [ 1724815854.893, "0.9183345338546615" ], [ 1724815859.853, "0.9196698588709751" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815804.765, "657.5271875" ], [ 1724815809.769, "658.0003125" ], [ 1724815814.765, "658.3203125" ], [ 1724815819.765, "658.87625" ], [ 1724815824.766, "659.3309375" ], [ 1724815829.766, "659.918125" ], [ 1724815834.765, "661.245625" ], [ 1724815839.765, "662.998125" ], [ 1724815844.77, "665.089375" ], [ 1724815849.877, "668.28125" ], [ 1724815854.766, "671.7965625" ], [ 1724815859.766, "673.4021875" ] ], "Step2History": [ [ 1724815804.765, "0.18472444488897327" ], [ 1724815809.769, "0.0945493605115871" ], [ 1724815814.765, "0.06405124099280424" ], [ 1724815819.765, "0.11118750000000546" ], [ 1724815824.766, "0.0909193161367681" ], [ 1724815829.766, "0.11743750000000545" ], [ 1724815834.765, "0.2655531106221217" ], [ 1724815839.765, "0.35049999999998815" ], [ 1724815844.77, "0.41783216783217964" ], [ 1724815849.877, "0.6249999999999964" ], [ 1724815854.766, "0.719024851707926" ], [ 1724815859.766, "0.3211249999999836" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815804.771, "3422.655625" ], [ 1724815809.771, "3425.7834375" ], [ 1724815814.806, "3428.78375" ], [ 1724815819.793, "3430.7909375" ], [ 1724815824.77, "3432.81125" ], [ 1724815829.77, "3435.3896875" ], [ 1724815834.77, "3437.940625" ], [ 1724815839.807, "3441.090625" ], [ 1724815844.771, "3444.4559375" ], [ 1724815849.777, "3447.7721875" ], [ 1724815854.77, "3450.79" ], [ 1724815859.771, "3453.3978125" ] ], "Step2History": [ [ 1724815804.771, "0.46243749999994177" ], [ 1724815809.771, "0.6255625000000691" ], [ 1724815814.806, "0.5958912611717649" ], [ 1724815819.793, "0.4024839582915325" ], [ 1724815824.77, "0.40592977697413557" ], [ 1724815829.77, "0.5156874999999672" ], [ 1724815834.77, "0.5101875000000291" ], [ 1724815839.807, "0.625372245384085" ], [ 1724815844.771, "0.6779436946011427" ], [ 1724815849.777, "0.6624550539353394" ], [ 1724815854.77, "0.60440867214095" ], [ 1724815859.771, "0.5214582083583101" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815804.776, "4343.0834375" ], [ 1724815809.774, "4345.7246875" ], [ 1724815814.789, "4347.8746875" ], [ 1724815819.776, "4350.93625" ], [ 1724815824.776, "4354.4703125" ], [ 1724815829.775, "4357.1771875" ], [ 1724815834.776, "4359.8353125" ], [ 1724815839.841, "4363.000625" ], [ 1724815844.775, "4366.26" ], [ 1724815849.782, "4368.994375" ], [ 1724815854.789, "4371.2934375" ], [ 1724815859.863, "4374.413125" ] ], "Step2History": [ [ 1724815804.776, "0.8648648648649887" ], [ 1724815809.774, "0.528461384553756" ], [ 1724815814.789, "0.4287138584246533" ], [ 1724815819.776, "0.6139086625225659" ], [ 1724815824.776, "0.7068124999999782" ], [ 1724815829.775, "0.5414832966594847" ], [ 1724815834.776, "0.5315186962607332" ], [ 1724815839.841, "0.624938302072921" ], [ 1724815844.775, "0.6605948520471312" ], [ 1724815849.782, "0.546110445376473" ], [ 1724815854.789, "0.45916966247258206" ], [ 1724815859.863, "0.6148378990933385" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815804.809, "122952.1578125" ], [ 1724815809.808, "122952.865" ], [ 1724815814.797, "122953.4575" ], [ 1724815819.796, "122954.059375" ], [ 1724815824.808, "122954.6484375" ], [ 1724815829.792, "122955.26875" ], [ 1724815834.793, "122955.8365625" ], [ 1724815839.792, "122956.3340625" ], [ 1724815844.791, "122956.715625" ], [ 1724815849.792, "122957.10375" ], [ 1724815854.804, "122957.496875" ], [ 1724815859.79, "122957.891875" ] ], "Step2History": [ [ 1724815804.809, "0.2142197484001151" ], [ 1724815809.808, "0.14146579316084407" ], [ 1724815814.797, "0.11876127480433671" ], [ 1724815819.796, "0.12039907981456593" ], [ 1724815824.808, "0.11753042697584007" ], [ 1724815829.792, "0.1244607744789146" ], [ 1724815834.793, "0.1135397920411261" ], [ 1724815839.792, "0.09951990398033041" ], [ 1724815844.791, "0.07632776555287775" ], [ 1724815849.792, "0.07760947810391355" ], [ 1724815854.804, "0.07843675179615489" ], [ 1724815859.79, "0.07922182109989462" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815804.811, "121455.769375" ], [ 1724815809.793, "121457.0215625" ], [ 1724815814.798, "121457.695625" ], [ 1724815819.791, "121458.3453125" ], [ 1724815824.789, "121458.909375" ], [ 1724815829.797, "121459.6084375" ], [ 1724815834.813, "121460.4428125" ], [ 1724815839.798, "121461.2521875" ], [ 1724815844.792, "121461.8559375" ], [ 1724815849.795, "121462.73125" ], [ 1724815854.799, "121463.2984375" ], [ 1724815859.789, "121463.693125" ] ], "Step2History": [ [ 1724815804.811, "0.15282755874188494" ], [ 1724815809.793, "0.25134233239557635" ], [ 1724815814.798, "0.13467782217677549" ], [ 1724815819.791, "0.13011966753478152" ], [ 1724815824.789, "0.11285764305896981" ], [ 1724815829.797, "0.13958915734894017" ], [ 1724815834.813, "0.16634270334754164" ], [ 1724815839.798, "0.16236208625819248" ], [ 1724815844.792, "0.12089507409077158" ], [ 1724815849.795, "0.17495752548331303" ], [ 1724815854.799, "0.11334682254138483" ], [ 1724815859.789, "0.07909569138498185" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815804.766, "693.51625" ], [ 1724815809.766, "693.55" ], [ 1724815814.766, "693.561875" ], [ 1724815819.765, "693.57375" ], [ 1724815824.765, "693.636875" ], [ 1724815829.775, "694.445625" ], [ 1724815834.767, "695.4015625" ], [ 1724815839.769, "696.7715625" ], [ 1724815844.773, "698.7621875" ], [ 1724815849.767, "702.030625" ], [ 1724815854.769, "704.406875" ], [ 1724815859.803, "708.0746875" ] ], "Step2History": [ [ 1724815804.766, "0.04006249999999909" ], [ 1724815809.766, "0.006749999999988177" ], [ 1724815814.766, "0.0023750000000063664" ], [ 1724815819.765, "0.002375475095025372" ], [ 1724815824.765, "0.012625000000002728" ], [ 1724815829.775, "0.1614271457085665" ], [ 1724815834.767, "0.1914938902243599" ], [ 1724815839.769, "0.2738904438224719" ], [ 1724815844.773, "0.3978067545963275" ], [ 1724815849.767, "0.65447286744093" ], [ 1724815854.769, "0.47505997600960165" ], [ 1724815859.803, "0.7286079658323338" ] ] } }, "health_pd": { "172.30.65.197:1008": { "Step1History": [ [ 1724815744.785, "2" ], [ 1724815749.776, "2" ], [ 1724815754.768, "2" ], [ 1724815759.771, "2" ], [ 1724815764.769, "2" ], [ 1724815769.801, "2" ], [ 1724815774.809, "2" ], [ 1724815779.775, "2" ], [ 1724815784.775, "2" ], [ 1724815789.821, "2" ], [ 1724815794.82, "2" ], [ 1724815799.843, "2" ], [ 1724815804.77, "2" ], [ 1724815809.775, "2" ], [ 1724815814.799, "2" ], [ 1724815819.795, "2" ], [ 1724815824.769, "2" ], [ 1724815829.769, "2" ], [ 1724815834.77, "2" ], [ 1724815839.803, "2" ], [ 1724815844.77, "2" ], [ 1724815849.77, "2" ], [ 1724815854.773, "2" ], [ 1724815859.768, "2" ] ], "Step2History": [ [ 1724815744.785, "1" ], [ 1724815749.776, "1" ], [ 1724815754.768, "1" ], [ 1724815759.771, "1" ], [ 1724815764.769, "1" ], [ 1724815769.801, "1" ], [ 1724815774.809, "1" ], [ 1724815779.775, "1" ], [ 1724815784.775, "1" ], [ 1724815789.821, "1" ], [ 1724815794.82, "1" ], [ 1724815799.843, "0" ], [ 1724815804.77, "0" ], [ 1724815809.775, "0" ], [ 1724815814.799, "0" ], [ 1724815819.795, "0" ], [ 1724815824.769, "0" ], [ 1724815829.769, "0" ], [ 1724815834.77, "0" ], [ 1724815839.803, "0" ], [ 1724815844.77, "0" ], [ 1724815849.77, "0" ], [ 1724815854.773, "0" ], [ 1724815859.768, "0" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815744.771, "1" ], [ 1724815749.771, "1" ], [ 1724815754.769, "1" ], [ 1724815759.78, "1" ], [ 1724815764.77, "1" ], [ 1724815769.77, "1" ], [ 1724815774.771, "1" ], [ 1724815779.771, "1" ], [ 1724815784.77, "1" ], [ 1724815789.77, "1" ], [ 1724815794.77, "1" ], [ 1724815799.77, "1" ], [ 1724815804.77, "1" ], [ 1724815809.77, "1" ], [ 1724815814.771, "1" ], [ 1724815819.78, "1" ], [ 1724815824.775, "1" ], [ 1724815829.77, "1" ], [ 1724815834.772, "1" ], [ 1724815839.772, "1" ], [ 1724815844.77, "1" ], [ 1724815849.793, "1" ], [ 1724815854.787, "1" ], [ 1724815859.789, "1" ] ], "Step2History": [ [ 1724815744.771, "0" ], [ 1724815749.771, "0" ], [ 1724815754.769, "0" ], [ 1724815759.78, "0" ], [ 1724815764.77, "0" ], [ 1724815769.77, "0" ], [ 1724815774.771, "0" ], [ 1724815779.771, "0" ], [ 1724815784.77, "0" ], [ 1724815789.77, "0" ], [ 1724815794.77, "0" ], [ 1724815799.77, "0" ], [ 1724815804.77, "0" ], [ 1724815809.77, "0" ], [ 1724815814.771, "0" ], [ 1724815819.78, "0" ], [ 1724815824.775, "0" ], [ 1724815829.77, "0" ], [ 1724815834.772, "0" ], [ 1724815839.772, "0" ], [ 1724815844.77, "0" ], [ 1724815849.793, "0" ], [ 1724815854.787, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ], "Step2History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815744.908, "0" ], [ 1724815749.834, "0" ], [ 1724815754.769, "0" ], [ 1724815759.77, "0" ], [ 1724815764.769, "0" ], [ 1724815769.795, "0" ], [ 1724815774.769, "0" ], [ 1724815779.779, "0" ], [ 1724815784.769, "0" ], [ 1724815789.778, "0" ], [ 1724815794.782, "0" ], [ 1724815799.769, "0" ], [ 1724815804.783, "0" ], [ 1724815809.768, "0" ], [ 1724815814.77, "0" ], [ 1724815819.768, "0" ], [ 1724815824.767, "0" ], [ 1724815829.767, "0" ], [ 1724815834.768, "0" ], [ 1724815839.769, "0" ], [ 1724815844.767, "0" ], [ 1724815849.772, "0" ], [ 1724815854.782, "0" ], [ 1724815859.781, "0" ] ], "Step2History": [ [ 1724815744.908, "0" ], [ 1724815749.834, "0" ], [ 1724815754.769, "0" ], [ 1724815759.77, "0" ], [ 1724815764.769, "0" ], [ 1724815769.795, "0" ], [ 1724815774.769, "0" ], [ 1724815779.779, "0" ], [ 1724815784.769, "0" ], [ 1724815789.778, "0" ], [ 1724815794.782, "0" ], [ 1724815799.769, "0" ], [ 1724815804.783, "0" ], [ 1724815809.768, "0" ], [ 1724815814.77, "0" ], [ 1724815819.768, "0" ], [ 1724815824.767, "0" ], [ 1724815829.767, "0" ], [ 1724815834.768, "0" ], [ 1724815839.769, "0" ], [ 1724815844.767, "0" ], [ 1724815849.772, "0" ], [ 1724815854.782, "0" ], [ 1724815859.781, "0" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815744.795, "8" ], [ 1724815749.78, "8" ], [ 1724815754.772, "8" ], [ 1724815759.791, "8" ], [ 1724815764.788, "8" ], [ 1724815769.796, "8" ], [ 1724815774.779, "8" ], [ 1724815779.798, "8" ], [ 1724815784.776, "8" ], [ 1724815789.775, "8" ], [ 1724815794.779, "8" ], [ 1724815799.782, "8" ], [ 1724815804.783, "8" ], [ 1724815809.787, "8" ], [ 1724815814.786, "8" ], [ 1724815819.773, "8" ], [ 1724815824.773, "8" ], [ 1724815829.779, "8" ], [ 1724815834.77, "8" ], [ 1724815839.77, "8" ], [ 1724815844.77, "8" ], [ 1724815849.769, "8" ], [ 1724815854.774, "8" ], [ 1724815859.77, "8" ] ], "Step2History": [ [ 1724815744.795, "0" ], [ 1724815749.78, "0" ], [ 1724815754.772, "0" ], [ 1724815759.791, "0" ], [ 1724815764.788, "0" ], [ 1724815769.796, "0" ], [ 1724815774.779, "0" ], [ 1724815779.798, "0" ], [ 1724815784.776, "0" ], [ 1724815789.775, "0" ], [ 1724815794.779, "0" ], [ 1724815799.782, "0" ], [ 1724815804.783, "0" ], [ 1724815809.787, "0" ], [ 1724815814.786, "0" ], [ 1724815819.773, "0" ], [ 1724815824.773, "0" ], [ 1724815829.779, "0" ], [ 1724815834.77, "0" ], [ 1724815839.77, "0" ], [ 1724815844.77, "0" ], [ 1724815849.769, "0" ], [ 1724815854.774, "0" ], [ 1724815859.77, "0" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815744.777, "3" ], [ 1724815749.772, "3" ], [ 1724815754.771, "3" ], [ 1724815759.78, "3" ], [ 1724815764.778, "3" ], [ 1724815769.774, "3" ], [ 1724815774.775, "3" ], [ 1724815779.783, "3" ], [ 1724815784.79, "3" ], [ 1724815789.777, "3" ], [ 1724815794.771, "3" ], [ 1724815799.773, "3" ], [ 1724815804.79, "3" ], [ 1724815809.773, "3" ], [ 1724815814.772, "3" ], [ 1724815819.772, "3" ], [ 1724815824.772, "3" ], [ 1724815829.775, "3" ], [ 1724815834.78, "3" ], [ 1724815839.776, "3" ], [ 1724815844.771, "3" ], [ 1724815849.773, "3" ], [ 1724815854.771, "3" ], [ 1724815859.782, "3" ] ], "Step2History": [ [ 1724815744.777, "0" ], [ 1724815749.772, "0" ], [ 1724815754.771, "0" ], [ 1724815759.78, "0" ], [ 1724815764.778, "0" ], [ 1724815769.774, "0" ], [ 1724815774.775, "0" ], [ 1724815779.783, "0" ], [ 1724815784.79, "0" ], [ 1724815789.777, "0" ], [ 1724815794.771, "0" ], [ 1724815799.773, "0" ], [ 1724815804.79, "0" ], [ 1724815809.773, "0" ], [ 1724815814.772, "0" ], [ 1724815819.772, "0" ], [ 1724815824.772, "0" ], [ 1724815829.775, "0" ], [ 1724815834.78, "0" ], [ 1724815839.776, "0" ], [ 1724815844.771, "0" ], [ 1724815849.773, "0" ], [ 1724815854.771, "0" ], [ 1724815859.782, "0" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815739.92, "8" ], [ 1724815744.841, "8" ], [ 1724815749.819, "8" ], [ 1724815754.851, "8" ], [ 1724815759.849, "8" ], [ 1724815764.858, "8" ], [ 1724815769.814, "8" ], [ 1724815774.814, "8" ], [ 1724815779.781, "8" ], [ 1724815784.794, "8" ], [ 1724815789.789, "8" ], [ 1724815794.912, "8" ], [ 1724815799.933, "8" ], [ 1724815804.806, "8" ], [ 1724815809.779, "8" ], [ 1724815814.813, "8" ], [ 1724815819.783, "8" ], [ 1724815824.812, "8" ], [ 1724815829.78, "8" ], [ 1724815834.795, "8" ], [ 1724815839.889, "8" ], [ 1724815844.778, "8" ], [ 1724815850.034, "8" ], [ 1724815854.893, "8" ], [ 1724815859.853, "8" ] ], "Step2History": [ [ 1724815739.92, "0" ], [ 1724815744.841, "0" ], [ 1724815749.819, "0" ], [ 1724815754.851, "0" ], [ 1724815759.849, "0" ], [ 1724815764.858, "0" ], [ 1724815769.814, "0" ], [ 1724815774.814, "0" ], [ 1724815779.781, "0" ], [ 1724815784.794, "0" ], [ 1724815789.789, "0" ], [ 1724815794.912, "0" ], [ 1724815799.933, "0" ], [ 1724815804.806, "0" ], [ 1724815809.779, "0" ], [ 1724815814.813, "0" ], [ 1724815819.783, "0" ], [ 1724815824.812, "0" ], [ 1724815829.78, "0" ], [ 1724815834.795, "0" ], [ 1724815839.889, "0" ], [ 1724815844.778, "0" ], [ 1724815850.034, "0" ], [ 1724815854.893, "0" ], [ 1724815859.853, "0" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815744.766, "0" ], [ 1724815749.765, "0" ], [ 1724815754.767, "0" ], [ 1724815759.77, "0" ], [ 1724815764.765, "0" ], [ 1724815769.77, "0" ], [ 1724815774.765, "0" ], [ 1724815779.766, "0" ], [ 1724815784.771, "0" ], [ 1724815789.768, "0" ], [ 1724815794.765, "0" ], [ 1724815799.766, "0" ], [ 1724815804.765, "0" ], [ 1724815809.769, "0" ], [ 1724815814.765, "0" ], [ 1724815819.765, "0" ], [ 1724815824.766, "0" ], [ 1724815829.766, "0" ], [ 1724815834.765, "0" ], [ 1724815839.765, "0" ], [ 1724815844.77, "0" ], [ 1724815849.877, "0" ], [ 1724815854.766, "0" ], [ 1724815859.766, "0" ] ], "Step2History": [ [ 1724815744.766, "0" ], [ 1724815749.765, "0" ], [ 1724815754.767, "0" ], [ 1724815759.77, "0" ], [ 1724815764.765, "0" ], [ 1724815769.77, "0" ], [ 1724815774.765, "0" ], [ 1724815779.766, "0" ], [ 1724815784.771, "0" ], [ 1724815789.768, "0" ], [ 1724815794.765, "0" ], [ 1724815799.766, "0" ], [ 1724815804.765, "0" ], [ 1724815809.769, "0" ], [ 1724815814.765, "0" ], [ 1724815819.765, "0" ], [ 1724815824.766, "0" ], [ 1724815829.766, "0" ], [ 1724815834.765, "0" ], [ 1724815839.765, "0" ], [ 1724815844.77, "0" ], [ 1724815849.877, "0" ], [ 1724815854.766, "0" ], [ 1724815859.766, "0" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815744.771, "7" ], [ 1724815749.77, "7" ], [ 1724815754.77, "7" ], [ 1724815759.771, "7" ], [ 1724815764.837, "7" ], [ 1724815769.817, "7" ], [ 1724815774.772, "7" ], [ 1724815779.773, "7" ], [ 1724815784.771, "7" ], [ 1724815789.776, "7" ], [ 1724815794.771, "7" ], [ 1724815799.771, "7" ], [ 1724815804.771, "7" ], [ 1724815809.771, "7" ], [ 1724815814.806, "7" ], [ 1724815819.793, "7" ], [ 1724815824.77, "7" ], [ 1724815829.77, "7" ], [ 1724815834.77, "7" ], [ 1724815839.807, "7" ], [ 1724815844.771, "7" ], [ 1724815849.777, "7" ], [ 1724815854.77, "7" ], [ 1724815859.771, "7" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.77, "1" ], [ 1724815754.77, "1" ], [ 1724815759.771, "1" ], [ 1724815764.837, "1" ], [ 1724815769.817, "1" ], [ 1724815774.772, "1" ], [ 1724815779.773, "1" ], [ 1724815784.771, "1" ], [ 1724815789.776, "1" ], [ 1724815794.771, "1" ], [ 1724815799.771, "0" ], [ 1724815804.771, "0" ], [ 1724815809.771, "0" ], [ 1724815814.806, "0" ], [ 1724815819.793, "0" ], [ 1724815824.77, "0" ], [ 1724815829.77, "0" ], [ 1724815834.77, "0" ], [ 1724815839.807, "0" ], [ 1724815844.771, "0" ], [ 1724815849.777, "0" ], [ 1724815854.77, "0" ], [ 1724815859.771, "0" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815744.811, "2" ], [ 1724815749.776, "2" ], [ 1724815754.775, "2" ], [ 1724815759.783, "2" ], [ 1724815764.797, "2" ], [ 1724815769.804, "2" ], [ 1724815774.782, "2" ], [ 1724815779.777, "2" ], [ 1724815784.782, "2" ], [ 1724815789.778, "2" ], [ 1724815794.785, "2" ], [ 1724815799.781, "2" ], [ 1724815804.776, "2" ], [ 1724815809.774, "2" ], [ 1724815814.789, "2" ], [ 1724815819.776, "2" ], [ 1724815824.776, "2" ], [ 1724815829.775, "2" ], [ 1724815834.776, "2" ], [ 1724815839.841, "2" ], [ 1724815844.775, "2" ], [ 1724815849.782, "2" ], [ 1724815854.789, "2" ], [ 1724815859.863, "2" ] ], "Step2History": [ [ 1724815744.811, "1" ], [ 1724815749.776, "1" ], [ 1724815754.775, "1" ], [ 1724815759.783, "1" ], [ 1724815764.797, "1" ], [ 1724815769.804, "1" ], [ 1724815774.782, "1" ], [ 1724815779.777, "1" ], [ 1724815784.782, "1" ], [ 1724815789.778, "1" ], [ 1724815794.785, "1" ], [ 1724815799.781, "0" ], [ 1724815804.776, "0" ], [ 1724815809.774, "0" ], [ 1724815814.789, "0" ], [ 1724815819.776, "0" ], [ 1724815824.776, "0" ], [ 1724815829.775, "0" ], [ 1724815834.776, "0" ], [ 1724815839.841, "0" ], [ 1724815844.775, "0" ], [ 1724815849.782, "0" ], [ 1724815854.789, "0" ], [ 1724815859.863, "0" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815744.792, "12049" ], [ 1724815749.792, "12049" ], [ 1724815754.816, "12049" ], [ 1724815759.808, "12049" ], [ 1724815764.794, "12049" ], [ 1724815769.806, "12049" ], [ 1724815774.795, "12049" ], [ 1724815779.796, "12049" ], [ 1724815784.791, "12049" ], [ 1724815789.797, "12049" ], [ 1724815794.794, "12049" ], [ 1724815799.801, "12049" ], [ 1724815804.809, "12049" ], [ 1724815809.808, "12049" ], [ 1724815814.797, "12049" ], [ 1724815819.796, "12049" ], [ 1724815824.808, "12049" ], [ 1724815829.792, "12049" ], [ 1724815834.793, "12049" ], [ 1724815839.792, "12049" ], [ 1724815844.791, "12049" ], [ 1724815849.792, "12049" ], [ 1724815854.804, "12049" ], [ 1724815859.79, "12049" ] ], "Step2History": [ [ 1724815744.792, "0" ], [ 1724815749.792, "0" ], [ 1724815754.816, "0" ], [ 1724815759.808, "0" ], [ 1724815764.794, "0" ], [ 1724815769.806, "0" ], [ 1724815774.795, "0" ], [ 1724815779.796, "0" ], [ 1724815784.791, "0" ], [ 1724815789.797, "0" ], [ 1724815794.794, "0" ], [ 1724815799.801, "0" ], [ 1724815804.809, "0" ], [ 1724815809.808, "0" ], [ 1724815814.797, "0" ], [ 1724815819.796, "0" ], [ 1724815824.808, "0" ], [ 1724815829.792, "0" ], [ 1724815834.793, "0" ], [ 1724815839.792, "0" ], [ 1724815844.791, "0" ], [ 1724815849.792, "0" ], [ 1724815854.804, "0" ], [ 1724815859.79, "0" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815744.785, "10813" ], [ 1724815749.785, "10813" ], [ 1724815754.785, "10813" ], [ 1724815759.795, "10813" ], [ 1724815764.791, "10813" ], [ 1724815769.804, "10813" ], [ 1724815774.789, "10813" ], [ 1724815779.793, "10813" ], [ 1724815784.791, "10813" ], [ 1724815789.79, "10813" ], [ 1724815794.789, "10813" ], [ 1724815799.789, "10813" ], [ 1724815804.811, "10813" ], [ 1724815809.793, "10813" ], [ 1724815814.798, "10813" ], [ 1724815819.791, "10813" ], [ 1724815824.789, "10813" ], [ 1724815829.797, "10813" ], [ 1724815834.813, "10813" ], [ 1724815839.798, "10813" ], [ 1724815844.792, "10813" ], [ 1724815849.795, "10813" ], [ 1724815854.799, "10813" ], [ 1724815859.789, "10813" ] ], "Step2History": [ [ 1724815744.785, "0" ], [ 1724815749.785, "0" ], [ 1724815754.785, "0" ], [ 1724815759.795, "0" ], [ 1724815764.791, "0" ], [ 1724815769.804, "0" ], [ 1724815774.789, "0" ], [ 1724815779.793, "0" ], [ 1724815784.791, "0" ], [ 1724815789.79, "0" ], [ 1724815794.789, "0" ], [ 1724815799.789, "0" ], [ 1724815804.811, "0" ], [ 1724815809.793, "0" ], [ 1724815814.798, "0" ], [ 1724815819.791, "0" ], [ 1724815824.789, "0" ], [ 1724815829.797, "0" ], [ 1724815834.813, "0" ], [ 1724815839.798, "0" ], [ 1724815844.792, "0" ], [ 1724815849.795, "0" ], [ 1724815854.799, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815744.77, "0" ], [ 1724815749.767, "0" ], [ 1724815754.766, "0" ], [ 1724815759.766, "0" ], [ 1724815764.767, "0" ], [ 1724815769.768, "0" ], [ 1724815774.765, "0" ], [ 1724815779.767, "0" ], [ 1724815784.77, "0" ], [ 1724815789.767, "0" ], [ 1724815794.766, "0" ], [ 1724815799.766, "0" ], [ 1724815804.766, "0" ], [ 1724815809.766, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.775, "0" ], [ 1724815834.767, "0" ], [ 1724815839.769, "0" ], [ 1724815844.773, "0" ], [ 1724815849.767, "0" ], [ 1724815854.769, "0" ], [ 1724815859.803, "0" ] ], "Step2History": [ [ 1724815744.77, "0" ], [ 1724815749.767, "0" ], [ 1724815754.766, "0" ], [ 1724815759.766, "0" ], [ 1724815764.767, "0" ], [ 1724815769.768, "0" ], [ 1724815774.765, "0" ], [ 1724815779.767, "0" ], [ 1724815784.77, "0" ], [ 1724815789.767, "0" ], [ 1724815794.766, "0" ], [ 1724815799.766, "0" ], [ 1724815804.766, "0" ], [ 1724815809.766, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.775, "0" ], [ 1724815834.767, "0" ], [ 1724815839.769, "0" ], [ 1724815844.773, "0" ], [ 1724815849.767, "0" ], [ 1724815854.769, "0" ], [ 1724815859.803, "0" ] ] } }, "health_tikv": { "172.30.65.197:1008": { "Step1History": [ [ 1724815744.785, "6054" ], [ 1724815749.776, "6054" ], [ 1724815754.768, "6055" ], [ 1724815759.771, "6055" ], [ 1724815764.769, "6055" ], [ 1724815769.801, "6055" ], [ 1724815774.809, "6055" ], [ 1724815779.775, "6055" ], [ 1724815784.775, "6055" ], [ 1724815789.821, "6055" ], [ 1724815794.82, "6055" ], [ 1724815799.843, "6055" ], [ 1724815804.77, "6055" ], [ 1724815809.775, "6056" ], [ 1724815814.799, "6056" ], [ 1724815819.795, "6056" ], [ 1724815824.769, "6056" ], [ 1724815829.769, "6056" ], [ 1724815834.77, "6056" ], [ 1724815839.803, "6056" ], [ 1724815844.77, "6056" ], [ 1724815849.77, "6057" ], [ 1724815854.773, "6057" ], [ 1724815859.768, "6057" ] ], "Step2History": [ [ 1724815744.785, "1" ], [ 1724815749.776, "1" ], [ 1724815754.768, "2" ], [ 1724815759.771, "2" ], [ 1724815764.769, "2" ], [ 1724815769.801, "2" ], [ 1724815774.809, "2" ], [ 1724815779.775, "2" ], [ 1724815784.775, "2" ], [ 1724815789.821, "2" ], [ 1724815794.82, "2" ], [ 1724815799.843, "2" ], [ 1724815804.77, "2" ], [ 1724815809.775, "3" ], [ 1724815814.799, "3" ], [ 1724815819.795, "3" ], [ 1724815824.769, "3" ], [ 1724815829.769, "3" ], [ 1724815834.77, "3" ], [ 1724815839.803, "3" ], [ 1724815844.77, "3" ], [ 1724815849.77, "4" ], [ 1724815854.773, "4" ], [ 1724815859.768, "3" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815744.771, "5965" ], [ 1724815749.771, "5965" ], [ 1724815754.769, "5965" ], [ 1724815759.78, "5966" ], [ 1724815764.77, "5966" ], [ 1724815769.77, "5966" ], [ 1724815774.771, "5967" ], [ 1724815779.771, "5967" ], [ 1724815784.77, "5967" ], [ 1724815789.77, "5967" ], [ 1724815794.77, "5967" ], [ 1724815799.77, "5967" ], [ 1724815804.77, "5967" ], [ 1724815809.77, "5968" ], [ 1724815814.771, "5968" ], [ 1724815819.78, "5968" ], [ 1724815824.775, "5968" ], [ 1724815829.77, "5968" ], [ 1724815834.772, "5968" ], [ 1724815839.772, "5968" ], [ 1724815844.77, "5968" ], [ 1724815849.793, "5970" ], [ 1724815854.787, "5970" ], [ 1724815859.789, "5970" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.771, "1" ], [ 1724815754.769, "1" ], [ 1724815759.78, "2" ], [ 1724815764.77, "2" ], [ 1724815769.77, "2" ], [ 1724815774.771, "3" ], [ 1724815779.771, "3" ], [ 1724815784.77, "3" ], [ 1724815789.77, "3" ], [ 1724815794.77, "3" ], [ 1724815799.77, "3" ], [ 1724815804.77, "3" ], [ 1724815809.77, "4" ], [ 1724815814.771, "4" ], [ 1724815819.78, "4" ], [ 1724815824.775, "4" ], [ 1724815829.77, "4" ], [ 1724815834.772, "4" ], [ 1724815839.772, "4" ], [ 1724815844.77, "4" ], [ 1724815849.793, "6" ], [ 1724815854.787, "6" ], [ 1724815859.789, "5" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815744.765, "378" ], [ 1724815749.765, "378" ], [ 1724815754.765, "378" ], [ 1724815759.765, "378" ], [ 1724815764.765, "378" ], [ 1724815769.765, "378" ], [ 1724815774.765, "378" ], [ 1724815779.765, "378" ], [ 1724815784.766, "378" ], [ 1724815789.765, "378" ], [ 1724815794.765, "378" ], [ 1724815799.765, "378" ], [ 1724815804.765, "378" ], [ 1724815809.765, "378" ], [ 1724815814.766, "378" ], [ 1724815819.765, "378" ], [ 1724815824.765, "378" ], [ 1724815829.765, "378" ], [ 1724815834.766, "378" ], [ 1724815839.765, "378" ], [ 1724815844.765, "378" ], [ 1724815849.765, "378" ], [ 1724815854.765, "378" ], [ 1724815859.765, "378" ] ], "Step2History": [ [ 1724815744.765, "0" ], [ 1724815749.765, "0" ], [ 1724815754.765, "0" ], [ 1724815759.765, "0" ], [ 1724815764.765, "0" ], [ 1724815769.765, "0" ], [ 1724815774.765, "0" ], [ 1724815779.765, "0" ], [ 1724815784.766, "0" ], [ 1724815789.765, "0" ], [ 1724815794.765, "0" ], [ 1724815799.765, "0" ], [ 1724815804.765, "0" ], [ 1724815809.765, "0" ], [ 1724815814.766, "0" ], [ 1724815819.765, "0" ], [ 1724815824.765, "0" ], [ 1724815829.765, "0" ], [ 1724815834.766, "0" ], [ 1724815839.765, "0" ], [ 1724815844.765, "0" ], [ 1724815849.765, "0" ], [ 1724815854.765, "0" ], [ 1724815859.765, "0" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815744.908, "1490" ], [ 1724815749.834, "1490" ], [ 1724815754.769, "1491" ], [ 1724815759.77, "1491" ], [ 1724815764.769, "1491" ], [ 1724815769.795, "1499" ], [ 1724815774.769, "1505" ], [ 1724815779.779, "1505" ], [ 1724815784.769, "1505" ], [ 1724815789.778, "1505" ], [ 1724815794.782, "1505" ], [ 1724815799.769, "1507" ], [ 1724815804.783, "1507" ], [ 1724815809.768, "1508" ], [ 1724815814.77, "1508" ], [ 1724815819.768, "1508" ], [ 1724815824.767, "1508" ], [ 1724815829.767, "1508" ], [ 1724815834.768, "1508" ], [ 1724815839.769, "1508" ], [ 1724815844.767, "1508" ], [ 1724815849.772, "1508" ], [ 1724815854.782, "1508" ], [ 1724815859.781, "1508" ] ], "Step2History": [ [ 1724815744.908, "2" ], [ 1724815749.834, "2" ], [ 1724815754.769, "3" ], [ 1724815759.77, "3" ], [ 1724815764.769, "3" ], [ 1724815769.795, "11" ], [ 1724815774.769, "17" ], [ 1724815779.779, "17" ], [ 1724815784.769, "17" ], [ 1724815789.778, "17" ], [ 1724815794.782, "17" ], [ 1724815799.769, "19" ], [ 1724815804.783, "19" ], [ 1724815809.768, "20" ], [ 1724815814.77, "19" ], [ 1724815819.768, "19" ], [ 1724815824.767, "19" ], [ 1724815829.767, "19" ], [ 1724815834.768, "19" ], [ 1724815839.769, "19" ], [ 1724815844.767, "19" ], [ 1724815849.772, "19" ], [ 1724815854.782, "19" ], [ 1724815859.781, "18" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815744.795, "7104" ], [ 1724815749.78, "7104" ], [ 1724815754.772, "7105" ], [ 1724815759.791, "7105" ], [ 1724815764.788, "7105" ], [ 1724815769.796, "7107" ], [ 1724815774.779, "7107" ], [ 1724815779.798, "7107" ], [ 1724815784.776, "7107" ], [ 1724815789.775, "7107" ], [ 1724815794.779, "7107" ], [ 1724815799.782, "7107" ], [ 1724815804.783, "7107" ], [ 1724815809.787, "7108" ], [ 1724815814.786, "7108" ], [ 1724815819.773, "7108" ], [ 1724815824.773, "7108" ], [ 1724815829.779, "7108" ], [ 1724815834.77, "7108" ], [ 1724815839.77, "7108" ], [ 1724815844.77, "7108" ], [ 1724815849.769, "7108" ], [ 1724815854.774, "7108" ], [ 1724815859.77, "7108" ] ], "Step2History": [ [ 1724815744.795, "3" ], [ 1724815749.78, "3" ], [ 1724815754.772, "4" ], [ 1724815759.791, "4" ], [ 1724815764.788, "4" ], [ 1724815769.796, "6" ], [ 1724815774.779, "6" ], [ 1724815779.798, "6" ], [ 1724815784.776, "5" ], [ 1724815789.775, "5" ], [ 1724815794.779, "5" ], [ 1724815799.782, "5" ], [ 1724815804.783, "5" ], [ 1724815809.787, "6" ], [ 1724815814.786, "6" ], [ 1724815819.773, "5" ], [ 1724815824.773, "5" ], [ 1724815829.779, "5" ], [ 1724815834.77, "5" ], [ 1724815839.77, "5" ], [ 1724815844.77, "5" ], [ 1724815849.769, "5" ], [ 1724815854.774, "5" ], [ 1724815859.77, "4" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815744.777, "5861" ], [ 1724815749.772, "5861" ], [ 1724815754.771, "5862" ], [ 1724815759.78, "5862" ], [ 1724815764.778, "5862" ], [ 1724815769.774, "5865" ], [ 1724815774.775, "5867" ], [ 1724815779.783, "5867" ], [ 1724815784.79, "5867" ], [ 1724815789.777, "5867" ], [ 1724815794.771, "5867" ], [ 1724815799.773, "5868" ], [ 1724815804.79, "5868" ], [ 1724815809.773, "5869" ], [ 1724815814.772, "5869" ], [ 1724815819.772, "5869" ], [ 1724815824.772, "5869" ], [ 1724815829.775, "5869" ], [ 1724815834.78, "5870" ], [ 1724815839.776, "5870" ], [ 1724815844.771, "5870" ], [ 1724815849.773, "5871" ], [ 1724815854.771, "5871" ], [ 1724815859.782, "5871" ] ], "Step2History": [ [ 1724815744.777, "2" ], [ 1724815749.772, "2" ], [ 1724815754.771, "3" ], [ 1724815759.78, "3" ], [ 1724815764.778, "3" ], [ 1724815769.774, "6" ], [ 1724815774.775, "8" ], [ 1724815779.783, "7" ], [ 1724815784.79, "7" ], [ 1724815789.777, "7" ], [ 1724815794.771, "7" ], [ 1724815799.773, "8" ], [ 1724815804.79, "8" ], [ 1724815809.773, "9" ], [ 1724815814.772, "9" ], [ 1724815819.772, "9" ], [ 1724815824.772, "9" ], [ 1724815829.775, "9" ], [ 1724815834.78, "10" ], [ 1724815839.776, "10" ], [ 1724815844.771, "10" ], [ 1724815849.773, "11" ], [ 1724815854.771, "11" ], [ 1724815859.782, "10" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815739.92, "6087" ], [ 1724815744.841, "6087" ], [ 1724815749.819, "6087" ], [ 1724815754.851, "6089" ], [ 1724815759.849, "6089" ], [ 1724815764.858, "6089" ], [ 1724815769.814, "6092" ], [ 1724815774.814, "6102" ], [ 1724815779.781, "6102" ], [ 1724815784.794, "6104" ], [ 1724815789.789, "6109" ], [ 1724815794.912, "6110" ], [ 1724815799.933, "6111" ], [ 1724815804.806, "6111" ], [ 1724815809.779, "6114" ], [ 1724815814.813, "6114" ], [ 1724815819.783, "6115" ], [ 1724815824.812, "6115" ], [ 1724815829.78, "6115" ], [ 1724815834.795, "6115" ], [ 1724815839.889, "6116" ], [ 1724815844.778, "6116" ], [ 1724815850.034, "6120" ], [ 1724815854.893, "6120" ], [ 1724815859.853, "6120" ] ], "Step2History": [ [ 1724815739.92, "63" ], [ 1724815744.841, "62" ], [ 1724815749.819, "62" ], [ 1724815754.851, "64" ], [ 1724815759.849, "64" ], [ 1724815764.858, "64" ], [ 1724815769.814, "67" ], [ 1724815774.814, "77" ], [ 1724815779.781, "76" ], [ 1724815784.794, "78" ], [ 1724815789.789, "83" ], [ 1724815794.912, "84" ], [ 1724815799.933, "73" ], [ 1724815804.806, "73" ], [ 1724815809.779, "76" ], [ 1724815814.813, "76" ], [ 1724815819.783, "74" ], [ 1724815824.812, "43" ], [ 1724815829.78, "43" ], [ 1724815834.795, "42" ], [ 1724815839.889, "43" ], [ 1724815844.778, "42" ], [ 1724815850.034, "46" ], [ 1724815854.893, "45" ], [ 1724815859.853, "33" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815744.766, "508" ], [ 1724815749.765, "508" ], [ 1724815754.767, "509" ], [ 1724815759.77, "509" ], [ 1724815764.765, "509" ], [ 1724815769.77, "509" ], [ 1724815774.765, "509" ], [ 1724815779.766, "509" ], [ 1724815784.771, "509" ], [ 1724815789.768, "509" ], [ 1724815794.765, "509" ], [ 1724815799.766, "509" ], [ 1724815804.765, "509" ], [ 1724815809.769, "510" ], [ 1724815814.765, "510" ], [ 1724815819.765, "510" ], [ 1724815824.766, "510" ], [ 1724815829.766, "510" ], [ 1724815834.765, "510" ], [ 1724815839.765, "510" ], [ 1724815844.77, "510" ], [ 1724815849.877, "510" ], [ 1724815854.766, "510" ], [ 1724815859.766, "510" ] ], "Step2History": [ [ 1724815744.766, "1" ], [ 1724815749.765, "1" ], [ 1724815754.767, "2" ], [ 1724815759.77, "2" ], [ 1724815764.765, "2" ], [ 1724815769.77, "2" ], [ 1724815774.765, "2" ], [ 1724815779.766, "2" ], [ 1724815784.771, "2" ], [ 1724815789.768, "2" ], [ 1724815794.765, "2" ], [ 1724815799.766, "2" ], [ 1724815804.765, "2" ], [ 1724815809.769, "3" ], [ 1724815814.765, "3" ], [ 1724815819.765, "3" ], [ 1724815824.766, "3" ], [ 1724815829.766, "3" ], [ 1724815834.765, "3" ], [ 1724815839.765, "3" ], [ 1724815844.77, "3" ], [ 1724815849.877, "3" ], [ 1724815854.766, "3" ], [ 1724815859.766, "2" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815744.771, "5977" ], [ 1724815749.77, "5977" ], [ 1724815754.77, "5978" ], [ 1724815759.771, "5978" ], [ 1724815764.837, "5978" ], [ 1724815769.817, "5978" ], [ 1724815774.772, "5978" ], [ 1724815779.773, "5978" ], [ 1724815784.771, "5978" ], [ 1724815789.776, "5978" ], [ 1724815794.771, "5978" ], [ 1724815799.771, "5978" ], [ 1724815804.771, "5978" ], [ 1724815809.771, "5979" ], [ 1724815814.806, "5979" ], [ 1724815819.793, "5979" ], [ 1724815824.77, "5979" ], [ 1724815829.77, "5979" ], [ 1724815834.77, "5979" ], [ 1724815839.807, "5979" ], [ 1724815844.771, "5979" ], [ 1724815849.777, "5979" ], [ 1724815854.77, "5979" ], [ 1724815859.771, "5979" ] ], "Step2History": [ [ 1724815744.771, "1" ], [ 1724815749.77, "1" ], [ 1724815754.77, "2" ], [ 1724815759.771, "2" ], [ 1724815764.837, "2" ], [ 1724815769.817, "2" ], [ 1724815774.772, "2" ], [ 1724815779.773, "2" ], [ 1724815784.771, "2" ], [ 1724815789.776, "2" ], [ 1724815794.771, "2" ], [ 1724815799.771, "2" ], [ 1724815804.771, "2" ], [ 1724815809.771, "3" ], [ 1724815814.806, "3" ], [ 1724815819.793, "3" ], [ 1724815824.77, "3" ], [ 1724815829.77, "3" ], [ 1724815834.77, "3" ], [ 1724815839.807, "3" ], [ 1724815844.771, "3" ], [ 1724815849.777, "3" ], [ 1724815854.77, "3" ], [ 1724815859.771, "2" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815744.811, "6229" ], [ 1724815749.776, "6229" ], [ 1724815754.775, "6230" ], [ 1724815759.783, "6230" ], [ 1724815764.797, "6230" ], [ 1724815769.804, "6230" ], [ 1724815774.782, "6234" ], [ 1724815779.777, "6234" ], [ 1724815784.782, "6236" ], [ 1724815789.778, "6236" ], [ 1724815794.785, "6236" ], [ 1724815799.781, "6236" ], [ 1724815804.776, "6236" ], [ 1724815809.774, "6237" ], [ 1724815814.789, "6237" ], [ 1724815819.776, "6237" ], [ 1724815824.776, "6238" ], [ 1724815829.775, "6238" ], [ 1724815834.776, "6238" ], [ 1724815839.841, "6238" ], [ 1724815844.775, "6238" ], [ 1724815849.782, "6238" ], [ 1724815854.789, "6238" ], [ 1724815859.863, "6238" ] ], "Step2History": [ [ 1724815744.811, "5" ], [ 1724815749.776, "5" ], [ 1724815754.775, "6" ], [ 1724815759.783, "6" ], [ 1724815764.797, "6" ], [ 1724815769.804, "6" ], [ 1724815774.782, "9" ], [ 1724815779.777, "9" ], [ 1724815784.782, "11" ], [ 1724815789.778, "11" ], [ 1724815794.785, "11" ], [ 1724815799.781, "11" ], [ 1724815804.776, "10" ], [ 1724815809.774, "11" ], [ 1724815814.789, "11" ], [ 1724815819.776, "9" ], [ 1724815824.776, "10" ], [ 1724815829.775, "10" ], [ 1724815834.776, "10" ], [ 1724815839.841, "10" ], [ 1724815844.775, "10" ], [ 1724815849.782, "10" ], [ 1724815854.789, "10" ], [ 1724815859.863, "9" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815744.792, "1256663" ], [ 1724815749.792, "1256663" ], [ 1724815754.816, "1256663" ], [ 1724815759.808, "1256663" ], [ 1724815764.794, "1256663" ], [ 1724815769.806, "1256663" ], [ 1724815774.795, "1256663" ], [ 1724815779.796, "1256663" ], [ 1724815784.791, "1256663" ], [ 1724815789.797, "1256663" ], [ 1724815794.794, "1256663" ], [ 1724815799.801, "1256663" ], [ 1724815804.809, "1256663" ], [ 1724815809.808, "1256663" ], [ 1724815814.797, "1256663" ], [ 1724815819.796, "1256663" ], [ 1724815824.808, "1256663" ], [ 1724815829.792, "1256663" ], [ 1724815834.793, "1256663" ], [ 1724815839.792, "1256663" ], [ 1724815844.791, "1256663" ], [ 1724815849.792, "1256663" ], [ 1724815854.804, "1256663" ], [ 1724815859.79, "1256663" ] ], "Step2History": [ [ 1724815744.792, "0" ], [ 1724815749.792, "0" ], [ 1724815754.816, "0" ], [ 1724815759.808, "0" ], [ 1724815764.794, "0" ], [ 1724815769.806, "0" ], [ 1724815774.795, "0" ], [ 1724815779.796, "0" ], [ 1724815784.791, "0" ], [ 1724815789.797, "0" ], [ 1724815794.794, "0" ], [ 1724815799.801, "0" ], [ 1724815804.809, "0" ], [ 1724815809.808, "0" ], [ 1724815814.797, "0" ], [ 1724815819.796, "0" ], [ 1724815824.808, "0" ], [ 1724815829.792, "0" ], [ 1724815834.793, "0" ], [ 1724815839.792, "0" ], [ 1724815844.791, "0" ], [ 1724815849.792, "0" ], [ 1724815854.804, "0" ], [ 1724815859.79, "0" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815744.785, "239099" ], [ 1724815749.785, "239099" ], [ 1724815754.785, "239099" ], [ 1724815759.795, "239099" ], [ 1724815764.791, "239099" ], [ 1724815769.804, "239099" ], [ 1724815774.789, "239099" ], [ 1724815779.793, "239099" ], [ 1724815784.791, "239099" ], [ 1724815789.79, "239099" ], [ 1724815794.789, "239099" ], [ 1724815799.789, "239099" ], [ 1724815804.811, "239099" ], [ 1724815809.793, "239099" ], [ 1724815814.798, "239099" ], [ 1724815819.791, "239099" ], [ 1724815824.789, "239099" ], [ 1724815829.797, "239099" ], [ 1724815834.813, "239099" ], [ 1724815839.798, "239099" ], [ 1724815844.792, "239099" ], [ 1724815849.795, "239099" ], [ 1724815854.799, "239099" ], [ 1724815859.789, "239099" ] ], "Step2History": [ [ 1724815744.785, "0" ], [ 1724815749.785, "0" ], [ 1724815754.785, "0" ], [ 1724815759.795, "0" ], [ 1724815764.791, "0" ], [ 1724815769.804, "0" ], [ 1724815774.789, "0" ], [ 1724815779.793, "0" ], [ 1724815784.791, "0" ], [ 1724815789.79, "0" ], [ 1724815794.789, "0" ], [ 1724815799.789, "0" ], [ 1724815804.811, "0" ], [ 1724815809.793, "0" ], [ 1724815814.798, "0" ], [ 1724815819.791, "0" ], [ 1724815824.789, "0" ], [ 1724815829.797, "0" ], [ 1724815834.813, "0" ], [ 1724815839.798, "0" ], [ 1724815844.792, "0" ], [ 1724815849.795, "0" ], [ 1724815854.799, "0" ], [ 1724815859.789, "0" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815744.77, "887" ], [ 1724815749.767, "887" ], [ 1724815754.766, "887" ], [ 1724815759.766, "888" ], [ 1724815764.767, "888" ], [ 1724815769.768, "888" ], [ 1724815774.765, "888" ], [ 1724815779.767, "888" ], [ 1724815784.77, "888" ], [ 1724815789.767, "888" ], [ 1724815794.766, "888" ], [ 1724815799.766, "888" ], [ 1724815804.766, "888" ], [ 1724815809.766, "888" ], [ 1724815814.766, "888" ], [ 1724815819.765, "888" ], [ 1724815824.765, "889" ], [ 1724815829.775, "889" ], [ 1724815834.767, "889" ], [ 1724815839.769, "889" ], [ 1724815844.773, "889" ], [ 1724815849.767, "889" ], [ 1724815854.769, "889" ], [ 1724815859.803, "889" ] ], "Step2History": [ [ 1724815744.77, "1" ], [ 1724815749.767, "1" ], [ 1724815754.766, "1" ], [ 1724815759.766, "2" ], [ 1724815764.767, "2" ], [ 1724815769.768, "2" ], [ 1724815774.765, "2" ], [ 1724815779.767, "2" ], [ 1724815784.77, "2" ], [ 1724815789.767, "2" ], [ 1724815794.766, "2" ], [ 1724815799.766, "2" ], [ 1724815804.766, "2" ], [ 1724815809.766, "2" ], [ 1724815814.766, "2" ], [ 1724815819.765, "2" ], [ 1724815824.765, "3" ], [ 1724815829.775, "3" ], [ 1724815834.767, "3" ], [ 1724815839.769, "3" ], [ 1724815844.773, "3" ], [ 1724815849.767, "3" ], [ 1724815854.769, "3" ], [ 1724815859.803, "3" ] ] } }, "memory": { "172.30.65.197:1008": { "Step1History": [ [ 1724815804.77, "0.24263713228970615" ], [ 1724815809.775, "0.2380175909461568" ], [ 1724815814.799, "0.20954973315366215" ], [ 1724815819.795, "0.2129663933239538" ], [ 1724815824.769, "0.22278062174639884" ], [ 1724815829.769, "0.23035520941219562" ], [ 1724815834.77, "0.2402673784218882" ], [ 1724815839.803, "0.24422311397859042" ], [ 1724815844.77, "0.23774517628687003" ], [ 1724815849.77, "0.22118337967414184" ], [ 1724815854.773, "0.1759448513443784" ], [ 1724815859.768, "0.19206860666985917" ] ], "Step2History": [ [ 1724815804.77, "0.24263713228970615" ], [ 1724815809.775, "0.2380175909461568" ], [ 1724815814.799, "0.20954973315366215" ], [ 1724815819.795, "0.2129663933239538" ], [ 1724815824.769, "0.22278062174639884" ], [ 1724815829.769, "0.23035520941219562" ], [ 1724815834.77, "0.2402673784218882" ], [ 1724815839.803, "0.24422311397859042" ], [ 1724815844.77, "0.23774517628687003" ], [ 1724815849.77, "0.22118337967414184" ], [ 1724815854.773, "0.1759448513443784" ], [ 1724815859.768, "0.19206860666985917" ] ] }, "172.30.65.210:1008": { "Step1History": [ [ 1724815804.77, "0.18209772280400868" ], [ 1724815809.77, "0.15836163896736097" ], [ 1724815814.771, "0.1416561177910345" ], [ 1724815819.78, "0.1416561177910345" ], [ 1724815824.775, "0.1416561177910345" ], [ 1724815829.77, "0.14430156544076977" ], [ 1724815834.772, "0.153477849835982" ], [ 1724815839.772, "0.206075224310549" ], [ 1724815844.77, "0.20064233681415294" ], [ 1724815849.793, "0.2301867367532726" ], [ 1724815854.787, "0.22245635982740677" ], [ 1724815859.789, "0.20878353212240902" ] ], "Step2History": [ [ 1724815804.77, "0.18209772280400868" ], [ 1724815809.77, "0.15836163896736097" ], [ 1724815814.771, "0.1416561177910345" ], [ 1724815819.78, "0.1416561177910345" ], [ 1724815824.775, "0.1416561177910345" ], [ 1724815829.77, "0.14430156544076977" ], [ 1724815834.772, "0.153477849835982" ], [ 1724815839.772, "0.206075224310549" ], [ 1724815844.77, "0.20064233681415294" ], [ 1724815849.793, "0.2301867367532726" ], [ 1724815854.787, "0.22245635982740677" ], [ 1724815859.789, "0.20878353212240902" ] ] }, "172.30.66.182:1008": { "Step1History": [ [ 1724815804.765, "0.027509673411760048" ], [ 1724815809.765, "0.027509673411760048" ], [ 1724815814.766, "0.027509673411760048" ], [ 1724815819.765, "0.027509673411760048" ], [ 1724815824.765, "0.027509673411760048" ], [ 1724815829.765, "0.027509673411760048" ], [ 1724815834.766, "0.027509673411760048" ], [ 1724815839.765, "0.027509673411760048" ], [ 1724815844.765, "0.027509673411760048" ], [ 1724815849.765, "0.027509673411760048" ], [ 1724815854.765, "0.027509673411760048" ], [ 1724815859.765, "0.027509673411760048" ] ], "Step2History": [ [ 1724815804.765, "0.027509673411760048" ], [ 1724815809.765, "0.027509673411760048" ], [ 1724815814.766, "0.027509673411760048" ], [ 1724815819.765, "0.027509673411760048" ], [ 1724815824.765, "0.027509673411760048" ], [ 1724815829.765, "0.027509673411760048" ], [ 1724815834.766, "0.027509673411760048" ], [ 1724815839.765, "0.027509673411760048" ], [ 1724815844.765, "0.027509673411760048" ], [ 1724815849.765, "0.027509673411760048" ], [ 1724815854.765, "0.027509673411760048" ], [ 1724815859.765, "0.027509673411760048" ] ] }, "172.30.69.27:1008": { "Step1History": [ [ 1724815804.783, "0.18863338542807398" ], [ 1724815809.768, "0.13688103466836934" ], [ 1724815814.77, "0.13213998017677564" ], [ 1724815819.768, "0.13874147273440468" ], [ 1724815824.767, "0.1547977895001256" ], [ 1724815829.767, "0.15743847314151085" ], [ 1724815834.768, "0.22739156312682285" ], [ 1724815839.769, "0.2176118582973558" ], [ 1724815844.767, "0.2380711087022965" ], [ 1724815849.772, "0.22479901453084417" ], [ 1724815854.782, "0.20415551455621098" ], [ 1724815859.781, "0.187785453812011" ] ], "Step2History": [ [ 1724815804.783, "0.18863338542807398" ], [ 1724815809.768, "0.13688103466836934" ], [ 1724815814.77, "0.13213998017677564" ], [ 1724815819.768, "0.13874147273440468" ], [ 1724815824.767, "0.1547977895001256" ], [ 1724815829.767, "0.15743847314151085" ], [ 1724815834.768, "0.22739156312682285" ], [ 1724815839.769, "0.2176118582973558" ], [ 1724815844.767, "0.2380711087022965" ], [ 1724815849.772, "0.22479901453084417" ], [ 1724815854.782, "0.20415551455621098" ], [ 1724815859.781, "0.187785453812011" ] ] }, "172.30.77.112:1008": { "Step1History": [ [ 1724815804.783, "0.17858411197028248" ], [ 1724815809.787, "0.17929679524539593" ], [ 1724815814.786, "0.176972887224167" ], [ 1724815819.773, "0.17505187819482443" ], [ 1724815824.773, "0.21841549082278755" ], [ 1724815829.779, "0.1276186269014271" ], [ 1724815834.77, "0.1276186269014271" ], [ 1724815839.77, "0.09252824067027741" ], [ 1724815844.77, "0.043529981698783506" ], [ 1724815849.769, "0.044682376757578794" ], [ 1724815854.774, "0.044682376757578794" ], [ 1724815859.77, "0.044499983295035656" ] ], "Step2History": [ [ 1724815804.783, "0.17858411197028248" ], [ 1724815809.787, "0.17929679524539593" ], [ 1724815814.786, "0.176972887224167" ], [ 1724815819.773, "0.17505187819482443" ], [ 1724815824.773, "0.21841549082278755" ], [ 1724815829.779, "0.1276186269014271" ], [ 1724815834.77, "0.1276186269014271" ], [ 1724815839.77, "0.09252824067027741" ], [ 1724815844.77, "0.043529981698783506" ], [ 1724815849.769, "0.044682376757578794" ], [ 1724815854.774, "0.044682376757578794" ], [ 1724815859.77, "0.044499983295035656" ] ] }, "172.30.77.5:1008": { "Step1History": [ [ 1724815804.79, "0.16319500385451585" ], [ 1724815809.773, "0.16071375981726008" ], [ 1724815814.772, "0.16013657236404943" ], [ 1724815819.772, "0.16013657236404943" ], [ 1724815824.772, "0.15959161930499924" ], [ 1724815829.775, "0.15918995771787914" ], [ 1724815834.78, "0.165552693025863" ], [ 1724815839.776, "0.16010625594727665" ], [ 1724815844.771, "0.16010625594727665" ], [ 1724815849.773, "0.15969079729701302" ], [ 1724815854.771, "0.1759422527943693" ], [ 1724815859.782, "0.17331598491851072" ] ], "Step2History": [ [ 1724815804.79, "0.16319500385451585" ], [ 1724815809.773, "0.16071375981726008" ], [ 1724815814.772, "0.16013657236404943" ], [ 1724815819.772, "0.16013657236404943" ], [ 1724815824.772, "0.15959161930499924" ], [ 1724815829.775, "0.15918995771787914" ], [ 1724815834.78, "0.165552693025863" ], [ 1724815839.776, "0.16010625594727665" ], [ 1724815844.771, "0.16010625594727665" ], [ 1724815849.773, "0.15969079729701302" ], [ 1724815854.771, "0.1759422527943693" ], [ 1724815859.782, "0.17331598491851072" ] ] }, "172.30.81.38:1008": { "Step1History": [ [ 1724815799.933, "0.3140681166575717" ], [ 1724815804.806, "0.3130256650122565" ], [ 1724815809.779, "0.31266465217170725" ], [ 1724815814.813, "0.32144812242387794" ], [ 1724815819.783, "0.31690641384012486" ], [ 1724815824.812, "0.31225352443812543" ], [ 1724815829.78, "0.31632687531785836" ], [ 1724815834.795, "0.3130504749778195" ], [ 1724815839.889, "0.3176026396318473" ], [ 1724815844.778, "0.31197758317525487" ], [ 1724815850.034, "0.3199667014377406" ], [ 1724815854.893, "0.3129332308762187" ], [ 1724815859.853, "0.31149313920927363" ] ], "Step2History": [ [ 1724815799.933, "0.3140681166575717" ], [ 1724815804.806, "0.3130256650122565" ], [ 1724815809.779, "0.31266465217170725" ], [ 1724815814.813, "0.32144812242387794" ], [ 1724815819.783, "0.31690641384012486" ], [ 1724815824.812, "0.31225352443812543" ], [ 1724815829.78, "0.31632687531785836" ], [ 1724815834.795, "0.3130504749778195" ], [ 1724815839.889, "0.3176026396318473" ], [ 1724815844.778, "0.31197758317525487" ], [ 1724815850.034, "0.3199667014377406" ], [ 1724815854.893, "0.3129332308762187" ], [ 1724815859.853, "0.31149313920927363" ] ] }, "172.30.83.151:1008": { "Step1History": [ [ 1724815804.765, "0.16552095359360905" ], [ 1724815809.769, "0.16944129937399693" ], [ 1724815814.765, "0.16944129937399693" ], [ 1724815819.765, "0.16073535253043086" ], [ 1724815824.766, "0.15882207728087727" ], [ 1724815829.766, "0.15821141802873998" ], [ 1724815834.765, "0.16320620236765027" ], [ 1724815839.765, "0.1592533747121487" ], [ 1724815844.77, "0.19570960832426934" ], [ 1724815849.877, "0.24901100424058614" ], [ 1724815854.766, "0.2290746192196183" ], [ 1724815859.766, "0.23411413574082804" ] ], "Step2History": [ [ 1724815804.765, "0.16552095359360905" ], [ 1724815809.769, "0.16944129937399693" ], [ 1724815814.765, "0.16944129937399693" ], [ 1724815819.765, "0.16073535253043086" ], [ 1724815824.766, "0.15882207728087727" ], [ 1724815829.766, "0.15821141802873998" ], [ 1724815834.765, "0.16320620236765027" ], [ 1724815839.765, "0.1592533747121487" ], [ 1724815844.77, "0.19570960832426934" ], [ 1724815849.877, "0.24901100424058614" ], [ 1724815854.766, "0.2290746192196183" ], [ 1724815859.766, "0.23411413574082804" ] ] }, "172.30.84.204:1008": { "Step1History": [ [ 1724815804.771, "0.23773125548324986" ], [ 1724815809.771, "0.24411657342821752" ], [ 1724815814.806, "0.22533153167199368" ], [ 1724815819.793, "0.21710650219082514" ], [ 1724815824.77, "0.2369101755506142" ], [ 1724815829.77, "0.24715446399956442" ], [ 1724815834.77, "0.22005486652733489" ], [ 1724815839.807, "0.25203850061189664" ], [ 1724815844.771, "0.23628992641153856" ], [ 1724815849.777, "0.22982621887462987" ], [ 1724815854.77, "0.22690049904534224" ], [ 1724815859.771, "0.22760148884541473" ] ], "Step2History": [ [ 1724815804.771, "0.23773125548324986" ], [ 1724815809.771, "0.24411657342821752" ], [ 1724815814.806, "0.22533153167199368" ], [ 1724815819.793, "0.21710650219082514" ], [ 1724815824.77, "0.2369101755506142" ], [ 1724815829.77, "0.24715446399956442" ], [ 1724815834.77, "0.22005486652733489" ], [ 1724815839.807, "0.25203850061189664" ], [ 1724815844.771, "0.23628992641153856" ], [ 1724815849.777, "0.22982621887462987" ], [ 1724815854.77, "0.22690049904534224" ], [ 1724815859.771, "0.22760148884541473" ] ] }, "172.30.87.250:1008": { "Step1History": [ [ 1724815804.776, "0.1797639774148882" ], [ 1724815809.774, "0.17198794025314826" ], [ 1724815814.789, "0.1781031327376467" ], [ 1724815819.776, "0.24580624963651235" ], [ 1724815824.776, "0.23315848803986422" ], [ 1724815829.775, "0.2277457702411578" ], [ 1724815834.776, "0.25882294346421364" ], [ 1724815839.841, "0.2529507772758039" ], [ 1724815844.775, "0.24565540999550822" ], [ 1724815849.782, "0.2440245105136096" ], [ 1724815854.789, "0.2345670258852703" ], [ 1724815859.863, "0.25535777702708556" ] ], "Step2History": [ [ 1724815804.776, "0.1797639774148882" ], [ 1724815809.774, "0.17198794025314826" ], [ 1724815814.789, "0.1781031327376467" ], [ 1724815819.776, "0.24580624963651235" ], [ 1724815824.776, "0.23315848803986422" ], [ 1724815829.775, "0.2277457702411578" ], [ 1724815834.776, "0.25882294346421364" ], [ 1724815839.841, "0.2529507772758039" ], [ 1724815844.775, "0.24565540999550822" ], [ 1724815849.782, "0.2440245105136096" ], [ 1724815854.789, "0.2345670258852703" ], [ 1724815859.863, "0.25535777702708556" ] ] }, "172.30.88.130:1008": { "Step1History": [ [ 1724815804.809, "0.315289990246793" ], [ 1724815809.808, "0.315828460900512" ], [ 1724815814.797, "0.3165582074426355" ], [ 1724815819.796, "0.31829108599076206" ], [ 1724815824.808, "0.32033803520501947" ], [ 1724815829.792, "0.3233179875793098" ], [ 1724815834.793, "0.3453882468581576" ], [ 1724815839.792, "0.3642132683807374" ], [ 1724815844.791, "0.38052076596899964" ], [ 1724815849.792, "0.39913946639401526" ], [ 1724815854.804, "0.41716359616841486" ], [ 1724815859.79, "0.4321467765639376" ] ], "Step2History": [ [ 1724815804.809, "0.315289990246793" ], [ 1724815809.808, "0.315828460900512" ], [ 1724815814.797, "0.3165582074426355" ], [ 1724815819.796, "0.31829108599076206" ], [ 1724815824.808, "0.32033803520501947" ], [ 1724815829.792, "0.3233179875793098" ], [ 1724815834.793, "0.3453882468581576" ], [ 1724815839.792, "0.3642132683807374" ], [ 1724815844.791, "0.38052076596899964" ], [ 1724815849.792, "0.39913946639401526" ], [ 1724815854.804, "0.41716359616841486" ], [ 1724815859.79, "0.4321467765639376" ] ] }, "172.30.88.59:1008": { "Step1History": [ [ 1724815804.811, "0.22900191206591125" ], [ 1724815809.793, "0.26508167877739275" ], [ 1724815814.798, "0.2658520797229772" ], [ 1724815819.791, "0.2669548304168547" ], [ 1724815824.789, "0.2685220576702732" ], [ 1724815829.797, "0.27810328420592595" ], [ 1724815834.813, "0.3006624126863914" ], [ 1724815839.798, "0.33820980336281214" ], [ 1724815844.792, "0.3648435194301485" ], [ 1724815849.795, "0.36148931555308894" ], [ 1724815854.799, "0.3196161115556831" ], [ 1724815859.789, "0.32142059729172373" ] ], "Step2History": [ [ 1724815804.811, "0.22900191206591125" ], [ 1724815809.793, "0.26508167877739275" ], [ 1724815814.798, "0.2658520797229772" ], [ 1724815819.791, "0.2669548304168547" ], [ 1724815824.789, "0.2685220576702732" ], [ 1724815829.797, "0.27810328420592595" ], [ 1724815834.813, "0.3006624126863914" ], [ 1724815839.798, "0.33820980336281214" ], [ 1724815844.792, "0.3648435194301485" ], [ 1724815849.795, "0.36148931555308894" ], [ 1724815854.799, "0.3196161115556831" ], [ 1724815859.789, "0.32142059729172373" ] ] }, "172.30.89.188:1008": { "Step1History": [ [ 1724815804.766, "0.15876088761518692" ], [ 1724815809.766, "0.15876905448664408" ], [ 1724815814.766, "0.1587933076200623" ], [ 1724815819.765, "0.1587933076200623" ], [ 1724815824.765, "0.15880141262128114" ], [ 1724815829.775, "0.16320979084147236" ], [ 1724815834.767, "0.16857610596144496" ], [ 1724815839.769, "0.16133351399442922" ], [ 1724815844.773, "0.24437147881006208" ], [ 1724815849.767, "0.2376609708925277" ], [ 1724815854.769, "0.23099500954657778" ], [ 1724815859.803, "0.2664031613216968" ] ], "Step2History": [ [ 1724815804.766, "0.15876088761518692" ], [ 1724815809.766, "0.15876905448664408" ], [ 1724815814.766, "0.1587933076200623" ], [ 1724815819.765, "0.1587933076200623" ], [ 1724815824.765, "0.15880141262128114" ], [ 1724815829.775, "0.16320979084147236" ], [ 1724815834.767, "0.16857610596144496" ], [ 1724815839.769, "0.16133351399442922" ], [ 1724815844.773, "0.24437147881006208" ], [ 1724815849.767, "0.2376609708925277" ], [ 1724815854.769, "0.23099500954657778" ], [ 1724815859.803, "0.2664031613216968" ] ] } } }

[djshow832]

I didn't see 172.30.72.136 in the metrics. Maybe the instance is unavailable.

[salarali]

Yeah. The server is actually not there anymore. The autoscaling removed it. But for some reason, tiproxy was unable to detect that.
I see this in the logs for tiproxy:

Aug 28 01:16:58 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:16:58.584 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:200] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev="down, err: Get \"http://172.30.72.136:1008/status\": dial tcp 172.30.72.136:1008: connect: connection refused: connect status port failed"] [cur=healthy]

But I still see this after this:

Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.010 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]
Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.019 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]

So for some reason, its list of backends didnt update properly.

[djshow832]

Yeah. The server is actually not there anymore. The autoscaling removed it. But for some reason, tiproxy was unable to detect that. I see this in the logs for tiproxy:

Aug 28 01:16:58 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:16:58.584 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:200] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev="down, err: Get \"http://172.30.72.136:1008/status\": dial tcp 172.30.72.136:1008: connect: connection refused: connect status port failed"] [cur=healthy]

But I still see this after this:

Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.010 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]
Aug 28 01:17:05 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 01:17:05.019 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.76.23:5678] [to=172.30.72.136:5678] [reason=cpu]

So for some reason, its list of backends didnt update properly.

The first log indicates that the instance was unhealthy but now is healthy. prev indicates the previous status and cur indicates the current status. I guess there should be another update backend afterward saying that it's down again.

[salarali]

Right. You are correct. Sorry.

Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.288 +00:00] [INFO] [main.nsmgr.observer] [observer/backend_observer.go:215] [update backend] [namespace=default] [backend_addr=172.30.72.136:5678] [prev=healthy] [cur="down, err: http status 500: connect status port failed"]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.289 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.77.5:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.299 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.309 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]
Aug 28 03:08:07 ip-172-30-94-236.ec2.internal tiproxy[3947488]: [2024/08/28 03:08:07.319 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:329] [backends to balance] [namespace=default] [from=172.30.72.136:5678] [to=172.30.81.38:5678] [reason=status]

After this failure, the reason did change to status.

[djshow832]

If there's no more problem, I'll close this issue.

[salarali]

So, the issue is still there. Even though tiproxy detected that the backend is down, its still trying to route from it. My naive guess is that this did not get run:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L298

[salarali]

I'm still testing this out to see if this is an issue, but one possible issue I can see is here:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L277-L283

As far as I can see, the backend wont be removed if there are still connections associated to it. Is that a possibility?

I will let you know if I confirm this hypothesis.

[salarali]

It seems like it is possible. I put some extra logs in there and I see this:

Aug 28 04:55:09 ip-172-30-94-236.ec2.internal tiproxy[4064141]: [2024/08/28 04:55:09.214 +00:00] [INFO] [main.nsmgr.router] [router/router_score.go:291] [current backends] [namespace=default] [addr=172.30.77.5:5678] [healthy=false] [connScore=-3] [connListLen=5]

I put this log:

	for addr, backend := range router.backends {
		router.logger.Info("current backends", zap.String("addr", addr), zap.Bool("healthy", backend.Healthy()), zap.Int("connScore", backend.connScore), zap.Int("connListLen", backend.connList.Len()))
	}

in this function:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L250

[djshow832]

Even though tiproxy detected that the backend is down, its still trying to route from it.

It's expected. TiProxy should migrate connections from the unhealthy backend. Why shouldn't it? The problem should be that the connection migrations never succeed. If so, the errors should be logged with redirect connection failed.

[connScore=-3] [connListLen=5]

This is unexpected. The connScore should always be >=0. connScore = connections on it + connections migrating to it - connections migrating from it. connections migrating from it should never be greater than connections on it.
Now that you can reproduce the problem, can you help to locate the reason?

[salarali]

Right, I am getting those errors.

Aug 28 10:19:30 ip-172-30-94-236.ec2.internal tiproxy[4092693]: [2024/08/28 10:19:30.290 +00:00] [WARN] [main.proxy.conn.be] [backend/backend_conn_mgr.go:598] [redirect connection failed] [connID=1237819] [client_addr=172.30.72.32:34772] [addr=0.0.0.0:5678] [ns=default] [from=172.30.78.14:5678] [to=172.30.87.124:5678] [redirect_err="this is an error from the backend connection: failed to read the connection: EOF"] [redirect_errVerbose="this is an error from the backend connection: failed to read the connection: EOF\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*basicReadWriter).Read\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:105\ngithub.com/pingcap/tiproxy/pkg/proxy/net.ReadFull\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:183\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).readOnePacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:247\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).ReadPacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:270\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*CmdProcessor).query\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/cmd_processor_query.go:30\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).querySessionStates\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:431\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).tryRedirect\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:510\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:458\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:460\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).Connect.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:214\ngithub.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithRecover.func1\n\t/Users/salar/repos/tiproxy/lib/util/waitgroup/waitgroup.go:38\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.22.4/libexec/src/runtime/asm_arm64.s:1222"]
Aug 28 10:19:30 ip-172-30-94-236.ec2.internal tiproxy[4092693]: [2024/08/28 10:19:30.343 +00:00] [WARN] [main.proxy.conn.be] [backend/backend_conn_mgr.go:598] [redirect connection failed] [connID=1239985] [client_addr=172.30.72.32:36640] [addr=0.0.0.0:5678] [ns=default] [from=172.30.78.14:5678] [to=172.30.87.124:5678] [redirect_err="this is an error from the backend connection: failed to read the connection: EOF"] [redirect_errVerbose="this is an error from the backend connection: failed to read the connection: EOF\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*basicReadWriter).Read\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:105\ngithub.com/pingcap/tiproxy/pkg/proxy/net.ReadFull\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:183\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).readOnePacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:247\ngithub.com/pingcap/tiproxy/pkg/proxy/net.(*PacketIO).ReadPacket\n\t/Users/salar/repos/tiproxy/pkg/proxy/net/packetio.go:270\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*CmdProcessor).query\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/cmd_processor_query.go:30\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).querySessionStates\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:431\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).tryRedirect\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:510\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:458\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).processSignals\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:460\ngithub.com/pingcap/tiproxy/pkg/proxy/backend.(*BackendConnManager).Connect.func1\n\t/Users/salar/repos/tiproxy/pkg/proxy/backend/backend_conn_mgr.go:214\ngithub.com/pingcap/tiproxy/lib/util/waitgroup.(*WaitGroup).RunWithRecover.func1\n\t/Users/salar/repos/tiproxy/lib/util/waitgroup/waitgroup.go:38\nruntime.goexit\n\t/opt/homebrew/Cellar/go/1.22.4/libexec/src/runtime/asm_arm64.s:1222"]

I will debug more and try to figure out the reason

[djshow832]

These errors should not be related to the problem. When the query fails, the connections are disconnected and then the router clears them.
https://github.com/pingcap/tiproxy/blob/v1.2.0/pkg/proxy/backend/backend_conn_mgr.go#L513

[salarali]

One thing I wanted to run by you. The reason why connScore might be negative, might be because of this:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L385

It decrements and increments the score, but the redirect can fail: https://github.com/pingcap/tiproxy/blob/main/pkg/proxy/backend/backend_conn_mgr.go#L595

That has fixed that the connScore is never negative for me, but I am still getting connLen to be greater than 0 for bad nodes, so still investigating that.

[djshow832]

One thing I wanted to run by you. The reason why connScore might be negative, might be because of this:
https://github.com/pingcap/tiproxy/blob/main/pkg/balance/router/router_score.go#L385

It decrements and increments the score, but the redirect can fail: https://github.com/pingcap/tiproxy/blob/main/pkg/proxy/backend/backend_conn_mgr.go#L595

Yes, that's possible, it should be handled. Appreciate that.