3.2.1-rc1

What's Changed

• Set default value of spark.tispark.telemetry.enable as false (#2621) by @ti-chi-bot in #2625
• service_safe_point (#2642) by @ti-chi-bot in #2646
• Probe tiflash (#2640) by @ti-chi-bot in #2648
• Expose forward and set it to false in default (#2641) by @ti-chi-bot in #2647
• Fix read from range column partition table error (#2639) by @ti-chi-bot in #2654
• Fix the bug jvm not exit when enableAuth (#2653) by @ti-chi-bot in #2664
• fix mutil_uniq (#2650) by @ti-chi-bot in #2666

Full Changelog: v3.2.0...v3.2.1-RC1

TiSpark v3.2.0

New Features

• Normalize the Java client in TiSpark and use the official client-java #2491
• Compatible with TiDB v6.5.0 #2598
• Support writing into the table with auto random primary key #2545
• Support follower read #2546
• Support writing into partition table with new collations #2565
• Support partition pruning with to_days function when read from TiKV/TiFlash #2593
• Support reading TiFlash load balancing with Round-Robin strategy client-java #662

Fixes

• Bump Spark version from 3.0.2 to 3.0.3, from 3.1.1 to 3.1.3, from 3.2.1 to 3.2.3, from 3.3.0 to 3.3.1 #2544 #2607
• Fix break change in partition read. For example, to_days is not supported #2552
• Fix exception will be thrown when we insert into a table partitioned by year and the first col of the table is not date type #2554
• Fix the bug that cluster index can not be used if the clustered index is not the Integer type #2560
• Fix the bug that CBO may not choose min cost between TiKV table scan, TiKV index scan, and TiFlash scan #2563
• Fix the bug that statistics are not collected which may affect the choice of the plan #2578
• Fix the bug that partition pruning fails with the uppercase column #2593

Documents

• Optimize user guide and dev guide #2533

TiSpark 3.1.2

New Features

• Support write into partition table with new collations #2570
• Support read TiFlash load balancing with Round-Robin strategy #2576
• Support partition pruning with to_days function when read from TiKV/TiFlash #2594 #2600

Fixes

• Fix CBO to let TiSpark choose the min cost between TiKV table scan, TiKV index scan and TiFlash scan correctly #2568
• Fix the region not find error when reading from TiKV #2575
• Fix the bug that statistics is not collected which may affect the choice of plans #2589
• Compatible with TiDB v6.5.0 #2602

TiSpark 2.4.4

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460.

Fixes

• Fix the bug that single column condition is in the incorrect if branch #2393.
• Fix region may be missed with too many tables #2442.
• Fix the bug that count can not be pushed down #2468 #2483

TiSpark 3.1.1

Fixes

• Fix fail to read from some partition table #2553
• Bump spark version to avoid vulnerabilities #2555
• Fix error will be thrown when date col is not the first col in hash partition table #2556

TiSpark 3.1.0

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2457

New Features

• Add authorization check for datasource api #2366
• Make TiSpark's Explain clearer and easier to read #2439
• Support host mapping in TiSpark #2436
• Support bypass-TiDB write into partition table #2451
• Support insert sql #2471
• Support Spark 3.3 #2492
• Only do auth check for tables in TiDB #2489
• Support new Collation #2524

Fixes

• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw table not found exceptions #2433
• Fix the bug that count/avg can not push down #2445
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict. And the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2455
• Fix the bug that exception would through when the size of pdAddresse is > 1 #2473
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2476
• Upgraded Spark 3.1 support version from 3.0.2 to 3.0.3, upgraded Spark 3.1 support version from 3.1.1 to 3.1.3, upgraded Spark 3.2 support version from 3.2.1 to 3.2.2 #2486
• Fix the bug that exception will be throw when date col is not the first col ref #2538

TiSpark 3.0.2

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2460

Fixes

• Fix the bug that single column condition is in the incorrect if branch #2395
• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2440
• Fix the bug that count/avg can not push down #2470
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2515
• Fix exception would through when the size of pdAddresse is > 1 #2478
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2485
• Upgraded Spark3.1 support version from 3.0.2 to 3.0.3, Upgraded Spark3.1 support version from 3.1.1 to 3.1.3, Upgraded Spark3.2 support version from 3.2.1 to 3.2.2 #2488
• Only do auth check for tables in TiDB #2500
• Changed profile #2518

TiSpark 2.5.2

Compatibility Changes

• We will not provide the mysql-connector-java dependency because of the limit of the GPL license #2461

Fixes

• Fix the bug that single column condition is incorrect if branch #2394
• Fix when TiDB has more than 10,000 tables in one Database, TiSpark may throw Table not found exceptions #2441
• Fix the bug that count/avg can not push down #2469
• Fix the bug that when the primary key is not integer type, the two rows with null unique index will conflict and the bug that when the unique index conflicts, the conflicting unique index column cannot be deleted correctly #2516
• Fix exception would through when the size of pdAddresse is > 1 #2477
• Fix the bug that Count(bit) should not be pushed down before TiKV 6.0.0 #2484
• Upgraded Spark3.1 support version from 3.0.2 to 3.0.3，Upgraded Spark3.1 support version from 3.1.1 to 3.1.3 #2487
• Only do auth check for tables in TiDB #2502
• Change spark profile #2517

TiSpark 3.0.1

Fixes

• Fix the bug that the single column condition is in the incorrect if branch #2395
• Fix the bug that the spark-shell is stuck when exiting #2402
• Fix the bug that the maven dependency of TiSpark 3.0.0 can not be imported #2401
• Shutdown recycler when closing ChannelFactory to avoid resource leak #2405

TiSpark 3.0.0

Compatibility Changes

• TiSpark without catalog plugin is no more supported. You must configure catalog configs and use tidb_catalog now #2252
• TiSpark's jar has a new naming rule like tispark-assembly-{$spark_version}_{$scala_version}-{$tispark_verison} #2370

New Feature

• Support DELETE statement #2276
• Support Spark 3.2 #2287
• Support telemetry to collect information #2316
• Support stale read to read historical versions of data #2322
• Support TLS with reload capability #2306 #2349 #2365 #2377

Fixes

• Fix the wrong result of _tidb_rowid when set spark.tispark.show_rowid=true #2270
• Fix sum not push down bug #2314
• Fix limit not push down bug #2329
• Avoid NoSuchElementException when setting catalog #2220
• Avoid ClassCastException when cluster index with type Timestamp and Date #2319
• Improve retry logic in write so that it will not retry in some scenarios which needn't retry #2279
• Delete unused configuration spark.tispark.statistics.auto_load #2300
• Upgrade jackson-databind from 2.9.10.8 to 2.12.6.1 #2285
• Upgrade guava from 26.0-android to 29.0-android #2340
• Upgrade mysql-connector-java from 5.1.44 to 5.1.49 #2367

Documents

• Update communication channels #2228
• Add limitation: new collations are not supported #2238