FlinkCDC 有没有类似Maxwell bootstrap全量同步表的功能 #1102
Unanswered
guojiel-mega
asked this question in
Q&A
Replies: 3 comments
-
|
sure, please refer https://ververica.github.io/flink-cdc-connectors/release-2.2/content/connectors/mysql-cdc.html |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
请问 maxwell可能丢失数据的问题你们是怎么解决的呢？期待回复 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
如题，最近在搭建离线数仓，使用Maxwell接入正式服的MySQL进行业务表的增量同步，由于该MySQL是阿里云做了一层代理的，host是192.168.x.x，端口号改为了32306，结果maxwell读取binlog正常，但是使用bootstrap时会报错，报错信息如下：
`[root@bigdata101 scripts]# ./mysql_to_kafka_inc_init.sh auth_user
connecting to jdbc:mysql://192.168.2.170:32306/maxwell?allowPublicKeyRetrieval=true&connectTimeout=5000&zeroDateTimeBehavior=convertToNull
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@fd9e3cd -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3789aa0c -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.8.0_212]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[?:1.8.0_212]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71125 [WARN] BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@2a1e8035 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception:
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47) ~[mysql-connector-java-5.1.27-bin.jar:?]
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_212]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_212]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44) ~[c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908) [c3p0-0.9.5.5.jar:0.9.5.5]
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696) [mchange-commons-java-0.2.19.jar:0.2.19]
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_212]
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249) ~[mysql-connector-java-5.1.27-bin.jar:?]
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308) ~[mysql-connector-java-5.1.27-bin.jar:?]
... 20 more
71151 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71152 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71152 [WARN] BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@7e07db1f is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.
71156 [ERROR] MaxwellBootstrapUtility: failed to connect to mysql server @ jdbc:mysql://192.168.2.170:32306/maxwell?allowPublicKeyRetrieval=true&connectTimeout=5000&zeroDateTimeBehavior=convertToNull
71156 [ERROR] MaxwellBootstrapUtility: Connections could not be acquired from the underlying database!
java.sql.SQLException: Connections could not be acquired from the underlying database!
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:692)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at com.zendesk.maxwell.util.C3P0ConnectionPool.getConnection(C3P0ConnectionPool.java:18)
at com.zendesk.maxwell.bootstrap.MaxwellBootstrapUtility.run(MaxwellBootstrapUtility.java:40)
at com.zendesk.maxwell.bootstrap.MaxwellBootstrapUtility.main(MaxwellBootstrapUtility.java:253)
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1507)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 4 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:358)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311)
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1176)
at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1163)
at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1908)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Caused by: java.net.UnknownHostException: null
at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:249)
at com.mysql.jdbc.MysqlIO.(MysqlIO.java:308)
... 20 more
`
不知如何解决，想看看FlinkCDC有没有比较好的解决方法，感谢提供思路的大佬们
（使用maxwell连接本地搭建的MySQL时bootstrap和获取binlog一切正常）
Beta Was this translation helpful? Give feedback.
All reactions