使用flink-cdc从mysql和postgresql输出到es过程中遇到的一些问题的总结

[Roc893158890]

测试环境的docker-compose.yml

version: '2.2'
services:
  jobmanager:
    image: flink:1.13
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager        

  taskmanager:
    image: flink:1.13
    depends_on:
      - jobmanager
    command: taskmanager
    scale: 1
    environment:
      - |
        FLINK_PROPERTIES=
        jobmanager.rpc.address: jobmanager
        taskmanager.numberOfTaskSlots: 2   

  mysql:
    image: debezium/example-mysql:1.1
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=123456
      - MYSQL_USER=mysqluser
      - MYSQL_PASSWORD=mysqlpw
      
   postgres:
    image: debezium/example-postgres:1.1
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
      
  elasticsearch:
    image: elastic/elasticsearch:7.6.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536

  kibana:
    image: elastic/kibana:7.6.0
    ports:
      - "5601:5601"

使用到的jar

flink-sql-connector-postgres-cdc-2.1.1.jar
flink-sql-connector-mysql-cdc-2.1.1.jar
flink-sql-connector-elasticsearch7_2.12-1.13.6.jar

要注意的问题有

1. 几个cdc文件如果用2.1.1的都用2.1.1，不能混着用。
2. cdc2.1目前只支持flink1.13，用1.14的话可能会出现各种毛病。
3. 类继承顺序问题，在运行flink的地方（如果是集群的话包括及群里的jobmanager节点和taskmanager节点都要改）修改flink-conf.yaml文件，增加一句classloader.resolve-order: parent-first
4. 有2种jar包，一种叫flink-sql-connector-XXX.jar 一种叫flink-connector-XXX.jar 这俩不是一个东西，注意别用错了。有知道的大佬请教一下这两种包有什么差别。是一个给sql-client用的吗？