{"id":1680,"date":"2023-03-25T22:34:15","date_gmt":"2023-03-25T14:34:15","guid":{"rendered":"https:\/\/www.appblog.cn\/?p=1680"},"modified":"2023-04-23T21:48:44","modified_gmt":"2023-04-23T13:48:44","slug":"alibaba-canal-client-synchronize-adapter-clientadapter","status":"publish","type":"post","link":"https:\/\/www.appblog.cn\/index.php\/2023\/03\/25\/alibaba-canal-client-synchronize-adapter-clientadapter\/","title":{"rendered":"\u963f\u91cc\u5df4\u5df4Canal Client\u540c\u6b65\u9002\u914d\u5668ClientAdapter"},"content":{"rendered":"<p>\u53c2\u8003\uff1a<a target=\"_blank\" rel=\"noopener\" href=\"https:\/\/github.com\/alibaba\/canal\/wiki\/ClientAdapter\">https:\/\/github.com\/alibaba\/canal\/wiki\/ClientAdapter<\/a><\/p>\n<h2>\u57fa\u672c\u8bf4\u660e<\/h2>\n<p>canal 1.1.1\u7248\u672c\u4e4b\u540e\uff0c\u589e\u52a0\u5ba2\u6237\u7aef\u6570\u636e\u843d\u5730\u7684\u9002\u914d\u53ca\u542f\u52a8\u529f\u80fd\uff0c\u76ee\u524d\u652f\u6301\u529f\u80fd\uff1a<\/p>\n<p><!-- more --><\/p>\n<ul>\n<li>\u5ba2\u6237\u7aef\u542f\u52a8\u5668<\/li>\n<li>\u540c\u6b65\u7ba1\u7406REST\u63a5\u53e3<\/li>\n<li>\u65e5\u5fd7\u9002\u914d\u5668, \u4f5c\u4e3aDEMO<\/li>\n<li>\u5173\u7cfb\u578b\u6570\u636e\u5e93\u7684\u6570\u636e\u540c\u6b65(\u8868\u5bf9\u8868\u540c\u6b65)\uff0cETL\u529f\u80fd<\/li>\n<li>HBase\u7684\u6570\u636e\u540c\u6b65(\u8868\u5bf9\u8868\u540c\u6b65)\uff0cETL\u529f\u80fd<\/li>\n<li>(\u540e\u7eed\u652f\u6301) ElasticSearch\u591a\u8868\u6570\u636e\u540c\u6b65\uff0cETL\u529f\u80fd<\/li>\n<\/ul>\n<h2>\u73af\u5883\u7248\u672c<\/h2>\n<ul>\n<li>\u64cd\u4f5c\u7cfb\u7edf\uff1a\u65e0\u8981\u6c42<\/li>\n<li>Java\u7248\u672c: jdk1.8 \u4ee5\u4e0a<\/li>\n<li>canal \u7248\u672c: \u8bf7\u4e0b\u8f7d\u6700\u65b0\u7684\u5b89\u88c5\u5305\uff0c\u672c\u6587\u4ee5\u5f53\u524dv1.1.1 \u7684<code>canal.deployer-1.1.1.tar.gz<\/code>\u4e3a\u4f8b<\/li>\n<li>MySQL\u7248\u672c\uff1a5.7.18<\/li>\n<li>HBase\u7248\u672c: Apache HBase 1.1.2\uff0c\u82e5\u548c\u670d\u52a1\u7aef\u7248\u672c\u4e0d\u4e00\u81f4\u53ef\u81ea\u884c\u66ff\u6362\u5ba2\u6237\u7aefHBase\u4f9d\u8d56<\/li>\n<\/ul>\n<h2>\u9002\u914d\u5668\u6574\u4f53\u7ed3\u6784<\/h2>\n<p><code>client-adapter<\/code>\u5206\u4e3a\u9002\u914d\u5668\u548c\u542f\u52a8\u5668\u4e24\u90e8\u5206\uff0c\u9002\u914d\u5668\u4e3a\u591a\u4e2a<code>fat jar<\/code>\uff0c\u6bcf\u4e2a\u9002\u914d\u5668\u4f1a\u5c06\u81ea\u5df1\u6240\u9700\u7684\u4f9d\u8d56\u6253\u6210\u4e00\u4e2a\u5305\uff0c\u4ee5SPI\u7684\u65b9\u5f0f\u8ba9\u542f\u52a8\u5668\u52a8\u6001\u52a0\u8f7d\uff0c\u76ee\u524d\u6240\u6709\u652f\u6301\u7684\u9002\u914d\u5668\u90fd\u653e\u7f6e\u5728<code>plugin<\/code>\u76ee\u5f55\u4e0b<\/p>\n<p>\u542f\u52a8\u5668\u4e3a Spring Boot \u9879\u76ee\uff0c\u652f\u6301<code>canal-client<\/code>\u542f\u52a8\u7684\u540c\u65f6\u63d0\u4f9b\u76f8\u5173REST\u7ba1\u7406\u63a5\u53e3\uff0c\u8fd0\u884c\u76ee\u5f55\u7ed3\u6784\u4e3a\uff1a<\/p>\n<pre><code class=\"language-yml\">- bin\n    restart.sh\n    startup.bat\n    startup.sh\n    stop.sh\n- lib\n   ...\n- plugin \n    client-adapter.logger-1.1.1-jar-with-dependencies.jar\n    client-adapter.hbase-1.1.1-jar-with-dependencies.jar\n    ...\n- conf\n    application.yml\n    - hbase\n        mytest_person2.yml\n- logs<\/code><\/pre>\n<p>\u4ee5\u4e0a\u76ee\u5f55\u7ed3\u6784\u6700\u7ec8\u4f1a\u6253\u5305\u6210<code>canal-adapter-*.tar.gz<\/code>\u538b\u7f29\u5305<\/p>\n<h2>\u9002\u914d\u5668\u914d\u7f6e\u4ecb\u7ecd<\/h2>\n<h3>\u603b\u914d\u7f6e\u6587\u4ef6 application.yml<\/h3>\n<p><strong>adapter\u5b9a\u4e49\u914d\u7f6e\u90e8\u5206<\/strong><\/p>\n<pre><code class=\"language-yml\">canal.conf:\n  canalServerHost: 127.0.0.1:11111          # \u5bf9\u5e94\u5355\u673a\u6a21\u5f0f\u4e0b\u7684canal server\u7684ip:port\n  zookeeperHosts: slave1:2181               # \u5bf9\u5e94\u96c6\u7fa4\u6a21\u5f0f\u4e0b\u7684zk\u5730\u5740, \u5982\u679c\u914d\u7f6e\u4e86canalServerHost, \u5219\u4ee5canalServerHost\u4e3a\u51c6\n  mqServers: slave1:6667 #or rocketmq       # kafka\u6216rocketMQ\u5730\u5740, \u4e0ecanalServerHost\u4e0d\u80fd\u5e76\u5b58\n  flatMessage: true                         # \u6241\u5e73message\u5f00\u5173, \u662f\u5426\u4ee5json\u5b57\u7b26\u4e32\u5f62\u5f0f\u6295\u9012\u6570\u636e, \u4ec5\u5728kafka\/rocketMQ\u6a21\u5f0f\u4e0b\u6709\u6548\n  batchSize: 50                             # \u6bcf\u6b21\u83b7\u53d6\u6570\u636e\u7684\u6279\u5927\u5c0f, \u5355\u4f4d\u4e3aK\n  syncBatchSize: 1000                       # \u6bcf\u6b21\u540c\u6b65\u7684\u6279\u6570\u91cf\n  retries: 0                                # \u91cd\u8bd5\u6b21\u6570, -1\u4e3a\u65e0\u9650\u91cd\u8bd5\n  timeout:                                  # \u540c\u6b65\u8d85\u65f6\u65f6\u95f4, \u5355\u4f4d\u6beb\u79d2\n  mode: tcp # kafka rocketMQ                # canal client\u7684\u6a21\u5f0f: tcp kafka rocketMQ\n  srcDataSources:                           # \u6e90\u6570\u636e\u5e93\n    defaultDS:                              # \u81ea\u5b9a\u4e49\u540d\u79f0\n      url: jdbc:mysql:\/\/127.0.0.1:3306\/mytest?useUnicode=true   # jdbc url \n      username: root                                            # jdbc \u8d26\u53f7\n      password: 121212                                          # jdbc \u5bc6\u7801\n  canalAdapters:                            # \u9002\u914d\u5668\u5217\u8868\n  - instance: example                       # canal \u5b9e\u4f8b\u540d\u6216\u8005 MQ topic \u540d\n    groups:                                 # \u5206\u7ec4\u5217\u8868\n    - groupId: g1                           # \u5206\u7ec4id, \u5982\u679c\u662fMQ\u6a21\u5f0f\u5c06\u7528\u5230\u8be5\u503c\n      outerAdapters:                        # \u5206\u7ec4\u5185\u9002\u914d\u5668\u5217\u8868\n      - name: logger                        # \u65e5\u5fd7\u6253\u5370\u9002\u914d\u5668\n......<\/code><\/pre>\n<p>\u8bf4\u660e\uff1a<\/p>\n<ul>\n<li>\u4e00\u4efd\u6570\u636e\u53ef\u4ee5\u88ab\u591a\u4e2agroup\u540c\u65f6\u6d88\u8d39\uff0c\u591a\u4e2agroup\u4e4b\u95f4\u4f1a\u662f\u4e00\u4e2a\u5e76\u884c\u6267\u884c\uff0c\u4e00\u4e2agroup\u5185\u90e8\u662f\u4e00\u4e2a\u4e32\u884c\u6267\u884c\u591a\u4e2a<code>outerAdapters<\/code>\uff0c\u6bd4\u5982\u4f8b\u5b50\u4e2d<code>logger<\/code>\u548c<code>hbase<\/code><\/li>\n<li>\u76ee\u524d<code>client-adapter<\/code>\u6570\u636e\u8ba2\u9605\u7684\u65b9\u5f0f\u652f\u6301\u4e24\u79cd\uff0c\u76f4\u8fde<code>canal server<\/code>\u6216\u8005\u8ba2\u9605<code>kafka\/RocketMQ<\/code>\u7684\u6d88\u606f<\/li>\n<\/ul>\n<h3>\u4f7f\u7528\u8fdc\u7a0b\u914d\u7f6e(Mysql)<\/h3>\n<p>\u53ef\u4ee5\u4f7f\u7528\u8fdc\u7a0b\u914d\u7f6e\u4e2d\u5fc3(Mysql\uff0c\u53ef\u6269\u5c55)\u4f5c\u4e3a\u7edf\u4e00\u914d\u7f6e\u7ba1\u7406<\/p>\n<h4>\u521b\u5efamysql schema<\/h4>\n<pre><code class=\"language-sql\">CREATE SCHEMA `canal_manager` DEFAULT CHARACTER SET utf8mb4;<\/code><\/pre>\n<h4>\u521d\u59cb\u5316\u6570\u636e<\/h4>\n<p>\u4f7f\u7528<code>manager_ddl.sql<\/code>\u811a\u672c\u5efa\u8868\u5e76\u521d\u59cb\u5316Demo\u6570\u636e\uff0c\u5176\u4e2d<code>canal_config<\/code>\u8868<code>id=2<\/code>\u7684\u6570\u636e\u5bf9\u5e94adapter\u4e0b\u7684<code>application.yml<\/code>\u6587\u4ef6\uff0c<code>canal_adapter_config<\/code>\u8868\u5bf9\u5e94\u6bcf\u4e2aadapter\u7684\u5b50\u914d\u7f6e\u6587\u4ef6<\/p>\n<h4>\u4fee\u6539bootstrap.yml\u914d\u7f6e<\/h4>\n<pre><code class=\"language-yml\">canal:\n  manager:\n    jdbc:\n      url: jdbc:mysql:\/\/127.0.0.1:3306\/canal_manager?useUnicode=true&amp;characterEncoding=UTF-8\n      username: root\n      password: 121212<\/code><\/pre>\n<p>\u53ef\u4ee5\u5c06\u672c\u5730<code>application.yml<\/code>\u6587\u4ef6\u548c\u5176\u4ed6\u5b50\u914d\u7f6e\u6587\u4ef6\u5220\u9664\u6216\u6e05\u7a7a\uff0c\u542f\u52a8\u5de5\u7a0b\u5c06\u81ea\u52a8\u4ece\u8fdc\u7a0b\u52a0\u8f7d\u914d\u7f6e<\/p>\n<p>\u4fee\u6539MySQL\u4e2d\u7684\u914d\u7f6e\u4fe1\u606f\u540e\u4f1a\u81ea\u52a8\u5237\u65b0\u5230\u672c\u5730\u52a8\u6001\u52a0\u8f7d\u76f8\u5e94\u7684\u5b9e\u4f8b\u6216\u8005\u5e94\u7528<\/p>\n<h2>\u9002\u914d\u5668\u542f\u52a8<\/h2>\n<h3>\u542f\u52a8canal-adapter\u793a\u4f8b<\/h3>\n<h4>\u542f\u52a8canal server (\u5355\u673a\u6a21\u5f0f)<\/h4>\n<h4>\u4fee\u6539conf\/application.yml<\/h4>\n<pre><code class=\"language-yml\">server:\n  port: 8081\nlogging:\n  level:\n    com.alibaba.otter.canal.client.adapter.hbase: DEBUG\nspring:\n  jackson:\n    date-format: yyyy-MM-dd HH:mm:ss\n    time-zone: GMT+8\n    default-property-inclusion: non_null\n\ncanal.conf:\n  canalServerHost: 127.0.0.1:11111\n  batchSize: 500\n  syncBatchSize: 1000\n  retries: 0\n  timeout:\n  mode: tcp\n  canalAdapters:\n  - instance: example\n    groups:\n    - groupId: g1\n      outerAdapters:\n      - name: logger<\/code><\/pre>\n<h4>\u542f\u52a8<\/h4>\n<pre><code class=\"language-bash\">bin\/startup.sh<\/code><\/pre>\n<h3>adapter\u7ba1\u7406REST\u63a5\u53e3<\/h3>\n<h4>\u67e5\u8be2\u6240\u6709\u8ba2\u9605\u540c\u6b65\u7684<code>canal instance<\/code>\u6216<code>MQ topic<\/code><\/h4>\n<pre><code class=\"language-bash\">curl http:\/\/127.0.0.1:8081\/destinations<\/code><\/pre>\n<h4>\u6570\u636e\u540c\u6b65\u5f00\u5173<\/h4>\n<pre><code class=\"language-bash\">curl http:\/\/127.0.0.1:8081\/syncSwitch\/example\/off -X PUT<\/code><\/pre>\n<p>\u9488\u5bf9 example \u8fd9\u4e2a<code>canal instance\/MQ topic<\/code>\u8fdb\u884c\u5f00\u5173\u64cd\u4f5c\u3002<code>off<\/code>\u4ee3\u8868\u5173\u95ed\uff0c<code>instance\/topic<\/code>\u4e0b\u7684\u540c\u6b65\u5c06\u963b\u585e\u6216\u8005\u65ad\u5f00\u8fde\u63a5\u4e0d\u518d\u63a5\u6536\u6570\u636e\uff0c<code>on<\/code>\u4ee3\u8868\u5f00\u542f<\/p>\n<blockquote>\n<p>\u6ce8: \u5982\u679c\u5728\u914d\u7f6e\u6587\u4ef6\u4e2d\u914d\u7f6e\u4e86<code>zookeeperHosts<\/code>\u9879\uff0c\u5219\u4f1a\u4f7f\u7528\u5206\u5e03\u5f0f\u9501\u6765\u63a7\u5236HA\u4e2d\u7684\u6570\u636e\u540c\u6b65\u5f00\u5173\uff0c\u5982\u679c\u662f\u5355\u673a\u6a21\u5f0f\u5219\u4f7f\u7528\u672c\u5730\u9501\u6765\u63a7\u5236\u5f00\u5173<\/p>\n<\/blockquote>\n<h4>\u6570\u636e\u540c\u6b65\u5f00\u5173\u72b6\u6001<\/h4>\n<pre><code class=\"language-bash\">curl http:\/\/127.0.0.1:8081\/syncSwitch\/example<\/code><\/pre>\n<p>\u67e5\u770b\u6307\u5b9a<code>canal instance\/MQ topic<\/code>\u7684\u6570\u636e\u540c\u6b65\u5f00\u5173\u72b6\u6001<\/p>\n<h4>\u624b\u52a8ETL<\/h4>\n<pre><code class=\"language-bash\">curl http:\/\/127.0.0.1:8081\/etl\/hbase\/mytest_person2.yml -X POST -d &quot;params=2018-10-21 00:00:00&quot;<\/code><\/pre>\n<p>\u5bfc\u5165\u6570\u636e\u5230\u6307\u5b9a\u7c7b\u578b\u7684\u5e93\uff0c\u5982\u679cparams\u53c2\u6570\u4e3a\u7a7a\u5219\u5168\u8868\u5bfc\u5165\uff0c\u53c2\u6570\u5bf9\u5e94\u7684\u67e5\u8be2\u6761\u4ef6\u5728\u914d\u7f6e\u4e2d\u7684<code>etlCondition<\/code>\u6307\u5b9a<\/p>\n<h4>\u67e5\u770b\u76f8\u5173\u5e93\u603b\u6570\u636e<\/h4>\n<pre><code class=\"language-bash\">curl http:\/\/127.0.0.1:8081\/count\/hbase\/mytest_person2.yml<\/code><\/pre>\n<h2>\u9002\u914d\u5668\u5217\u8868<\/h2>\n<h3>logger\u9002\u914d\u5668<\/h3>\n<p>\u6700\u7b80\u5355\u7684\u5904\u7406\uff0c\u5c06\u6536\u5230\u7684\u53d8\u66f4\u4e8b\u4ef6\u901a\u8fc7\u65e5\u5fd7\u6253\u5370\u7684\u65b9\u5f0f\u8fdb\u884c\u8f93\u51fa\uff0c\u5982\u914d\u7f6e\u6240\u793a\uff0c\u53ea\u9700\u8981\u5b9a\u4e49<code>name: logger<\/code>\u5373\u53ef<\/p>\n<pre><code class=\"language-yml\">  outerAdapters:\n  - name: logger<\/code><\/pre>\n<h3>Hbase\u9002\u914d\u5668<\/h3>\n<p>\u540c\u6b65HBase\u914d\u7f6e\uff1a<a target=\"_blank\" rel=\"noopener\" href=\"https:\/\/github.com\/alibaba\/canal\/wiki\/Sync-HBase\" title=\"Sync-HBase\">Sync-HBase<\/a><\/p>\n<h3>RDB\u9002\u914d\u5668<\/h3>\n<p>\u540c\u6b65\u5173\u7cfb\u578b\u6570\u636e\u5e93\u914d\u7f6e\uff1a<a target=\"_blank\" rel=\"noopener\" href=\"https:\/\/github.com\/alibaba\/canal\/wiki\/Sync-RDB\" title=\"Sync-RDB\">Sync-RDB<\/a><\/p>\n<p>\u76ee\u524d\u5185\u7f6e\u652f\u6301\u7684\u6570\u636e\u5e93\u5217\u8868:<\/p>\n<ul>\n<li>MySQL<\/li>\n<li>Oracle<\/li>\n<li>PostgresSQL<\/li>\n<li>SQLServer<\/li>\n<\/ul>\n<p>\u4f7f\u7528\u4e86JDBC driver\uff0c\u7406\u8bba\u4e0a\u652f\u6301\u7edd\u5927\u90e8\u5206\u7684\u5173\u7cfb\u578b\u6570\u636e\u5e93<\/p>\n<h3>ES\u9002\u914d\u5668<\/h3>\n<p>\u540c\u6b65\u5173ES\u914d\u7f6e\uff1a<a target=\"_blank\" rel=\"noopener\" href=\"https:\/\/github.com\/alibaba\/canal\/wiki\/Sync-ES\" title=\"Sync-ES\">Sync-ES<\/a><\/p>\n<h3>MongoDB\u9002\u914d\u5668<\/h3>\n<h3>Redis\u9002\u914d\u5668<\/h3>\n","protected":false},"excerpt":{"rendered":"<p>\u53c2\u8003\uff1ahttps:\/\/github.com\/alibaba\/canal\/wiki\/ClientAdapter  [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[417],"tags":[],"class_list":["post-1680","post","type-post","status-publish","format-standard","hentry","category-canal"],"_links":{"self":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts\/1680","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/comments?post=1680"}],"version-history":[{"count":0,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/posts\/1680\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/media?parent=1680"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/categories?post=1680"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appblog.cn\/index.php\/wp-json\/wp\/v2\/tags?post=1680"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}