diff --git a/Pasted image 20250606151723.png b/Pasted image 20250606151723.png new file mode 100644 index 0000000..dfedb62 Binary files /dev/null and b/Pasted image 20250606151723.png differ diff --git a/Pasted image 20250606151809.png b/Pasted image 20250606151809.png new file mode 100644 index 0000000..58ed397 Binary files /dev/null and b/Pasted image 20250606151809.png differ diff --git a/Pasted image 20250609141740.png b/Pasted image 20250609141740.png new file mode 100644 index 0000000..233e841 Binary files /dev/null and b/Pasted image 20250609141740.png differ diff --git a/Pasted image 20250609141858.png b/Pasted image 20250609141858.png new file mode 100644 index 0000000..47e9d2b Binary files /dev/null and b/Pasted image 20250609141858.png differ diff --git a/Pasted image 20250609142057.png b/Pasted image 20250609142057.png new file mode 100644 index 0000000..2018441 Binary files /dev/null and b/Pasted image 20250609142057.png differ diff --git a/Pasted image 20250609142309.png b/Pasted image 20250609142309.png new file mode 100644 index 0000000..3364ad7 Binary files /dev/null and b/Pasted image 20250609142309.png differ diff --git a/work常用/中间件/hertzbeat/1.开发环境搭建(1.7.x版本).md b/公司环境/hertzbeat/1.开发环境搭建(1.7.x版本).md similarity index 100% rename from work常用/中间件/hertzbeat/1.开发环境搭建(1.7.x版本).md rename to 公司环境/hertzbeat/1.开发环境搭建(1.7.x版本).md diff --git a/work常用/中间件/hertzbeat/2.单点改造.md b/公司环境/hertzbeat/2.单点改造.md similarity index 100% rename from work常用/中间件/hertzbeat/2.单点改造.md rename to 公司环境/hertzbeat/2.单点改造.md diff --git a/work常用/中间件/hertzbeat/3.环境部署.md b/公司环境/hertzbeat/3.环境部署.md similarity index 100% rename from work常用/中间件/hertzbeat/3.环境部署.md rename to 公司环境/hertzbeat/3.环境部署.md diff --git a/work常用/中间件/seatunnel/Kafka-TO-hive.md b/公司环境/seatunnel/Kafka-TO-hive.md similarity index 100% rename from work常用/中间件/seatunnel/Kafka-TO-hive.md rename to 公司环境/seatunnel/Kafka-TO-hive.md diff --git a/work常用/中间件/seatunnel/Kafka-To-Doris.md b/公司环境/seatunnel/Kafka-To-Doris.md similarity index 100% rename from work常用/中间件/seatunnel/Kafka-To-Doris.md rename to 公司环境/seatunnel/Kafka-To-Doris.md diff --git a/work常用/中间件/seatunnel/MySql-CDC.md b/公司环境/seatunnel/MySql-CDC.md similarity index 100% rename from work常用/中间件/seatunnel/MySql-CDC.md rename to 公司环境/seatunnel/MySql-CDC.md diff --git a/work常用/中间件/seatunnel/Postgresql-CDC.md b/公司环境/seatunnel/Postgresql-CDC.md similarity index 100% rename from work常用/中间件/seatunnel/Postgresql-CDC.md rename to 公司环境/seatunnel/Postgresql-CDC.md diff --git a/公司环境/seatunnel/Seatunnel升级步骤.md b/公司环境/seatunnel/Seatunnel升级步骤.md new file mode 100644 index 0000000..c665f7a --- /dev/null +++ b/公司环境/seatunnel/Seatunnel升级步骤.md @@ -0,0 +1,46 @@ +## 一、源码编译 +1. 从github 下载源码包并用idea 打开,注意:这里不能使用提供的github release,因为打出来的包版本有问题,必须用源码包。(另外官方提供的安装包不提供connector,需要单独下载) +![[Pasted image 20250609141740.png]] + +![[Pasted image 20250609141858.png]] + +2. 检查 jdk 版本,查看最外层pom.xml 文件java.version 标签 +![[Pasted image 20250609142057.png]] +3. 设置jdk版本与pom文件要求版本相同 +4. 在代码目录下执行命令,过程中如果有保存自行解决,大概率是环境问题 +```shell +# 安装依赖 +./mvnw install -Dmaven.test.skip +# 编译打包 +mvn clean package -pl seatunnel-dist -am -Dmaven.test.skip=true +``` +5. 查看打包 +![[Pasted image 20250609142309.png]] + +## 二、安装部署 +因为使用源码编译,connector也在其中,不用再次从网上下载。 +对于升级来说只需要复制lib和config目录即可。 +将tar包上传至服务器目录,例如 /opt/datasophon +下面给出参考命令,自行替换路径及版本号 +1. 替换lib +```shell +tar -zxvf apache-seatunnel-2.3.11-bin.tar.gz && cd /opt/datasophon/seatunnel/lib && zip -r lib.zip ./* && mv lib.zip /opt/datasophon/apache-seatunnel-2.3.11 && unzip -o /opt/datasophon/apache-seatunnel-2.3.11/lib.zip -d /opt/datasophon/apache-seatunnel-2.3.11/lib && rm -rf /opt/datasophon/apache-seatunnel-2.3.11/lib.zip && cd /opt/datasophon/apache-seatunnel-2.3.11 +``` +2. 替换config +```shell + cd /opt/datasophon/seatunnel/config && zip -r config.zip ./* && mv config.zip /opt/datasophon/apache-seatunnel-2.3.11 && unzip -o /opt/datasophon/apache-seatunnel-2.3.11/config.zip -d /opt/datasophon/apache-seatunnel-2.3.11/config && rm -rf /opt/datasophon/apache-seatunnel-2.3.11/config.zip && cd /opt/datasophon/apache-seatunnel-2.3.11 +``` +替换完成后记得检查配置和lib是否已存在 + +启动节点 +1. 停止旧服务,自行替换目录及版本号 +```shell +cd /opt/datasophon/seatunnel/bin && ./stop-seatunnel-cluster.sh && cd /opt/datasophon && mv seatunnel-2.3.8 seatunnel-2.3.8_bak && cd /opt/datasophon && mv apache-seatunnel-2.3.11 seatunnel-2.3.8 +``` +2. 启动新服务 +```shell +# 启动master +cd /opt/datasophon/seatunnel/bin && ./seatunnel-cluster.sh -d -r master +# 启动worker +cd /opt/datasophon/seatunnel/bin && ./seatunnel-cluster.sh -d -r worker +``` \ No newline at end of file diff --git a/公司环境/seatunnel/gbase8s-To-Doris.md b/公司环境/seatunnel/gbase8s-To-Doris.md new file mode 100644 index 0000000..a72ddc5 --- /dev/null +++ b/公司环境/seatunnel/gbase8s-To-Doris.md @@ -0,0 +1,75 @@ +seatunnel 配置 +```text +env { + # 任务名称 + job.name = "gbase_source测试" + # 任务并行度 + parallelism = "1" + # 任务模式 + job.mode = "BATCH" + # 检查点时间 + checkpoint.interval = "10000" +} +source { + Jdbc { + # 数据去向表 + plugin_output = "source_result" + # jdbc连接地址 + url = "jdbc:gbasedbt-sqli://172.100.40.173:9088/hk_szbj:GBASEDBTSERVER=gbaseserver" + # jdbc驱动 + driver = "com.gbasedbt.jdbc.Driver" + # 用户名 + user = "gbasedbt" + # 密码 + password = "Gbasedbt123" + # 表名 + query = "select id,cjrzh,cjsj,xgrzh,xgsj,lxdm,zddm,zdmc,bmid,bz,xh,sybj from acc_d_sjzd" + } +} +transform { + Sql { + # 数据来源表 + source_table_name = "source_result" + # 数据去向表 + result_table_name = "transform_result" + # 字段映射sql + query = "select id as ID,cjrzh as CJRZH,cjsj as CJSJ,xgrzh as XGRZH,xgsj as XGSJ,lxdm as LXDM,zddm as ZDDM,zdmc as ZDMC,bmid as BMID,bz as BZ,xh as XH,sybj as SYBJ from source_result" + } +} +sink { + Doris { + # 数据来源表 + plugin_input = "transform_result" + # be连接地址 + fenodes = "172.100.40.152:8030" + # 用户名 + username = "root" + # 密码 + password = "maxvision@123" + # 数据库地址 + database = "demo_jh_test" + # 表名 + table = "ods_acc_d_sjzd" + # 输出任务前缀 + sink.label-prefix = "993ec066075e48b9b1dd2be857bc14127b" + # 是否开启两端提交 + sink.enable-2pc = "false" + # 是否开启删除策略 + sink.enable-delete = "true" + data_save_mode = "DROP_DATA" + doris.config { + # 数据格式化类型 + format = "json" + # 读取每行JSON + read_json_by_line = "true" + } + } +} + +``` + +jdbc驱动包,需要放在$SEATUNNEL_HOME/lib 目录 + +![[Pasted image 20250606151723.png]] + +![[Pasted image 20250606151809.png]] \ No newline at end of file