聊聊流式数据湖Paimon(五)

Ryan_OVO

发布于 2023-12-30 08:03:33

3810

发布于 2023-12-30 08:03:33

文章被收录于专栏：程序随笔

从Demo入手，了解Paimon/Flink项目搭建的全过程。记录下采坑之旅。

创建Flink项目

在IDEA中创建Flink项目，由于没有Flink的archetype，因此需要手动创建一下。参考：idea快速创建flink项目，至此Flink的项目框架就搭建起来了。注意：必须注释掉pom文件中的provided；否则运行时会报错： Error: A JNI error has occurred, please check your installation and try again

搭建Flink伪集群

在 Flink包地址中，选择对应的版本，下载文件

解压后，其文件内容，如下

在bin目录下，运行start-cluster.bat脚本即可。打开浏览器访问：localhost:8081，就可以查看Flink的webui

高版本的Flink中已经没有bat脚本，可参考 flink新版本无bat启动文件的解决办法

补充缺失的依赖

Flink的框架搭建好之后，参考新一代数据湖存储技术Apache Paimon入门Demo 写一个简单的Paimon程序。但在这个过程中，必须补充缺失的POM依赖。而这些依赖在编译时并不会报错，一旦运行，各种各样的抛错： java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration Unable to create catalog xxx Unsupported SQL query! executeSql() 如下是所有需要的pom依赖：

<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-streaming-java</artifactId>
  <version>${flink.version}</version>
  <!--			<scope>provided</scope>-->
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-clients</artifactId>
  <version>${flink.version}</version>
  <!--			<scope>provided</scope>-->
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-api-java-bridge</artifactId>
  <version>1.18.0</version>
</dependency>
<dependency>
  <groupId>org.apache.paimon</groupId>
  <artifactId>paimon-flink-1.18</artifactId>
  <version>0.6.0-incubating</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-planner-loader</artifactId>
  <version>${flink.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-table-runtime</artifactId>
  <version>1.18.0</version>
</dependency>
<dependency>
  <groupId>org.apache.flink</groupId>
  <artifactId>flink-connector-base</artifactId>
  <version>${flink.version}</version>
</dependency>

<!-- Add connector dependencies here. They must be in the default scope (compile). -->

<!-- Example:

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>3.0.0-1.17</version>
</dependency>
-->

<!-- Add logging framework, to produce console output when running in the IDE. -->
<!-- These dependencies are excluded from the application JAR by default. -->
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-slf4j-impl</artifactId>
  <version>${log4j.version}</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-api</artifactId>
  <version>${log4j.version}</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-core</artifactId>
  <version>${log4j.version}</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>3.2.3</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-hdfs-client</artifactId>
  <version>3.2.3</version>
</dependency>

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2023-12-29，如有侵权请联系 cloudcommunity@tencent.com 删除

flink