在windows上编译Inlong所遇到的问题

冬天里的懒猫

发布于 2022-05-10 21:27:59

2.8K0

发布于 2022-05-10 21:27:59

1.Hadoop相关的问题

在windows中，由于Inlong的很多单元测试用例都需要用到hadoop的相关组件进行测试。可以采用winunits配合测试，winutils是windows上专门配合hadoop和yarn测试的工具。在github上下载该工具：

git clone https://github.com/steveloughran/winutils.git

之后选择一个版本，配置环境变量：

HADOOP_HOME    D:\workspace\github-work\winutils\hadoop-3.0.0

并将该目录中，hadoop.dll 拷贝到C:\Windows\System32 目录。否则会导致出现.FileNotFoundException: HADOOP_HOME错误：

[INFO] -------------------------------------------------------
[INFO] Running org.apache.inlong.sort.flink.clickhouse.ClickHouseRowConverterTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 s - in org.apache.inlong.sort.flink.clickhouse.ClickHouseRowConverterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest
log4j:WARN No appenders could be found for logger (org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.381 s - in org.apache.inlong.sort.flink.hive.formats.orc.OrcBulkWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.032 s <<< FAILURE! - in org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest
[ERROR] testWriteFile(org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest)  Time elapsed: 0.028 s  <<< ERROR!
java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
        at org.apache.inlong.sort.flink.hive.formats.orc.RowVectorizerTest.testWriteFile(RowVectorizerTest.java:111)
Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.

[INFO] Running org.apache.inlong.sort.flink.hive.formats.parquet.ParquetBulkWriterTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.771 s - in org.apache.inlong.sort.flink.hive.formats.parquet.ParquetBulkWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.formats.TextRowWriterTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.apache.inlong.sort.flink.hive.formats.TextRowWriterTest
[INFO] Running org.apache.inlong.sort.flink.hive.HiveSinkHelperTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.inlong.sort.flink.hive.HiveSinkHelperTest
[INFO] Running org.apache.inlong.sort.flink.hive.partition.JdbcHivePartitionCommitPolicyTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.inlong.sort.flink.hive.partition.JdbcHivePartitionCommitPolicyTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   RowVectorizerTest.testWriteFile:111 ? Runtime java.io.FileNotFoundException: j...
[INFO]
[ERROR] Tests run: 23, Failures: 0, Errors: 1, Skipped: 0

如果没有将hadoop.dll 拷贝到C:\Windows\System32 目录，则会出现如下错误：

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Formatting using clusterid: testClusterID
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.615 s <<< FAILURE! - in org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase
[ERROR] test(org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase)  Time elapsed: 2.322 s  <<< ERROR!
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
        at org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase.initializeHdfs(HiveSinkWithoutPartitionTestCase.java:144)
        at org.apache.inlong.sort.singletenant.flink.connectors.hive.HiveSinkWithoutPartitionTestCase.setUp(HiveSinkWithoutPartitionTestCase.java:85)

2.nodejs 相关的问题

Inlong的dashboard需要用到nodejs相关的内容。安装nodejs环境： 1.下载nodejs 下载64位的zip版本 node-v16.15.0-win-x64.zip，并解压到D:\Program Files\nodejs目录中。在D:\ProgramData\nodejs 中新增两个目录：node_global与 node_cache 配置环境变量：

NODE_HOME D:\Program Files\nodejs\node-v16.15.0-win-x64

NODE_PATH D:\ProgramData\nodejs\node_global\node_modules

PATH 中新增： %NODE_HOME% 和 D:\ProgramData\nodejs\node_global

打开cmd: 可以识别到node 和npm。执行如下命令：

npm config set prefix "D:\ProgramData\nodejs\node_global"
npm config set cache "D:\ProgramData\nodejs\node_cache"

安装yarn并设置镜像：

npm install yarn -g
npm install -g cnpm --registry=https://registry.npm.taobao.org

在安装完npm之后，再执行：

mvn clean install -Dmaven.javadoc.skip=true

仍然出现如下错误：

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.12.1:npm (npm run build) on project inlong-dashboard: Failed to run task: 'npm run build' failed. org.apache.commons.exec.ExecuteExcepti
on: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :inlong-dashboard

执行：

$ npm run build
npm ERR! code ENOENT
npm ERR! syscall open
npm ERR! path D:\workspace\github-work\incubator-inlong/package.json
npm ERR! errno -4058
npm ERR! enoent ENOENT: no such file or directory, open 'D:\workspace\github-work\incubator-inlong\package.json'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     D:\ProgramData\nodejs\node_cache\_logs\2022-04-30T05_39_14_204Z-debug-0.log

上述错误说明，没有找到package.json 切换到inlong-dashboard

haibo.duan@LAPTOP-M5HE61J0 MINGW64 /d/workspace/github-work/incubator-inlong/inlong-dashboard (master)
$ npm run build

> inlong@0.1.0 build
> react-app-rewired build

The following changes are being made to your tsconfig.json file:
  - compilerOptions.paths must not be set (aliased imports are not supported)

Creating an optimized production build...
Failed to compile.

./src/components/MetaData/StorageHive.tsx
  Line 1:3:      Delete `␍`  prettier/prettier
  Line 2:62:     Delete `␍`  prettier/prettier
  Line 3:64:     Delete `␍`  prettier/prettier
  Line 4:57:     Delete `␍`  prettier/prettier
  Line 5:62:     Delete `␍`  prettier/prettier
  Line 6:53:     Delete `␍`  prettier/prettier
  Line 7:62:     Delete `␍`  prettier/prettier
  Line 8:62:     Delete `␍`  prettier/prettier
  Line 9:3:      Delete `␍`  prettier/prettier
  Line 10:48:    Delete `␍`  prettier/prettier
  Line 11:3:     Delete `␍`  prettier/prettier
  Line 12:62:    Delete `␍`  prettier/prettier
  Line 13:63:    Delete `␍`  prettier/prettier
  Line 14:58:    Delete `␍`  prettier/prettier
  Line 15:61:    Delete `␍`  prettier/prettier
  Line 16:59:    Delete `␍`  prettier/prettier
  Line 17:22:    Delete `␍`  prettier/prettier
  Line 18:4:     Delete `␍`  prettier/prettier
  Line 19:1:     Delete `␍`  prettier/prettier
  Line 20:27:    Delete `␍`  prettier/prettier
  Line 21:40:    Delete `␍`  prettier/prettier
  Line 22:9:     Delete `␍`  prettier/prettier
  Line 23:21:    Delete `␍`  prettier/prettier
  Line 24:25:    Delete `␍`  prettier/prettier
  Line 25:28:    Delete `␍`  prettier/prettier
  Line 26:27:    Delete `␍`  prettier/prettier
  Line 27:45:    Delete `␍`  prettier/prettier
  Line 28:78:    Delete `␍`  prettier/prettier
  Line 29:39:    Delete `␍`  prettier/prettier
  Line 30:27:    Delete `␍`  prettier/prettier
  Line 31:41:    Delete `␍`  prettier/prettier
  Line 32:55:    Delete `␍`  prettier/prettier
  Line 33:1:     Delete `␍`  prettier/prettier
  Line 34:18:    Delete `␍`  prettier/prettier
  Line 35:25:    Delete `␍`  prettier/prettier
  Line 36:12:    Delete `␍`  prettier/prettier
  Line 37:13:    Delete `␍`  prettier/prettier
  Line 38:10:    Delete `␍`  prettier/prettier
  Line 39:13:    Delete `␍`  prettier/prettier
  Line 40:14:    Delete `␍`  prettier/prettier
  Line 41:9:     Delete `␍`  prettier/prettier
  Line 42:12:    Delete `␍`  prettier/prettier
  Line 43:11:    Delete `␍`  prettier/prettier
  Line 44:12:    Delete `␍`  prettier/prettier
  Line 45:13:    Delete `␍`  prettier/prettier
  Line 46:13:    Delete `␍`  prettier/prettier
  Line 47:13:    Delete `␍`  prettier/prettier
  Line 48:12:    Delete `␍`  prettier/prettier
  Line 49:15:    Delete `␍`  prettier/prettier
  Line 50:10:    Delete `␍`  prettier/prettier
  Line 51:17:    Delete `␍`  prettier/prettier
  Line 52:17:    Delete `␍`  prettier/prettier
  Line 53:15:    Delete `␍`  prettier/prettier
  Line 54:15:    Delete `␍`  prettier/prettier
  Line 55:5:     Delete `␍`  prettier/prettier
  Line 56:1:     Delete `␍`  prettier/prettier
  Line 57:44:    Delete `␍`  prettier/prettier
  Line 58:8:     Delete `␍`  prettier/prettier
  Line 59:72:    Delete `␍`  prettier/prettier
  Line 60:7:     Delete `␍`  prettier/prettier
  Line 61:19:    Delete `␍`  prettier/prettier
  Line 62:6:     Delete `␍`  prettier/prettier
  Line 63:21:    Delete `␍`  prettier/prettier
  Line 64:76:    Delete `␍`  prettier/prettier
  Line 65:22:    Delete `␍`  prettier/prettier
  Line 66:35:    Delete `␍`  prettier/prettier
  Line 67:15:    Delete `␍`  prettier/prettier
  Line 68:72:    Delete `␍`  prettier/prettier
  Line 69:9:     Delete `␍`  prettier/prettier
  Line 70:22:    Delete `␍`  prettier/prettier
  Line 71:7:     Delete `␍`  prettier/prettier
  Line 72:6:     Delete `␍`  prettier/prettier
  Line 73:21:    Delete `␍`  prettier/prettier
  Line 74:79:    Delete `␍`  prettier/prettier
  Line 75:25:    Delete `␍`  prettier/prettier
  Line 76:35:    Delete `␍`  prettier/prettier
  Line 77:15:    Delete `␍`  prettier/prettier
  Line 78:72:    Delete `␍`  prettier/prettier
  Line 79:9:     Delete `␍`  prettier/prettier
  Line 80:22:    Delete `␍`  prettier/prettier
  Line 81:7:     Delete `␍`  prettier/prettier
  Line 82:6:     Delete `␍`  prettier/prettier
  Line 83:21:    Delete `␍`  prettier/prettier
  Line 84:85:    Delete `␍`  prettier/prettier
  Line 85:36:    Delete `␍`  prettier/prettier
  Line 86:35:    Delete `␍`  prettier/prettier
  Line 87:23:    Delete `␍`  prettier/prettier
  Line 88:91:    Delete `␍`  prettier/prettier
  Line 89:15:    Delete `␍`  prettier/prettier
  Line 90:72:    Delete `␍`  prettier/prettier
  Line 91:19:    Delete `␍`  prettier/prettier
  Line 92:12:    Delete `␍`  prettier/prettier
  Line 93:40:    Delete `␍`  prettier/prettier
  Line 94:22:    Delete `␍`  prettier/prettier
  Line 95:13:    Delete `␍`  prettier/prettier
  Line 96:12:    Delete `␍`  prettier/prettier
  Line 97:39:    Delete `␍`  prettier/prettier
  Line 98:22:    Delete `␍`  prettier/prettier
  Line 99:13:    Delete `␍`  prettier/prettier
  Line 100:11:   Delete `␍`  prettier/prettier
  Line 101:9:    Delete `␍`  prettier/prettier
  Line 102:7:    Delete `␍`  prettier/prettier
  Line 103:6:    Delete `␍`  prettier/prettier
  Line 104:21:   Delete `␍`  prettier/prettier
  Line 105:73:   Delete `␍`  prettier/prettier
  Line 106:24:   Delete `␍`  prettier/prettier
  Line 107:35:   Delete `␍`  prettier/prettier
  Line 108:15:   Delete `␍`  prettier/prettier
  Line 109:72:   Delete `␍`  prettier/prettier
  Line 110:9:    Delete `␍`  prettier/prettier
  Line 111:22:   Delete `␍`  prettier/prettier
  Line 112:7:    Delete `␍`  prettier/prettier
  Line 113:6:    Delete `␍`  prettier/prettier
  Line 114:24:   Delete `␍`  prettier/prettier
  Line 115:73:   Delete `␍`  prettier/prettier
  Line 116:24:   Delete `␍`  prettier/prettier
  Line 117:35:   Delete `␍`  prettier/prettier
  Line 118:15:   Delete `␍`  prettier/prettier
  Line 119:72:   Delete `␍`  prettier/prettier
  Line 120:17:   Delete `␍`  prettier/prettier
  Line 121:25:   Delete `␍`  prettier/prettier
  Line 122:11:   Delete `␍`  prettier/prettier
  Line 123:9:    Delete `␍`  prettier/prettier
  Line 124:7:    Delete `␍`  prettier/prettier
  Line 125:6:    Delete `␍`  prettier/prettier
  Line 126:21:   Delete `␍`  prettier/prettier
  Line 127:25:   Delete `␍`  prettier/prettier
  Line 128:23:   Delete `␍`  prettier/prettier
  Line 129:35:   Delete `␍`  prettier/prettier
  Line 130:15:   Delete `␍`  prettier/prettier
  Line 131:53:   Delete `␍`  prettier/prettier'
  ... ...

上述这个错误是由于windows下和linux下的文本文件的换行符不一致造成的。 Windows在换行的时候，同时使用了回车符CR(carriage-return character)和换行符LF(linefeed character) 而Mac和Linux系统，仅仅使用了换行符LF 老版本的Mac系统使用的是回车符CR 如果你用的是windows，文件编码是UTF-8且包含中文，最好全局将autocrlf设置为false。在windows上的解决办法：设置git的全局属性：

git config --global core.autocrlf false

之后重新拉取代码即可。

至此，Inlong编译完成：

[INFO] Reactor Summary for Apache InLong 1.2.0-incubating-SNAPSHOT:
[INFO]
[INFO] Apache InLong ...................................... SUCCESS [  9.629 s]
[INFO] Apache InLong - Common ............................. SUCCESS [ 17.793 s]
[INFO] Apache InLong - SDK ................................ SUCCESS [  2.054 s]
[INFO] Apache InLong - SDK Common ......................... SUCCESS [  9.849 s]
[INFO] Apache InLong - TubeMQ ............................. SUCCESS [  1.407 s]
[INFO] Apache InLong - TubeMQ Core ........................ SUCCESS [ 18.614 s]
[INFO] Apache InLong - TubeMQ Client ...................... SUCCESS [ 30.988 s]
[INFO] Apache InLong - Sort SDK ........................... SUCCESS [ 19.058 s]
[INFO] Apache InLong - DataProxy SDK ...................... SUCCESS [ 12.105 s]
[INFO] Apache InLong - TubeMQ Example ..................... SUCCESS [  2.645 s]
[INFO] Apache InLong - TubeMQ Server ...................... SUCCESS [ 55.146 s]
[INFO] Apache InLong - TubeMQ Connectors .................. SUCCESS [  0.317 s]
[INFO] Apache InLong - TubeMQ Connectors-flink ............ SUCCESS [  4.607 s]
[INFO] Apache InLong - TubeMQ Connectors-flume ............ SUCCESS [  4.443 s]
[INFO] Apache InLong - TubeMQ Connectors-spark ............ SUCCESS [  4.890 s]
[INFO] Apache InLong - TubeMQ Manager ..................... SUCCESS [ 24.449 s]
[INFO] Apache InLong - TubeMQ Docker ...................... SUCCESS [  0.492 s]
[INFO] Apache InLong - TubeMQ Docker All .................. SUCCESS [  0.834 s]
[INFO] Apache InLong - TubeMQ Docker Build ................ SUCCESS [  0.484 s]
[INFO] Apache InLong - TubeMQ Docker Build C++ SDK Client . SUCCESS [  0.524 s]
[INFO] Apache InLong - TubeMQ Docker Manager .............. SUCCESS [  0.990 s]
[INFO] Apache InLong - Audit .............................. SUCCESS [  0.802 s]
[INFO] Apache InLong - Audit common ....................... SUCCESS [  4.468 s]
[INFO] Apache InLong - Audit Proxy ........................ SUCCESS [ 11.293 s]
[INFO] Apache InLong - Audit store ........................ SUCCESS [ 10.550 s]
[INFO] Apache InLong - Audit Sdk .......................... SUCCESS [  5.906 s]
[INFO] Apache InLong - Audit Release ...................... SUCCESS [ 13.903 s]
[INFO] Apache InLong - Audit Docker ....................... SUCCESS [  2.122 s]
[INFO] Apache InLong - DataProxy .......................... SUCCESS [  0.937 s]
[INFO] Apache InLong - DataProxy Source ................... SUCCESS [ 21.972 s]
[INFO] Apache InLong - DataProxy Dist ..................... SUCCESS [  7.096 s]
[INFO] Apache InLong - DataProxy Docker ................... SUCCESS [  1.630 s]
[INFO] Apache InLong - Agent .............................. SUCCESS [  0.244 s]
[INFO] Apache InLong - Agent Common ....................... SUCCESS [  8.736 s]
[INFO] Apache InLong - Agent Core ......................... SUCCESS [01:09 min]
[INFO] Apache InLong - Agent Plugins ...................... SUCCESS [01:22 min]
[INFO] Apache InLong - Agent Release ...................... SUCCESS [ 10.830 s]
[INFO] Apache InLong - Agent Docker ....................... SUCCESS [  1.169 s]
[INFO] Apache InLong - Manager ............................ SUCCESS [  0.645 s]
[INFO] Apache InLong - Manager Test ....................... SUCCESS [  1.100 s]
[INFO] Apache InLong - Manager Common ..................... SUCCESS [ 12.863 s]
[INFO] Apache InLong - Manager DAO ........................ SUCCESS [  9.847 s]
[INFO] Apache InLong - Manager Workflow ................... SUCCESS [  5.552 s]
[INFO] Apache InLong - Sort ............................... SUCCESS [  0.677 s]
[INFO] Apache InLong - Sort Formats ....................... SUCCESS [  0.264 s]
[INFO] Apache InLong - Sort Format-common ................. SUCCESS [  5.887 s]
[INFO] Apache InLong - Sort Common ........................ SUCCESS [ 11.738 s]
[INFO] Apache InLong - Sort Api ........................... SUCCESS [  1.566 s]
[INFO] Apache InLong - Manager Service .................... SUCCESS [ 19.376 s]
[INFO] Apache InLong - Manager Plugins .................... SUCCESS [  7.639 s]
[INFO] Apache InLong - Manager Client ..................... SUCCESS [  7.941 s]
[INFO] Apache InLong - Manager Client Tools ............... SUCCESS [  5.964 s]
[INFO] Apache InLong - Manager Web ........................ SUCCESS [ 36.285 s]
[INFO] Apache InLong - Manager Docker ..................... SUCCESS [  2.368 s]
[INFO] Apache InLong - Manager Client Examples ............ SUCCESS [  2.181 s]
[INFO] Apache InLong - Sort Format-base ................... SUCCESS [  2.350 s]
[INFO] Apache InLong - Sort Connectors .................... SUCCESS [ 18.909 s]
[INFO] Apache InLong - Sort Format-csv .................... SUCCESS [  3.999 s]
[INFO] Apache InLong - Sort Format-kv ..................... SUCCESS [  4.025 s]
[INFO] Apache InLong - Sort Format-inlongmsg-base ......... SUCCESS [  4.335 s]
[INFO] Apache InLong - Sort Format-inlongmsg-csv .......... SUCCESS [  5.904 s]
[INFO] Apache InLong - Sort Core .......................... SUCCESS [ 25.150 s]
[INFO] Apache InLong - Sort Dist .......................... SUCCESS [ 26.529 s]
[INFO] Apache InLong - Sort Format-json ................... SUCCESS [  3.079 s]
[INFO] Apache InLong - Sort Single Tenant ................. SUCCESS [02:58 min]
[INFO] Apache InLong - Sort Standalone .................... SUCCESS [  3.245 s]
[INFO] Apache InLong - Sort Standalone Common ............. SUCCESS [ 10.481 s]
[INFO] Apache InLong - Sort Standalone Source ............. SUCCESS [ 36.347 s]
[INFO] Apache InLong - Sort Standalone Dist ............... SUCCESS [ 25.288 s]
[INFO] Apache InLong - Dashboard .......................... SUCCESS [07:13 min]
[INFO] Apache InLong - Distribution ....................... SUCCESS [02:46 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:57 min
[INFO] Finished at: 2022-04-30T15:13:37+08:00
[INFO] ------------------------------------------------------------------------

本文参与腾讯云自媒体同步曝光计划，分享自作者个人站点/博客。

原始发表：2022-04-30，如有侵权请联系 cloudcommunity@tencent.com 删除

hadoop