搭建Hadoop集群(二)

用户9615083

发布于 2022-12-25 15:07:27

55100

代码可运行

文章被收录于专栏：Java后端开发博客Java后端开发博客

运行总次数：0

代码可运行

# 搭建Hadoop集群(二)

# 安装和配置MySQL

开启主节点虚拟机

mkdir /opt/mysql

cd /opt/mysql

使用wget命令下载mysql

wget http://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

安装MySQL官方的Yum Repository

rpm -ivh mysql57-community-release-el7-11.noarch.rpm

查看提供的MySQL版本

yum repolist all | grep mysql

安装MySQL

yum install -y mysql-community-server

在安装的过程中出现了以下问题

说xxx的公钥尚未安装

解决办法:

rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022

再次安装，成功

yum install mysql-server

再次安装中途可能需要需要你输入y

启动MySQL

systemctl start mysqld

使用命令systemctl status mysqld

[root@master ~]# systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2022-05-09 17:15:03 CST; 2min 5s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 3539 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 3486 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 3542 (mysqld)
    Tasks: 27
   CGroup: /system.slice/mysqld.service
           └─3542 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

5月 09 17:14:57 master systemd[1]: Starting MySQL Server...
5月 09 17:15:03 master systemd[1]: Started MySQL Server.

测试MySQL

mysql -u root -p

密码不知道是什么,使用命令查看一下 grep 'temporary password' /var/log/mysqld.log
可以看到...root@localhost:<---密码---->

g?Uq#GAeM8YK 这就是我的临时密码

使用root和初始化临时密码登录测试，输入你的临时密码

mysql -u root -p

出现mysql命令行，说明登陆成功

[root@master ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.38

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

设置root密码，对于个人开发环境，可以设置比较简单的密码

set global validate_password_policy=0;

设置密码为12345678

set password for 'root'@'localhost'=password('12345678');

设置八位哦

使用命令exit退出MySQL命令行，重新登录进行验证
展示所有的数据库

show databases;

# 安装和配置Hive

在MySQL中创建Hive所需用户和数据库并授权
- 在MySQL中创建用户hive,密码为12345678

create user 'hive' identified by '12345678';

创建数据库hive,使用命令

create database hive;

将将数据库hive的所有权限授权于用户hive,使用命令

grant all privileges on hive.* to 'hive'@'localhost' identified by '12345678';

刷新权限，使其立即生效

flush privileges;

使用hive登录,并查看是否能看到数据库hive

mysql> exit
Bye
[root@master ~]# mysql -u hive -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 14
Server version: 5.7.38 MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
+--------------------+
2 rows in set (0.00 sec)

# 下载Hive

下载地址 (opens new window)
钉钉群里分享的有😊

mkdir /opt/hive
cd /opt/hive

使用XFTP链接linux系统,把hive的压缩包传输到Linux系统中

tar -zxvf apache-hive-2.3.9-bin.tar.gz

解压后文件名字太复杂，配置环境变量时容易写错，重命名一下

mv apache-hive-2.3.9-bin hive-2.3.9

将解压后的文件移动到/usr/local/

mv hive-2.3.9 /usr/local

# 下载MySQL的JDBC驱动包

下载地址 (opens new window)
钉钉群里分享的有😊

mkdir /opt/mysql-connector-java
cd /opt/mysql-connector-java

使用XFTP传输到Linux系统

解压驱动包,

tar -zxvf mysql-connector-java-5.1.48.tar.gz

把mysql-connector-java-5.1.48下的 mysql-connector-java-5.1.48-bin.jar移动至/usr/local/hive-2.3.9/lib下

cd mysql-connector-java-5.1.48
mv mysql-connector-java-5.1.48-bin.jar /usr/local/hive-2.3.9/lib/

# 配置Hive

解压好的文件，bin目录放核心运行文件，也就是命令 conf目录放核心配置文件 lib目录放依赖的jar包

[root@master conf]# cd /usr/local/hive-2.3.9/conf/
[root@master conf]# ls
beeline-log4j2.properties.template  hive-exec-log4j2.properties.template  llap-cli-log4j2.properties.template
hive-default.xml.template           hive-log4j2.properties.template       llap-daemon-log4j2.properties.template
hive-env.sh.template                ivysettings.xml                       parquet-logging.properties

配置文件hive-env.sh

备份一下

cp hive-env.sh.template hive-env.sh

vim hive-env.sh

指定Hadoop安装路径

按Esc切换到一般模式,输入:set nu,显示行号

指定你的hadoop、Hive安装路径

配置文件hive-default.xml

cp hive-default.xml.template hive-default.xml

hive-default.xml是Hive默认加载的文件

配置文件hive-site.xml

vim hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
	<property>
		 <name>javax.jdo.option.ConnectionURL</name>
    		<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false</value>  <!-- 连接地址 中间的数据库名字不要写错 -->
	</property>
	<property>
		 <name>javax.jdo.option.ConnectionDriverName</name>
    		<value>com.mysql.jdbc.Driver</value>  <!-- 注册驱动类-->
	</property>
	<property>
		 <name>javax.jdo.option.ConnectionUserName</name>
    		<value>hive</value>  <!-- 连接用户名 -->
	</property>
	<property>
		 <name>javax.jdo.option.ConnectionPassword</name>
    		<value>12345678</value>  <!-- 连接用户密码 -->
	</property>
</configuration>

一次复制不上去，分几次复制

配置Hive的环境变量

vim /etc/profile

# set hive environment
export HIVE_HOME=/usr/local/hive-2.3.9
export PATH=$HIVE_HOME/bin:$PATH

使用命令source /etc/profile重新加载配置文件,使配置生效

source /etc/profile

初始化元数据

cd /usr/local/hive-2.3.9/bin

[root@master bin]# schematool -initSchema -dbType mysql
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hive-2.3.9/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:        jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&useSSL=false
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:       hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed