?.準(zhǔn)備?作

1.離線部署大綱

• MySQL離線部署
• CM離線部署
• Parcel?件離線源部署

2.規(guī)劃

linux版本:CentOS 7.2

創(chuàng)新互聯(lián)公司是一家專業(yè)提供李滄企業(yè)網(wǎng)站建設(shè),專注與網(wǎng)站制作、做網(wǎng)站、成都h5網(wǎng)站建設(shè)、小程序制作等業(yè)務(wù)。10年已為李滄眾多企業(yè)、政府機(jī)構(gòu)等服務(wù)。創(chuàng)新互聯(lián)專業(yè)網(wǎng)絡(luò)公司優(yōu)惠進(jìn)行中。

節(jié)點(diǎn)	MySQL組件	Parcel?件離線源	CM服務(wù)進(jìn)程	?數(shù)據(jù)組件
hadoop001	MySQL	Parcel	Alert Publisher Event Server	NN RM DN NM ZK
hadoop002			Alert Publisher Event Server	DN NM ZK
hadoop003			Host Monitor Service Monitor	DN NM ZK

3.下載源

• CM
  cloudera-manager-centos7-cm5.16.1x8664.tar.gz

• Parcel
  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
  manifest.json

• JDK
  JDK8
  下載jdk-8u202-linux-x64.tar.gz

• MySQL
  MYSQL5.7
  下載mysql-5.7.26-el7-x86_64.tar.gz

• MySQL JDBC jar
  mysql-connector-java-5.1.47.jar
  下載完成后要重命名去掉版本號(hào)

  mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar  

?.集群節(jié)點(diǎn)初始化

1.阿里云購買3臺(tái)虛擬機(jī)

(最低配置 2core 8G)，選擇按量付費(fèi)

CentOS7.2

2.當(dāng)前筆記本(win)hosts配置文件

路徑: C:\Windows\System32\drivers\etc\hosts

39.97.188.249   hadoop001   hadoop001
39.97.225.112   hadoop002   hadoop002
39.97.224.68    hadoop003   hadoop003

注意：IP是你虛擬機(jī)公網(wǎng)IP

3.設(shè)置所有節(jié)點(diǎn)的hosts文件

echo '172.17.144.104 hadoop001' >> /etc/hosts
echo '172.17.144.103 hadoop002' >> /etc/hosts
echo '172.17.144.105 hadoop003' >> /etc/hosts
#檢查
cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.144.104 hadoop001
172.17.144.103 hadoop002
172.17.144.105 hadoop003

注意:IP為內(nèi)網(wǎng)IP

4.關(guān)閉所有節(jié)點(diǎn)防火墻及清空規(guī)則

云主機(jī)

我們使用的云主機(jī)，無論阿里云還是騰訊云的防火墻都是關(guān)閉的，所以我們不需要關(guān)閉服務(wù)器的防火墻。但是，我們需要檢查下是否自動(dòng)開啟了web訪問端口，如果沒有，則自己添加

(1)打開安全組配置

進(jìn)入之后點(diǎn)擊配置規(guī)則

(2)添加安全組規(guī)則

注意:

1.點(diǎn)擊藍(lán)色感嘆號(hào)會(huì)有規(guī)則說明

2.授權(quán)對(duì)象如果在公司內(nèi)需要設(shè)置網(wǎng)段，就按照上圖，將ip網(wǎng)段規(guī)定好。不限制的話就直接0.0.0.0/0

內(nèi)網(wǎng)服務(wù)器

最好在內(nèi)部服務(wù)器部署時(shí)就將防火墻關(guān)閉，如果不行就暫時(shí)關(guān)閉，等部署成功再開啟

systemctl stop firewalld

systemctl disable firewalld

iptables -F   

5.關(guān)閉所有節(jié)點(diǎn)selinux

阿里云服務(wù)器已經(jīng)將selinux關(guān)閉了，所以不用配置

自己的服務(wù)器很可能會(huì)開啟selinux，這樣就需要關(guān)閉了

將SELINUX=disabled設(shè)置進(jìn)去，之后重啟才會(huì)生效

vim /etc/selinux/config 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

6.設(shè)置所有節(jié)點(diǎn)時(shí)區(qū)一致及時(shí)鐘同步

阿里云已經(jīng)將節(jié)點(diǎn)時(shí)區(qū)和時(shí)間做了同步

我們實(shí)操下公司環(huán)境的時(shí)區(qū)時(shí)間同步

6.1時(shí)區(qū)

[root@hadoop001 ~]# timedatectl
      Local time: Tue 2019-05-28 15:37:53 CST
  Universal time: Tue 2019-05-28 07:37:53 UTC
        RTC time: Tue 2019-05-28 15:37:53
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a
#查看命令幫助，學(xué)習(xí)?關(guān)重要，?需百度，太low
[root@hadoop001 ~]# timedatectl --help
timedatectl [OPTIONS...] COMMAND ...
Query or change system time and date settings.
-h --help Show this help message
--version Show package version
--no-pager Do not pipe output into a pager
--no-ask-password Do not prompt for password
-H --host=[USER@]HOST Operate on remote host
-M --machine=CONTAINER Operate on local container
--adjust-system-clock Adjust system clock when changing local RTC mode
Commands:
status Show current time settings
set-time TIME Set system time
set-timezone ZONE Set system time zone
list-timezones Show known time zones
set-local-rtc BOOL Control whether RTC is in local time
set-ntp BOOL Control whether NTP is enabled
#查看哪些時(shí)區(qū)
[root@hadoop001 ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
#所有節(jié)點(diǎn)設(shè)置亞洲上海時(shí)區(qū)
[root@hadoop001 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop002 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop003 ~]# timedatectl set-timezone Asia/Shanghai

6.2.時(shí)間

#所有節(jié)點(diǎn)安裝ntp
[root@hadoop001 ~]# yum install -y ntp
#選取hadoop001為ntp的主節(jié)點(diǎn)
[root@hadoop001 ~]# vi /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#當(dāng)外部時(shí)間不可用時(shí)，可使用本地硬件時(shí)間
server 127.127.1.0 iburst local clock
#允許哪些網(wǎng)段的機(jī)器來同步時(shí)間   修改成自己的內(nèi)網(wǎng)網(wǎng)段
restrict 172.17.144.0 mask 255.255.255.0 nomodify notrap
#開啟ntpd及查看狀態(tài)
[root@hadoop001 ~]# systemctl start ntpd
[root@hadoop001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: d
isabled)
Active: active (running) since Sat 2019-05-11 10:15:00 CST; 11min ago
Main PID: 18518 (ntpd)
CGroup: /system.slice/ntpd.service
!"18518 /usr/sbin/ntpd -u ntp:ntp -g
May 11 10:15:00 hadoop001 systemd[1]: Starting Network Time Service...
May 11 10:15:00 hadoop001 ntpd[18518]: proto: precision = 0.088 usec
May 11 10:15:00 hadoop001 ntpd[18518]: 0.0.0.0 c01d 0d kern kernel time sync enabl
ed
May 11 10:15:00 hadoop001 systemd[1]: Started Network Time Service.
#驗(yàn)證
[root@hadoop001 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 726 64 0 0.000 0.000 0.000
#其他從節(jié)點(diǎn)停?禁?ntpd服務(wù)
[root@hadoop002 ~]# systemctl stop ntpd
[root@hadoop002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop002 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步hadoop001節(jié)點(diǎn)時(shí)間[root@hadoop002 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
[root@hadoop003 ~]# systemctl stop ntpd
[root@hadoop004 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop005 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步hadoop001節(jié)點(diǎn)時(shí)間
[root@hadoop003 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001

7.JDK部署

mkdir /usr/java
tar -xzvf jdk-8u45-linux-x64.tar.gz -C /usr/java/
#切記必須修正所屬?戶及?戶組
chown -R root:root /usr/java/jdk1.8.0_45
[root@hadoop001 cdh6.16.1]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=${JAVA_HOME}/bin:${PATH}
source /etc/profile
which java

如果節(jié)點(diǎn)過多，那么就只做一臺(tái)鏡像模板，將基礎(chǔ)工作完成之后，分發(fā)克隆。。（最好請(qǐng)運(yùn)維小哥哥做~）

8.hadoop001節(jié)點(diǎn)離線部署MySQL5.7

（按照生產(chǎn)標(biāo)準(zhǔn)）

8.1 解壓及創(chuàng)建文件夾

#解壓
[root@hadoop001 cdh6.16.1]# tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local/
#切換目錄
[root@hadoop001 cdh6.16.1]# cd /usr/local/
#修改mysql名稱
[root@hadoop001 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64/ mysql
#創(chuàng)建文件夾
[root@hadoop001 local]# mkdir mysql/arch mysql/data mysql/tmp

8.2 創(chuàng)建my.cnf

rm /etc/my.cnf

vim /etc/my.cnf

[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500

#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根據(jù)生產(chǎn)需要，調(diào)整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

8.3 創(chuàng)建用戶組及用戶

[root@hadoop001 local]# groupadd -g 101 dba
[root@hadoop001 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin

[root@hadoop001 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)

## 一般不需要設(shè)置mysqladmin的密碼，直接從root或者LDAP用戶sudo切換

8.4 copy 環(huán)境變量配置文件

copy 環(huán)境變量配置文件(隱藏文件)至mysqladmin用戶的home目錄中,為了以下步驟配置個(gè)人環(huán)境變量

cp /etc/skel/.* /usr/local/mysql  

8.5 配置環(huán)境變量

[root@hadoop001 local]# vi mysql/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH

unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

8.6 賦權(quán)限和用戶組 切換用戶mysqladmin 安裝

[root@hadoop001 local]# chown  mysqladmin:dba /etc/my.cnf 
[root@hadoop001 local]# chmod  640 /etc/my.cnf  
[root@hadoop001 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop001 local]# chmod -R 755 /usr/local/mysql 

8.7 配置服務(wù)及開機(jī)自啟動(dòng)

[root@hadoop001 local]#  cd /usr/local/mysql
#將服務(wù)文件拷貝到init.d下，并重命名為mysql
[root@hadoop001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql 
#賦予可執(zhí)行權(quán)限
[root@hadoop001 mysql]# chmod +x /etc/rc.d/init.d/mysql
#刪除服務(wù)
[root@hadoop001 mysql]# chkconfig --del mysql
#添加服務(wù)
[root@hadoop001 mysql]# chkconfig --add mysql
[root@hadoop001 mysql]# chkconfig --level 345 mysql on

8.8 安裝libaio及安裝mysql的初始db

[root@hadoop001 mysql]# yum -y install libaio

[root@hadoop001 mysql]# su - mysqladmin
Last login: Tue May 28 17:04:49 CST 2019 on pts/0
hadoop001:mysqladmin:/usr/local/mysql:>bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize

在初始化時(shí)如果加上 –initial-insecure，則會(huì)創(chuàng)建空密碼的 root@localhost 賬號(hào)，否則會(huì)創(chuàng)建帶密碼的 root@localhost 賬號(hào)，密碼直接寫在 log-error 日志文件中（在5.6版本中是放在 ~/.mysql_secret 文件里，更加隱蔽，不熟悉的話可能會(huì)無所適從）

8.9 查看臨時(shí)密碼

#查看密碼
hadoop001:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password
2019-05-28T09:28:40.447701Z 1 [Note] A temporary password is generated for root@localhost: J=<z#diyC4fh

8.10 啟動(dòng)

hadoop001:mysqladmin:/usr/local/mysql:>/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &
[1] 21740
hadoop001:mysqladmin:/usr/local/mysql:>2019-05-28T09:38:16.127060Z mysqld_safe Logging to '/usr/local/mysql/data/hostname.err'.
2019-05-28T09:38:16.196799Z mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
#按兩次回車

##退出mysqladmin用戶
##查看mysql進(jìn)程號(hào)
[root@hadoop001 mysql]#ps -ef|grep mysql
mysqlad+ 21740     1  0 17:38 pts/0    00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf
mysqlad+ 22557 21740  0 17:38 pts/0    00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --log-error=/usr/local/mysql/data/hostname.err --pid-file=/usr/local/mysql/data/hostname.pid --socket=/usr/local/mysql/data/mysql.sock --port=3306
root     22609  9194  0 17:39 pts/0    00:00:00 grep --color=auto mysql
##通過mysql進(jìn)程號(hào)查看mysql端口號(hào)
[root@hadoop001 mysql]# netstat -nlp|grep 22557
#切換成mysqladmin
[root@hadoop001 mysql]# su - mysqladmin
Last login: Tue May 28 17:24:45 CST 2019 on pts/0
hadoop001:mysqladmin:/usr/local/mysql:>
##查看mysql是否運(yùn)行
hadoop001:mysqladmin:/usr/local/mysql:>service mysql status
MySQL running (22557)[  OK  ]

8.11 登錄及修改用戶密碼

#初始密碼
hadoop001:mysqladmin:/usr/local/mysql:>mysql -uroot -p'J=<z#diyC4fh'   
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#重置密碼
mysql> alter user root@localhost identified by 'ruozedata123';
mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'ruozedata123' ;
#刷權(quán)限
mysql> flush privileges;

8.12 重啟

9.創(chuàng)建CDH的元數(shù)據(jù)庫和用戶、 amon服務(wù)的數(shù)據(jù)庫及用戶

mysql> CREATE DATABASE `cmf`  DEFAULT CHARACTER SET utf8;

mysql> GRANT ALL PRIVILEGES ON cmf.* TO 'cmf'@'%' IDENTIFIED BY 'ruozedata123' ;

mysql> create database amon default character set utf8;

mysql> GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'ruozedata123' ;

--刷權(quán)限
mysql> flush privileges;

10.部署 mysql JDBC jar

[root@hadoop001 cdh6.16.1]# mkdir -p /usr/share/java
[root@hadoop001 cdh6.16.1]# ls -lh
total 3.5G
-rw-r--r-- 1 root root 2.0G May 15 10:01 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root   41 May 14 20:17 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 803M May 15 09:38 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 166M May 14 20:21 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root  65K May 14 20:17 manifest.json
-rw-r--r-- 1 root root 523M May 15 09:28 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 984K May 15 09:10 mysql-connector-java-5.1.47.jar
#mysql的jar包一定要去掉版本號(hào)~，有坑
[root@hadoop001 cdh6.16.1]# cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

三.CDH離線部署

1.部署CM Server 和Agent

1.1 所有節(jié)點(diǎn)創(chuàng)建?錄及解壓

mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager
/
sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.2 所有節(jié)點(diǎn)修改config.ini

所有節(jié)點(diǎn)修改agent的配置，指向server的節(jié)點(diǎn)hadoop001

sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.3 主節(jié)點(diǎn)修改server的配置

vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=ruozedata123
com.cloudera.cmf.db.setupType=EXTERNAL

1.4 所有節(jié)點(diǎn)創(chuàng)建cloudera-scm用戶

#創(chuàng)建cloudera-scm
useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
#修改cloudera-manager的權(quán)限
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

1.5 所有節(jié)點(diǎn)修改cloudera-manager用戶名用戶組

chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

2.hadoop001節(jié)點(diǎn)部署離線parcel源

2.1 部署離線parcel源

mkdir -p /opt/cloudera/parcel-repo

[root@hadoop001 opt]# cd ~/cdh6.16.1/
[root@hadoop001 cdh6.16.1]# ls -lh
total 3.5G
-rw-r--r-- 1 root root 2.0G May 15 10:01 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root   41 May 14 20:17 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 803M May 15 09:38 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 166M May 14 20:21 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root  65K May 14 20:17 manifest.json
-rw-r--r-- 1 root root 523M May 15 09:28 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 984K May 15 09:10 mysql-connector-java-5.1.47.jar

[root@hadoop001 cdh6.16.1]# mv CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha

#切記mv時(shí)，重命名去掉1，不然在部署過程CM認(rèn)為如上?件下載未完整，會(huì)持續(xù)下載

[root@hadoop001 cdh6.16.1]# mv CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/

[root@hadoop001 cdh6.16.1]# mv manifest.json  /opt/cloudera/parcel-repo/

如果你是通過網(wǎng)絡(luò)下載的parcel包，我們就需要對(duì)CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel 進(jìn)行校驗(yàn)，防止文件損壞!!!

[root@hadoop001 parcel-repo]# cat CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha
703728dfa7690861ecd3a9bcd412b04ac8de7148
#計(jì)算下載文件的值，進(jìn)行對(duì)比
[root@hadoop001 parcel-repo]# sha1sum CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
703728dfa7690861ecd3a9bcd412b04ac8de7148  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
#相同，可以正常使用

2.2 目錄修改用戶及用戶組

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

3.所有節(jié)點(diǎn)創(chuàng)建大數(shù)據(jù)軟件安裝目錄、用戶及用戶組權(quán)限

mkdir -p /opt/cloudera/parcels
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

4.hadoop001節(jié)點(diǎn)啟動(dòng)Server

4.1 啟動(dòng)server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start
4.2 阿?云web界?，設(shè)置該hadoop001節(jié)點(diǎn)防?墻放開7180端?
4.3 等待1min，打開 http://hadoop001:7180 賬號(hào)密碼:admin/admin
4.4 假如打不開，去看server的log，根據(jù)錯(cuò)誤仔細(xì)排查錯(cuò)誤 
    log路徑在/opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server

5.所有節(jié)點(diǎn)啟動(dòng)Agent

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

6.接下來，全部Web界面操作

http://hadoop001:7180/

賬號(hào)密碼:admin/admin

7.歡迎使?Cloudera Manager--最終?戶許可條款與條件。勾選

8.歡迎使?Cloudera Manager--您想要部署哪個(gè)版本？選擇Cloudera Express免費(fèi)版本

9.感謝您選擇Cloudera Manager和CDH

10.為CDH集群安裝指導(dǎo)主機(jī)。選擇[當(dāng)前管理的主機(jī)]，全部勾選

11.選擇存儲(chǔ)庫

12.集群安裝--正在安裝選定Parcel

假如本地parcel離線源配置正確，則"下載"階段瞬間完成，其余階段視節(jié)點(diǎn)數(shù)與內(nèi)部?絡(luò)情況決定。

13.檢查主機(jī)正確性

13.1.建議將/proc/sys/vm/swappiness設(shè)置為最?值10。
swappiness值控制操作系統(tǒng)嘗試交換內(nèi)存的積極；
swappiness=0：表示最?限度使?物理內(nèi)存，之后才是swap空間；
swappiness=100：表示積極使?swap分區(qū)，并且把內(nèi)存上的數(shù)據(jù)及時(shí)搬遷到swap空間；
如果是混合服務(wù)器，不建議完全禁?swap，可以嘗試降低swappiness。
臨時(shí)調(diào)整：
sysctl vm.swappiness=10
永久調(diào)整：
cat << EOF >> /etc/sysctl.conf
# Adjust swappiness value
vm.swappiness=10
EOF
13.2.已啟?透明???壓縮，可能會(huì)導(dǎo)致重?性能問題，建議禁?此設(shè)置。
臨時(shí)調(diào)整：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久調(diào)整：
cat << EOF >> /etc/rc.d/rc.local
# Disable transparent_hugepage
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
# centos7.x系統(tǒng)，需要為"/etc/rc.d/rc.local"?件賦予執(zhí)?權(quán)限
chmod +x /etc/rc.d/rc.local

14.自定義服務(wù)，選擇部署Zookeeper、 HDFS、 Yarn服務(wù)

15.自定義角色分配

16.數(shù)據(jù)庫設(shè)置

連接測(cè)試失敗的可能原因:
(1)mysql JDBC jar包沒有放到/usr/share/java或jar包沒有去掉版本號(hào)
(2)建數(shù)據(jù)庫cmf 和amon的時(shí)候，沒有將權(quán)限設(shè)置成%
(3)設(shè)置完權(quán)限之后，沒有flush privileges;

17.審改設(shè)置，默認(rèn)即可

18.?次運(yùn)?

19.恭喜您!

20.主頁

四.報(bào)錯(cuò)

1.在數(shù)據(jù)庫設(shè)置測(cè)試時(shí)發(fā)生報(bào)錯(cuò)

報(bào)錯(cuò)信息

ERROR 226616765@scm-web-17:com.cloudera.server.web.common.JsonResponse:
JsonResponse created with throwable: com.cloudera.server.web.cmf.MessageException:
A package was not selected.

原因:
測(cè)試連接時(shí)，等待時(shí)間過長(zhǎng)，我就點(diǎn)了返回鍵重新加載，然后出現(xiàn)packet找不到的異常。
解決:
返回到選擇大數(shù)據(jù)組件的頁面后，重新進(jìn)行操作，就可以測(cè)試成功了。

本文名稱：CDH5.16.1集群企業(yè)真正離線部署
文章分享：http://m.rwnh.cn/article42/jcgihc.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供響應(yīng)式網(wǎng)站、App設(shè)計(jì)、用戶體驗(yàn)、網(wǎng)站設(shè)計(jì)、軟件開發(fā)、小程序開發(fā)

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來源： 創(chuàng)新互聯(lián)