文章/答案/技术大牛

发布

社区首页 >专栏 >故障分析 | Greenplum Segment 故障处理

故障分析 | Greenplum Segment 故障处理

原创

爱可生开源社区

发布于 2023-02-02 08:46:45

2.1K02

代码可运行

文章被收录于专栏：爱可生开源社区爱可生开源社区

运行总次数：2

代码可运行

作者：杨文
DBA，负责客户项目的需求与维护，会点数据库，不限于MySQL、Redis、Cassandra、GreenPlum、ClickHouse、Elastic、TDSQL等等。
本文来源：原创投稿
*爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。

一、前情提要：

我们知道Greenplum集群由Master Severs和Segment Severs组成。其中故障存在三种类别：Master故障、Segment故障、数据异常。之前我们已经聊过“Master故障”和“数据异常”的处理方式，今天将介绍Segment故障的处理方式。

二、本地模拟故障环境：

2.1、第一种情况：段故障。

[gpadmin@master ~]$ gpstate
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: 
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:39:00:022659 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
...
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-Greenplum instance status summary
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Master instance                                           = Active
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Master standby                                            = standby
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total segment instance count from metadata                = 40
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Primary Segment Status
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total primary segments                                    = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total primary segment failures (at master)                = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number postmaster processes found                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Mirror Segment Status
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segments                                     = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 20
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total mirror segment failures (at master)                 = 0
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid files missing              = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing               = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number of /tmp lock files missing                   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number postmaster processes missing                 = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number postmaster processes found                   = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments   = 4                      <<<<<<<<
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 16
20221127:22:39:03:022659 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------

[gpadmin@master ~]$ gpstate -m
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -m
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--Current GPDB mirror list and status
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--Type = Group
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   Mirror       Datadir                            Port    Status              Data Status    
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg0    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg1    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg2    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg3    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg4    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg5    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg6    56000   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg7    56001   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg8    56002   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg9    56000   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg10   56001   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg11   56002   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg12   56000   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg13   56001   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg14   56002   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data02       /greenplum/gpdata/mirror/gpseg15   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data03       /greenplum/gpdata/mirror/gpseg16   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data04       /greenplum/gpdata/mirror/gpseg17   56003   Passive             Synchronized
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-data05       /greenplum/gpdata/mirror/gpseg18   56003   Failed                             <<<<<<<<
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:-   data01       /greenplum/gpdata/mirror/gpseg19   56003   Acting as Primary   Not In Sync
20221127:22:44:55:023196 gpstate:master:gpadmin-[INFO]:--------------------------------------------------------------
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 segment(s) configured as mirror(s) are acting as primaries
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 segment(s) configured as mirror(s) have failed
20221127:22:44:55:023196 gpstate:master:gpadmin-[WARNING]:-4 mirror segment(s) acting as primaries are not synchronized

2.2、第二种情况：表空间故障。

[gpadmin@data05 ~]$ cd /greenplum/gpdata/mirror/gpseg10
[gpadmin@data05 gpseg10]$ ls
backup_label.old    gpmetrics               pg_clog            pg_logical    pg_stat                PG_VERSION            postmaster.pid
base                gpperfmon               pg_distributedlog  pg_multixact  pg_stat_tmp            pg_xlog               recovery.conf
fts_probe_file.bak  gpsegconfig_dump        pg_dynshmem        pg_notify     pg_subtrans            postgresql.auto.conf  recovery.done
global              gpssh.conf              pg_hba.conf        pg_replslot   pg_tblspc              postgresql.conf
gpexpand.pid        internal.auto.conf      pg_ident.conf      pg_serial     pg_twophase            postgresql.conf.bak
gpexpand.status     internal.auto.conf.bak  pg_log             pg_snapshots  pg_utilitymodedtmredo  postmaster.opts
[gpadmin@data05 gpseg10]$ rm -rf pg_tblspc/

[gpadmin@master ~]$ gpstate -e
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:13:29:026114 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:23:13:30:026114 gpstate:master:gpadmin-[WARNING]:-pg_stat_replication shows no standby connections
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-Downed Segments (may include segments where status could not be retrieved)
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-   Segment      Port    Config status   Status
20221127:23:13:30:026114 gpstate:master:gpadmin-[INFO]:-   data05       56001   Up              Process error -- database process may be down

三、故障分析及解决：

3.1、针对“2.1”情况的处理：

在线生成一个配置文件：

[gpadmin@master ~]$ gprecoverseg -o ./recover1
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Starting gprecoverseg with args: -o ./recover1
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:48:41:023405 gprecoverseg:master:gpadmin-[INFO]:-Configuration file output to ./recover1 successfully.

[gpadmin@master ~]$ more recover1
data05|55000|/greenplum/gpdata/primary/gpseg12
data05|55001|/greenplum/gpdata/primary/gpseg13
data05|55002|/greenplum/gpdata/primary/gpseg14
data05|55003|/greenplum/gpdata/primary/gpseg19

通过生成的配置文件进行修复集群：

[gpadmin@master ~]$ gprecoverseg -i ./recover1 -a

检查状态：

[gpadmin@master ~]$ gpstate -e
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:22:56:57:024771 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-Segments with Primary and Mirror Roles Switched
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   Current Primary   Port    Mirror       Port
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56000   data05       55000
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56001   data05       55001
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data04            56002   data05       55002
20221127:22:56:58:024771 gpstate:master:gpadmin-[INFO]:-   data01            56003   data05       55003

[gpadmin@master ~]$ psql -c "select * from gp_segment_configuration order by content asc,dbid;"
 dbid | content | role | preferred_role | mode | status | port  | hostname | address |             datadir              
------+---------+------+----------------+------+--------+-------+----------+---------+-----------------------------------
   44 |      -1 | p    | p              | s    | u      |  5432 | master   | master  | /greenplum/gpdata/master/gpseg-1
   45 |      -1 | m    | m              | s    | u      |  5432 | standby  | standby | /greenplum/gpdata/master/gpseg-1
    2 |       0 | p    | p              | s    | u      | 55000 | data01   | data01  | /greenplum/gpdata/primary/gpseg0
   11 |       0 | m    | m              | s    | u      | 56000 | data02   | data02  | /greenplum/gpdata/mirror/gpseg0
    3 |       1 | p    | p              | s    | u      | 55001 | data01   | data01  | /greenplum/gpdata/primary/gpseg1
   12 |       1 | m    | m              | s    | u      | 56001 | data02   | data02  | /greenplum/gpdata/mirror/gpseg1
    4 |       2 | p    | p              | s    | u      | 55002 | data01   | data01  | /greenplum/gpdata/primary/gpseg2
   13 |       2 | m    | m              | s    | u      | 56002 | data02   | data02  | /greenplum/gpdata/mirror/gpseg2
    5 |       3 | p    | p              | s    | u      | 55000 | data02   | data02  | /greenplum/gpdata/primary/gpseg3
   14 |       3 | m    | m              | s    | u      | 56000 | data03   | data03  | /greenplum/gpdata/mirror/gpseg3
    6 |       4 | p    | p              | s    | u      | 55001 | data02   | data02  | /greenplum/gpdata/primary/gpseg4
   15 |       4 | m    | m              | s    | u      | 56001 | data03   | data03  | /greenplum/gpdata/mirror/gpseg4
    7 |       5 | p    | p              | s    | u      | 55002 | data02   | data02  | /greenplum/gpdata/primary/gpseg5
   16 |       5 | m    | m              | s    | u      | 56002 | data03   | data03  | /greenplum/gpdata/mirror/gpseg5
    8 |       6 | p    | p              | s    | u      | 55000 | data03   | data03  | /greenplum/gpdata/primary/gpseg6
   17 |       6 | m    | m              | s    | u      | 56000 | data01   | data01  | /greenplum/gpdata/mirror/gpseg6
    9 |       7 | p    | p              | s    | u      | 55001 | data03   | data03  | /greenplum/gpdata/primary/gpseg7
   18 |       7 | m    | m              | s    | u      | 56001 | data01   | data01  | /greenplum/gpdata/mirror/gpseg7
   10 |       8 | p    | p              | s    | u      | 55002 | data03   | data03  | /greenplum/gpdata/primary/gpseg8
   19 |       8 | m    | m              | s    | u      | 56002 | data01   | data01  | /greenplum/gpdata/mirror/gpseg8
   21 |       9 | p    | p              | s    | u      | 55000 | data04   | data04  | /greenplum/gpdata/primary/gpseg9
   30 |       9 | m    | m              | s    | u      | 56000 | data05   | data05  | /greenplum/gpdata/mirror/gpseg9
   22 |      10 | p    | p              | s    | u      | 55001 | data04   | data04  | /greenplum/gpdata/primary/gpseg10
   31 |      10 | m    | m              | s    | u      | 56001 | data05   | data05  | /greenplum/gpdata/mirror/gpseg10
   23 |      11 | p    | p              | s    | u      | 55002 | data04   | data04  | /greenplum/gpdata/primary/gpseg11
   32 |      11 | m    | m              | s    | u      | 56002 | data05   | data05  | /greenplum/gpdata/mirror/gpseg11
   24 |      12 | m    | p              | s    | u      | 55000 | data05   | data05  | /greenplum/gpdata/primary/gpseg12
   27 |      12 | p    | m              | s    | u      | 56000 | data04   | data04  | /greenplum/gpdata/mirror/gpseg12
   25 |      13 | m    | p              | s    | u      | 55001 | data05   | data05  | /greenplum/gpdata/primary/gpseg13
   28 |      13 | p    | m              | s    | u      | 56001 | data04   | data04  | /greenplum/gpdata/mirror/gpseg13
   26 |      14 | m    | p              | s    | u      | 55002 | data05   | data05  | /greenplum/gpdata/primary/gpseg14
   29 |      14 | p    | m              | s    | u      | 56002 | data04   | data04  | /greenplum/gpdata/mirror/gpseg14
   33 |      15 | p    | p              | s    | u      | 55003 | data01   | data01  | /greenplum/gpdata/primary/gpseg15
   39 |      15 | m    | m              | s    | u      | 56003 | data02   | data02  | /greenplum/gpdata/mirror/gpseg15
   34 |      16 | p    | p              | s    | u      | 55003 | data02   | data02  | /greenplum/gpdata/primary/gpseg16
   40 |      16 | m    | m              | s    | u      | 56003 | data03   | data03  | /greenplum/gpdata/mirror/gpseg16
   35 |      17 | p    | p              | s    | u      | 55003 | data03   | data03  | /greenplum/gpdata/primary/gpseg17
   41 |      17 | m    | m              | s    | u      | 56003 | data04   | data04  | /greenplum/gpdata/mirror/gpseg17
   36 |      18 | p    | p              | s    | u      | 55003 | data04   | data04  | /greenplum/gpdata/primary/gpseg18
   42 |      18 | m    | m              | s    | u      | 56003 | data05   | data05  | /greenplum/gpdata/mirror/gpseg18
   37 |      19 | m    | p              | s    | u      | 55003 | data05   | data05  | /greenplum/gpdata/primary/gpseg19
   38 |      19 | p    | m              | s    | u      | 56003 | data01   | data01  | /greenplum/gpdata/mirror/gpseg19
(42 rows)

可以看到所有段都是up状态了，但存在部分段角色异常。

修复角色状态：

[gpadmin@master ~]$ gprecoverseg -r

再次检查确认状态，此处省略。

3.2、针对“2.2”情况的处理：

如果可以自动生成配置文件，就使用自动生成的。如果无法自动生成，则手工创建：

[gpadmin@master ~]$ vi recover2
data05|56001|/greenplum/gpdata/mirror/gpseg10

通过生成的配置文件进行修复集群：

[gpadmin@master ~]$ gprecoverseg -i ./recover2 -a
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Starting gprecoverseg with args: -i ./recover2 -F
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Greenplum instance recovery parameters
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Recovery from configuration -i option supplied
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-Recovery 1 of 1
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Synchronization mode                 = Full
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance host                 = data05
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance address              = data05
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance directory            = /greenplum/gpdata/mirror/gpseg10
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Failed instance port                 = 56001
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance host        = data04
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance address     = data04
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance directory   = /greenplum/gpdata/primary/gpseg10
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Source instance port        = 55001
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:-   Recovery Target                      = in-place
20221127:23:15:43:026332 gprecoverseg:master:gpadmin-[INFO]:----------------------------------------------------------
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-1 segment(s) to recover
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20221127:23:15:47:026332 gprecoverseg:master:gpadmin-[INFO]:-Validating remote directories
20221127:23:15:48:026332 gprecoverseg:master:gpadmin-[INFO]:-Configuring new segments data05 (dbid 31): pg_basebackup: base backup completed
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating configuration with new mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Starting mirrors
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-era is c6f862530103c913_221127213422
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Process results...
20221127:23:15:51:026332 gprecoverseg:master:gpadmin-[INFO]:-Triggering FTS probe
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-******************************************************************
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-Updating segments for streaming is completed.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-Use  gpstate -s  to check the streaming progress.
20221127:23:15:52:026332 gprecoverseg:master:gpadmin-[INFO]:-******************************************************************

进程检查：

[gpadmin@data05 gpseg10]$ ps -ef |grep postgres
gpadmin   45364      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg13 -p 55001
gpadmin   45367      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg12 -p 55000
gpadmin   45369      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg14 -p 55002
gpadmin   45373      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/primary/gpseg19 -p 55003
gpadmin   45378      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg9 -p 56000
gpadmin   45380      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg18 -p 56003
gpadmin   45382      1  0 22:53 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg11 -p 56002
gpadmin   47899      1  0 23:15 ?        00:00:00 /usr/local/greenplum-db-6.7.0/bin/postgres -D /greenplum/gpdata/mirror/gpseg10 -p 56001
......

表空间检查：

[gpadmin@data05 gpseg10]$ ls
backup_label.old    gpmetrics               pg_clog            pg_logical    pg_stat                PG_VERSION            postmaster.pid
base                gpperfmon               pg_distributedlog  pg_multixact  pg_stat_tmp            pg_xlog               recovery.conf
fts_probe_file.bak  gpsegconfig_dump        pg_dynshmem        pg_notify     pg_subtrans            postgresql.auto.conf  recovery.done
global              gpssh.conf              pg_hba.conf        pg_replslot   pg_tblspc              postgresql.conf
gpexpand.pid        internal.auto.conf      pg_ident.conf      pg_serial     pg_twophase            postgresql.conf.bak
gpexpand.status     internal.auto.conf.bak  pg_log             pg_snapshots  pg_utilitymodedtmredo  postmaster.opts

状态检查：

[gpadmin@master ~]$ gpstate -e
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Starting gpstate with args: -e
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b'
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.7.0 build commit:2fbc274bc15a19b5de3c6e44ad5073464cd4f47b) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Apr 16 2020 02:24:06'
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Obtaining Segment details from master...
20221127:23:23:01:026934 gpstate:master:gpadmin-[INFO]:-Gathering data from segments...
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-Segment Mirroring Status Report
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-----------------------------------------------------
20221127:23:23:02:026934 gpstate:master:gpadmin-[INFO]:-All segments are running normally

对于这种情况，一般不会存在数据节点状态异常的情况：

[gpadmin@master ~]$ psql -c "select * from gp_segment_configuration order by content asc,dbid;"
 dbid | content | role | preferred_role | mode | status | port  | hostname | address |             datadir              
------+---------+------+----------------+------+--------+-------+----------+---------+-----------------------------------
   44 |      -1 | p    | p              | s    | u      |  5432 | master   | master  | /greenplum/gpdata/master/gpseg-1
   45 |      -1 | m    | m              | s    | u      |  5432 | standby  | standby | /greenplum/gpdata/master/gpseg-1
    2 |       0 | p    | p              | s    | u      | 55000 | data01   | data01  | /greenplum/gpdata/primary/gpseg0
   11 |       0 | m    | m              | s    | u      | 56000 | data02   | data02  | /greenplum/gpdata/mirror/gpseg0
    3 |       1 | p    | p              | s    | u      | 55001 | data01   | data01  | /greenplum/gpdata/primary/gpseg1
   12 |       1 | m    | m              | s    | u      | 56001 | data02   | data02  | /greenplum/gpdata/mirror/gpseg1
    4 |       2 | p    | p              | s    | u      | 55002 | data01   | data01  | /greenplum/gpdata/primary/gpseg2
   13 |       2 | m    | m              | s    | u      | 56002 | data02   | data02  | /greenplum/gpdata/mirror/gpseg2
    5 |       3 | p    | p              | s    | u      | 55000 | data02   | data02  | /greenplum/gpdata/primary/gpseg3
   14 |       3 | m    | m              | s    | u      | 56000 | data03   | data03  | /greenplum/gpdata/mirror/gpseg3
    6 |       4 | p    | p              | s    | u      | 55001 | data02   | data02  | /greenplum/gpdata/primary/gpseg4
   15 |       4 | m    | m              | s    | u      | 56001 | data03   | data03  | /greenplum/gpdata/mirror/gpseg4
    7 |       5 | p    | p              | s    | u      | 55002 | data02   | data02  | /greenplum/gpdata/primary/gpseg5
   16 |       5 | m    | m              | s    | u      | 56002 | data03   | data03  | /greenplum/gpdata/mirror/gpseg5
    8 |       6 | p    | p              | s    | u      | 55000 | data03   | data03  | /greenplum/gpdata/primary/gpseg6
   17 |       6 | m    | m              | s    | u      | 56000 | data01   | data01  | /greenplum/gpdata/mirror/gpseg6
    9 |       7 | p    | p              | s    | u      | 55001 | data03   | data03  | /greenplum/gpdata/primary/gpseg7
   18 |       7 | m    | m              | s    | u      | 56001 | data01   | data01  | /greenplum/gpdata/mirror/gpseg7
   10 |       8 | p    | p              | s    | u      | 55002 | data03   | data03  | /greenplum/gpdata/primary/gpseg8
   19 |       8 | m    | m              | s    | u      | 56002 | data01   | data01  | /greenplum/gpdata/mirror/gpseg8
   21 |       9 | p    | p              | s    | u      | 55000 | data04   | data04  | /greenplum/gpdata/primary/gpseg9
   30 |       9 | m    | m              | s    | u      | 56000 | data05   | data05  | /greenplum/gpdata/mirror/gpseg9
   22 |      10 | p    | p              | s    | u      | 55001 | data04   | data04  | /greenplum/gpdata/primary/gpseg10
   31 |      10 | m    | m              | s    | u      | 56001 | data05   | data05  | /greenplum/gpdata/mirror/gpseg10
   23 |      11 | p    | p              | s    | u      | 55002 | data04   | data04  | /greenplum/gpdata/primary/gpseg11
   32 |      11 | m    | m              | s    | u      | 56002 | data05   | data05  | /greenplum/gpdata/mirror/gpseg11
   24 |      12 | p    | p              | s    | u      | 55000 | data05   | data05  | /greenplum/gpdata/primary/gpseg12
   27 |      12 | m    | m              | s    | u      | 56000 | data04   | data04  | /greenplum/gpdata/mirror/gpseg12
   25 |      13 | p    | p              | s    | u      | 55001 | data05   | data05  | /greenplum/gpdata/primary/gpseg13
   28 |      13 | m    | m              | s    | u      | 56001 | data04   | data04  | /greenplum/gpdata/mirror/gpseg13
   26 |      14 | p    | p              | s    | u      | 55002 | data05   | data05  | /greenplum/gpdata/primary/gpseg14
   29 |      14 | m    | m              | s    | u      | 56002 | data04   | data04  | /greenplum/gpdata/mirror/gpseg14
   33 |      15 | p    | p              | s    | u      | 55003 | data01   | data01  | /greenplum/gpdata/primary/gpseg15
   39 |      15 | m    | m              | s    | u      | 56003 | data02   | data02  | /greenplum/gpdata/mirror/gpseg15
   34 |      16 | p    | p              | s    | u      | 55003 | data02   | data02  | /greenplum/gpdata/primary/gpseg16
   40 |      16 | m    | m              | s    | u      | 56003 | data03   | data03  | /greenplum/gpdata/mirror/gpseg16
   35 |      17 | p    | p              | s    | u      | 55003 | data03   | data03  | /greenplum/gpdata/primary/gpseg17
   41 |      17 | m    | m              | s    | u      | 56003 | data04   | data04  | /greenplum/gpdata/mirror/gpseg17
   36 |      18 | p    | p              | s    | u      | 55003 | data04   | data04  | /greenplum/gpdata/primary/gpseg18
   42 |      18 | m    | m              | s    | u      | 56003 | data05   | data05  | /greenplum/gpdata/mirror/gpseg18
   37 |      19 | p    | p              | s    | u      | 55003 | data05   | data05  | /greenplum/gpdata/primary/gpseg19
   38 |      19 | m    | m              | s    | u      | 56003 | data01   | data01  | /greenplum/gpdata/mirror/gpseg19

查看数据：

[gpadmin@master ~]$ psql -c "select gp_segment_id,count(*) from test_yw;"

同样可以看到所有数据节点上的数据都是正常的。

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

云数据库 MySQL

原创声明：本文系作者授权腾讯云开发者社区发表，未经许可，不得转载。

如有侵权，请联系 cloudcommunity@tencent.com 删除。

云数据库 MySQL

#分布式数据库

#GreenPlum

登录后参与评论

暂无评论

IP行业查询API：为用户分析提供帮助

api

在数字化时代，IP地址不仅代表着设备在互联网上的位置，还蕴含着丰富的信息。IP地址所属行业查询API应运而生，为用户分析提供了有力支持。本文将探讨这一工具的应用，以及对用户分析的帮助。

用户10428865

2023/09/27

2630

IP行业API助力于网络分析和数据挖掘

api

在当今数字化时代，数据成为了企业、科研机构和政府决策者的重要资源，而IP行业API则成为了数据分析及挖掘的工具之一。IP行业API是一种能够查询IP地址所属的行业分类信息的应用程序接口，它能够提供在网络分析、用户行为分析及大数据挖掘等领域的优秀性能。

用户10428865

2023/11/10

1890

“留量运营”的本质是用户运营，如何搭建用户“留量”体系？

企业数字营销解决方案

随着流量红利的消失殆尽，新用户增长速率变缓，企业处于激烈的存量争夺阶段，这也使得精准定位的互联网流量费用水涨船高，获客成本居高不下，且难以量化ROI。

盈鱼MA

2020/12/25

1K0

IP代理识别API：预防欺诈和保护网络安全的必要工具

api

随着互联网的快速发展，我们的生活变得越来越依赖于网络。然而，随着网络的发展，网络犯罪和网络欺诈也在不断增加。为了保护自己的网站和客户免受网络欺诈的侵害，许多企业和组织开始使用IP代理识别API作为一种必要工具。

用户10428865

2023/11/08

2250

保障网络安全：IP代理识别API的作用与应用

api tcp/ip

随着互联网的不断发展，网络安全问题已经变得愈发重要。在网络上，恶意用户可以利用IP代理隐藏其真实身份，从而发动各种网络攻击或欺诈行为。为了保障网络安全，IP代理识别API成为了一种不可或缺的工具，本文将深入探讨其作用与应用。

用户10428865

2023/09/18

2260

IP应用场景查询API：深入了解网络用户行为的利器

api

随着数字时代的不断发展，互联网已经成为人们生活的重要组成部分。而随着越来越多的业务和社交活动迁移到在线平台上，了解和理解网络用户行为变得至关重要。为了满足这个需求，IP 应用场景查询 API 崭露头角，成为深入了解网络用户行为的利器。

用户10428865

2023/09/07

3460

ip地址查询主要可以得出哪些信息

tcp/ip

对于手机或者计算机设备在运行的过程中，都是可以通过ip地址查询具体位置，还可以得出相应的信息。因此，查询的也是比较细致的，在每一个方面的信息上会介绍得比较全面一点。接下来小编为你介绍通过查询可以直接得出哪些信息呢？

用户8739990

2021/06/18

11K0

IP归属地的应用场景有哪些?

ipv6 cdn tcp/ip

IP归属地查询在各大行业当中的利用率可以说非常的高了，提供了各种的保障，比如安全保障、数据保障、性能保障等等。今天我就来详细说一说IP归属地的应用场景有哪些？

不是海碗

2023/01/29

1K0

一文带你看透IP归属地

ipv6 tcp/ip cdn

IP是指网络之间互连的协议，英文全称是 Internet Protocol，是TCP/IP体系中的网络层协议。设计IP的目的是提高网络的可扩展性：一是解决互联网问题，实现大规模、异构网络的互联互通；二是分割顶层网络应用和底层网络技术之间的耦合关系，以利于两者的独立发展。根据端到端的设计原则，IP只为主机提供一种无连接、不可靠的、尽力而为的数据包传输服务。IP又分为IPv4和IPv6。

不是海碗

2023/01/10

3.1K0

深度解析IP应用场景API：提升风险控制与反欺诈能力

api

在当今数字化时代，网络安全和用户数据保护成为企业日益关注的焦点。IP应用场景API作为一种强大的工具，不仅能够在线调用接口获取IP场景属性，而且具备识别IP真人度的能力，为企业提供了卓越的风险控制和反欺诈业务能力。本文将深度解析IP应用场景API，揭示其在提升安全性和业务可靠性方面的重要作用。

用户10428865

2023/12/06

3610

如何使用IP归属地查询API加强网络安全

api

在当今数字化时代，网络安全对于个人和组织来说至关重要。恶意网络活动的威胁不断增加，因此采取有效的措施来加强网络安全至关重要。其中之一是利用IP归属地查询API。这个工具可以为您的网络安全策略提供宝贵的信息，帮助您更好地保护自己的在线资产。

用户10428865

2023/10/30

3400

大数据和人工智能如何赋能保险行业？

大数据人工智能

科技对产业的变革将会越来越明显，越来越深刻，国务院在“十三五”国家信息化规划中也提到了云计算、大数据、人工智能、区块链等新技术对产业变革的重要性。保险这一古老的行业能否借助科技的浪潮实现豹变呢?显然是

挖掘大数据

2017/12/20

1.8K0

如何使用IP归属地查询API来追踪网络活动

api

在当今数字化世界中，了解网络活动的源头和位置对于网络安全、市场研究和用户体验至关重要。IP归属地查询API是一种强大的工具，可以帮助您追踪网络活动并获取有关IP地址的重要信息。本文将探讨如何使用IP归属地查询API来追踪网络活动，以及它在不同领域的实际应用。

用户10428865

2023/09/19

5960

美图GeoIP服务实践（文末赠书券）

tcp/ip

3.用户画像及其他在线业务类：如美拍账号经济体系需要检测校验用户的常用登陆地信息。

美图数据技术团队

2018/12/27

1.1K0

如何利用大数据进行预测性营销，看完这8个深度解答你就懂了

大数据数据分析

1、什么是精准营销，预测营销？说到精准营销，不得不先提个性化的用户画像，我们针对每一类数据实体，进一步分解可落地的数据维度，刻画他／她的每一个特征，在聚集起来形成人群画像。比如下面就是两种典型年轻消

BestSDK

2018/02/28

8.5K0

Web to App营销策略与openinstall的融合应用

app

Web to App是移动互联网时代的精准获客策略，其核心在于通过网页端构建"场景化引流-无缝跳转-精准追踪"的闭环。与传统的应用商店推广相比，Web to App更强调跨平台用户旅程的连贯性——用户在网页端被商品/内容吸引后，无需中断体验即可直达App内指定场景。

openinstall

2025/04/27

1270

大数据应用案例排行榜TOP100解读

大数据存储

当下，“大数据”几乎是每个IT人都在谈论的一个词汇，不单单是时代发展的趋势，也是革命技术的创新。大数据对于行业的用户也越来越重要。掌握了核心数据，不单单可以进行智能化的决策，还可以在竞争激烈的行业当中脱颖而出，所以对于大数据的战略布局让越来越多的企业引起了重视，并重新定义了自己的在行业的核心竞争。当然，越来越接地气越来越实为要。在当前的互联网领域，大数据的应用已十分广泛，尤其以企业为主，企业成为大数据应用的主体。大数据真能改变企业的运作方式吗？答案毋庸置疑是肯定的。随着企业开始利用大数据，我们每天都会

小莹莹

2018/04/23

4.5K0

Teradata：数字银行的根本是如何利用技术来实时掌控和分析数据

大数据微服务

两会期间，中国建设银行行长张建国委员半开玩笑地说：“银行也是弱势群体啊。”当时包含李克强总理在内的所有人都笑了。但仔细品味，“弱势”一词固然指的是利率市场化改革对金融的影响，另一方面也折射出互联网金融等创新模式给金融产业转型带来的压力。在银行的发展中，ATM、实体分支银行、呼叫中心和数字化银行代表了四类渠道。而从不同行业的对比来看，尽管传统观点总是认为由于种种因素，银行对新技术的采用比较慢，但实际上，却远超其他行业的速度。尤其是在中国。“从新技术采用程度来看，数字化银行增速猛，目前已经超过其他渠道的总和。

CSDN技术头条

2018/02/09

8870

【防止恶意用户注册】-- 手机在网状态 API 的防欺诈应用解析

api

手机在网状态 API 支持传入手机号码，查询手机号在网状态，返回在网、在网不可用、不在网（销号/未启用/停机）等多种状态，查询手机号在网状态之后，可以根据具体的业务需求来进行不同的处理。

不是海碗

2023/04/13

1K0

如何选择口子查、站点查、渠道查？解决使用中遇到的IP地址问题

互联网

在互联网时代，数据分析和市场调查是企业和组织获取用户反馈和市场信息的重要手段。在数据收集和分析过程中，口子查、站点查和渠道查是常用的工具。本文将介绍这三种工具的优缺点，如何选择使用，以及使用过程中可能遇到的IP地址问题和解决方案。

互联网的鱼

2023/05/04

2.4K0