Search code examples
metricsambarihdp

Ambari metrics not show metrics after Cleaning up Ambari Metrics System Data

we have ambari with HDP version 2.6.5

we want to clean all metrics data , according to the following instructions on link - https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

so we did the following

note - Metrics Service operation mode - distributed

we stop the metrics service from ambari

we clean all data: ( from hdfs )

hdfs dfs -rm -r -f  /apps/ams/metrics/*

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/.tmp' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/.tmp

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/MasterProcWALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/MasterProcWALs

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/WALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/WALs

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/archive' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/archive

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/data' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/data

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/hbase.id' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/hbase.id

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/hbase.version' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/hbase.version

20/02/13 06:10:01 INFO fs.TrashPolicyDefault: Moved: 'hdfs://hdfsha/apps/ams/metrics/oldWALs' to trash at: hdfs://hdfsha/user/hdfs/.Trash/Current/apps/ams/metrics/oldWALs

And we clean also the following folders

ls /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/

ls /var/lib/ambari-metrics-collector/hbase-tmp/phoenix-spool/

We start the metrics services from ambari

But from ambari metrics graphs not appears , and metrics collector service have alert

enter image description here

Not clearly why metrics not created after full metrics cleaning ,

From the log we can see the following:

2020-02-13 06:15:33,024 INFO  [ProcedureExecutorThread-5] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=6 owner=ams state=ROLLEDBACK exec-time=239msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG

2020-02-13 06:15:44,356 INFO  [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.

2020-02-13 06:16:21,223 INFO  [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the SYSTEM.CATALOG table fails

2020-02-13 06:16:21,236 INFO  [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {PRIORITY => '2000', coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}

2020-02-13 06:16:21,349 INFO  [ProcedureExecutorThread-6] procedure.CreateTableProcedure: CreateTableProcedure (table=SYSTEM.CATALOG) id=7 owner=ams state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION

2020-02-13 06:16:21,360 WARN  [ProcedureExecutorThread-6] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.

2020-02-13 06:16:21,652 INFO  [ProcedureExecutorThread-6] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=7 owner=ams state=ROLLEDBACK exec-time=305msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG

2020-02-13 06:17:14,354 INFO  [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.

2020-02-13 06:17:58,076 INFO  [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null List Table Descriptor for the SYSTEM.CATALOG table fails

2020-02-13 06:17:58,093 INFO  [RpcServer.FifoWFPBQ.default.handler=28,queue=1,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {PRIORITY => '2000', coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}

2020-02-13 06:17:58,206 INFO  [ProcedureExecutorThread-7] procedure.CreateTableProcedure: CreateTableProcedure (table=SYSTEM.CATALOG) id=8 owner=ams state=RUNNABLE execute state=CREATE_TABLE_PRE_OPERATION

2020-02-13 06:17:58,218 WARN  [ProcedureExecutorThread-7] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.

2020-02-13 06:17:58,484 INFO  [ProcedureExecutorThread-7] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=8 owner=ams state=ROLLEDBACK exec-time=279msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG

2020-02-13 06:19:24,358 INFO  [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.

2020-02-13 06:19:34,540 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=156.56 KB, freeSize=147.69 MB, max=147.84 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=30, evicted=0, evictedPerRun=0.0
Solution

  • Did you check the Ambari-server.log. You can definitely find something there.