With the previous Jewel release I had no problems. I have created a test cluster of 5 vms, all with Centos 7 and the Nautilus release of Ceph. 1 vm is a monitor, 3 are OSDs and 1 is admin-mgr. The deployment of the cluster is OK, the health is OK, but after creating MDS and pools...
ceph -s
cluster:
id: 87c90336-38bc-4ec2-bcde-2629e1e7b12f
health: HEALTH_WARN
Reduced data availability: 42 pgs inactive, 43 pgs peering
services:
mon: 1 daemons, quorum ceph1-mon (age 8m)
mgr: ceph1-admin(active, since 8m)
mds: cephfs:1 {0=ceph1-osd=up:active} 1 up:standby
osd: 3 osds: 3 up (since 7m), 3 in (since 20h)
data:
pools: 2 pools, 128 pgs
objects: 18 objects, 2.6 KiB
usage: 2.1 GiB used, 78 GiB / 80 GiB avail
pgs: 32.812% pgs unknown
67.188% pgs not active
86 peering
42 unknown
Examining the health..
ceph health detail
HEALTH_WARN Reduced data availability: 42 pgs inactive, 43 pgs peering
PG_AVAILABILITY Reduced data availability: 42 pgs inactive, 43 pgs peering
pg 9.0 is stuck peering for 254.671721, current state peering, last acting [0,1,2]
pg 9.1 is stuck peering for 254.671732, current state peering, last acting [0,2,1]
pg 9.4 is stuck peering for 254.670850, current state peering, last acting [0,1,2]
pg 9.5 is stuck inactive for 234.575775, current state unknown, last acting []
pg 9.7 is stuck inactive for 234.575775, current state unknown, last acting []
pg 9.8 is stuck inactive for 234.575775, current state unknown, last acting []
The output is really long. A lot of PGs are inactive or peering. I have used this configuration:
#ceph.conf
[global]
fsid = 87c90336-38bc-4ec2-bcde-2629e1e7b12f
mon_initial_members = ceph1-mon
mon_host = 10.2.0.117
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
mon_allow_pool_delete = true
mon_max_pg_per_osd = 128
osd max pg per osd hard ratio = 10 # < default is 2, try to set at least 5. It will be
I have created the OSD with these commands:
ceph-deploy --overwrite-conf osd create --data /dev/vdb ceph1-osd
ceph-deploy --overwrite-conf osd create --data /dev/vdb ceph2-osd
ceph-deploy --overwrite-conf osd create --data /dev/vdb ceph3-osd
I have created the MDS with these commands:
ceph-deploy mds create ceph1-osd
ceph-deploy mds create ceph2-osd
ceph-deploy mds create ceph3-osd
For the pools and filesystem I have used these commands:
ceph osd pool create cephfs_data 64
ceph osd pool create cephfs_metadata 64
ceph fs new cephfs cephfs_metadata cephfs_data
What's wrong?
In most cases such peering/unknown PG are related to connectivity issues. Are monitor and OSDs able to reach each other? Might there be a firewall problem or some foul routing causing the issues?
Additionally the OSD and monitor logs are also worth checking out. Are there errors in the logs (most certainly)?
Checking all this will guide you to solve your problem.
See also the Ceph troubleshooting guide.