Open Source Consulting, profile picture
Uploaded byOpen Source Consulting
2,308 views

Ceph issue 해결 사례

The document discusses an incident involving Ceph, an open-source distributed storage system, where an OSD (Object Storage Daemon) reached 90% capacity, causing read/write issues. It details the troubleshooting steps taken to resolve the problem, including monitoring the Ceph cluster health and eventually upgrading to Ceph 3.1 to fix underlying bugs. The recovery process involved reconfiguring virtual machines and addressing data inconsistencies.

Embed presentation

Downloaded 23 times
오픈인프라데이 Ceph issue 사례 2019.07.11 오픈소스컨설팅 이 영주 ( yjlee@osci.kr )
Contents 01. 구성도 02. Issue 발생 03. 해결 과정
구성도 01
01. 구성도 ● 전체 구성도 Controller Node Compute NodeStorage Node Deploy FireWall Router
5/26 01. 구성도 ● Ceph 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3
6/26 01. 구성도 ● Ceph OBJ 흐름 PG: Placement Group Object를 저장하기 위한 OSD의 group. 복제본 수에 맞춰 member의 수가 달라짐. OSD: Object Storage Daemon object를 최종 저장하는 곳 Monitor: ceph OSD의 변화를 monitoring 하여 crush map을 만드는 주체 주체. [root@ceph-osd01 ~]# rados ls -p vms rbd_data.1735e637a64d5.0000000000000000 rbd_header.1735e637a64d5 rbd_directory rbd_children rbd_info rbd_data.1735e637a64d5.0000000000000003 rbd_data.1735e637a64d5.0000000000000002 rbd_id.893f4f3d-f6d9-4521-997c-72caa861ac24_disk rbd_data.1735e637a64d5.0000000000000001 rbd_object_map.1735e637a64d5 [root@ceph-osd01 ~]# OBJ의 기본 크기는 크기는 주체4MBMB CRUSH: Controlled Replication Under Scalable Hashing Object를 분산 저장하기위한 알고리즘.
Issue 발생 02
02. Issue 발생 Ceph 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 OSD 중 1개가 90%가 되어 Read/Write가 안되는 주체 상태 [root@osc-ceph01 ~]# ceph pg dump |grep -i full_ratio dumped all in format plain full_ratio 0.9 nearfull_ratio 0.8 [root@osc-ceph01 ~]# ceph daemon mon.`hostname` config show |grep -i osd_full_ratio "mon_osd_full_ratio": "0.9", [root@osc-ceph01 ~]#
02. Issue 발생 - Ceph community Trouble shooting guide 참조 : http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space
02. Issue 발생 Ceph 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 osd.8의 pg 1.11f를 삭제
02. Issue 발생 [root@ceph-mon02 ~]# ceph -s cluster f5078395-0236-47fd-ad02-8a6daadc7475 health HEALTH_ERR 1 pgs are stuck inactive for more than 300 seconds 162 pgs backfill_wait 37 pgs backfilling 322 pgs degraded 1 pgs down 2 pgs peering 4 pgs recovering 119 pgs recovery_wait 1 pgs stuck inactive 322 pgs stuck unclean 199 pgs undersized recovery 592647/43243812 objects degraded (1.370%) recovery 488046/43243812 objects misplaced (1.129%) 1 mons down, quorum 0,2 ceph-mon01,ceph-mon03 monmap e1: 3 mons at {ceph-mon01=10.10.50.201:6789/0,ceph-mon02=10.10.50.202:6789/0,ceph-mon03=10.10.50.203:6789/0} election epoch 480, quorum 0,2 ceph-mon01,ceph-mon03 osdmap e27606: 128 osds: 125 up, 125 in; 198 remapped pgs flags sortbitwise pgmap v58287759: 10240 pgs, 4 pools, 54316 GB data, 14076 kobjects 157 TB used, 71440 GB / 227 TB avail 592647/43243812 objects degraded (1.370%) 488046/43243812 objects misplaced (1.129%) 9916 active+clean 162 active+undersized+degraded+remapped+wait_backfill 119 active+recovery_wait+degraded 37 active+undersized+degraded+remapped+backfilling 4 active+recovering+degraded 1 down+peering 1 peering 300초 넘게 통신이 안되는 pg가 1개 (1.11f) ... osd가 down되어 backfill을 기다리고 있는 pg가 162개 pglog의 범위를 벗어나 backfill을 진행 하고 있는 pg가 37개 3copy를 채우지 못해 성능이 떨어진 pg가 322개 문제의 down된 pg 1개... (1.11f) ) 상태를 결정중인 pg 2개 (recovery, backfill) recovery를 기다리고 있는 pg 119개 pglog를 보고 복구중인 pg 4개 (해당 pg I/O block됨) up상태의 osd가 없어서 inactive 된 pg 1개 (1.11f) 3벌 복제에 못미치는 pg가 322개 pool의 복제본 수에 못미치는 pg가 199개 Monitor 1개 죽음 pg 1.11f를 갖고 있는 OSD 3개 죽음
02. Issue 발생 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 Images Pool Openstack image들이 들어가 있음. Volumes Pool pg 1.11f는 모든 openstack volume들의 정보를 조금씩 갖고 있음. pg 1개가 down되면 해당 pool의 모든 data들을 쓸 수가 없다. [root@osc-ceph01 ~]# ceph pg dump |head dumped all in format plain ... pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 1.11f 0 0 0 0 0 0 3080 3080 active+clean 2019-07-10 08:12:46.623592 921'8580 10763:10591 [8,4,7] 8 [8,4,7] 8 921'8580 2019-07-10 08:12:46.623572 921'8580 2019-07-07 19:44:32.652377 ... Primary pg가 모든 I/O를 책임진다.
해결 과정 03
03. 해결 과정 writeout_from: 30174'649621, trimmed: -1> 2018-10-24 15:28:44.487997 7fb622e2d700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 30174'593316, trimmed: 0> 2018-10-24 15:28:44.502006 7fb61de23700 -1 osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const hobject_t&, const std::set<snapid_t>&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7fb61de23700 time 2018-10-24 15:28:44.497739 osd/SnapMapper.cc: 228: FAILED assert(r == -2) 분석 결과... 3벌 복제된 pg간 충돌이나서 해당 pg를 갖고 있는 osd가 down된다. 이것은 redhat ceph 3.1(luminous)에서 fix되었으니 upgrade를 해라!! 그러나... - Redhat Openstack 9(Mitaka)는 Redhat ceph 3.1을 지원 안함. - Redhat ceph 3.1로 upgrade하기전에 openstack을 10(Newton)까지 upgrade 필요. - Redhat openstack 9는 TripleO로 되어져 있음. (Upgrade process가 굉장히 복잡함...) - Redhat ceph upgrade 시 Error상태에서 해야 함. - 렁나ㅣ러아니ㅓㄹ아ㅣㄴ;ㅓㄹ아ㅣ;ㄴ며랴ㅓ야냋
03. 해결 과정 Openstack upgrade - 실패... - 재설치 후 모든 후 모든 모든 vm 복구 Ceph 3.1 upgrade - ceph ansible을 사용하지 않고 않고 manualy upgrade 함.
03. 해결 과정 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 vms Pool Nova에 의해 생성되는 vm image를 저장 12345_disk 기존 vm rbd 67890_disk 신규 vm rbd 신규 VM1 ID=67890 기존 VM1 ID=12345 복구 과정 - 신규 vm생성 (ID 67890) - vms pool에 있는 rbd 67890_disk삭제 - 12345_disk를 67890_disk로 이름변경 - 이걸 모든vm에 적용... [root@ceph01 ~]# rbd list -p vms 12345_disk 67890_disk [root@ceph01 ~]# rbd rm -p vms 67890_disk Removing image: 100% complete...done. [root@ceph01 ~]# [root@ceph01 ~]# rbd mv -p vms 12345_disk 67890_disk [root@ceph01 ~]# rbd ls -p vms 67890_disk
03. 해결 과정 Redhat Ceph 3.1 upgrade 후 ... - 비슷한 문제 발생 - pg 1.11f 를 갖고 있는 osd들이 up down을 반복 함. [root@ceph-mon01 osc]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR noscrub,nodeep-scrub flag(s) set 5 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in flags noscrub,nodeep-scrub data: pools: 4 pools, 10240 pgs objects: 12200k objects, 46865 GB usage: 137 TB used, 97628 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+inconsistent io: client: 0 B/s rd, 1232 kB/s wr, 19 op/s rd, 59 op/s wr [root@ceph-mon01 osc]# ceph health detail HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 5 scrub errors; Possible data damage: 1 pg inconsistent OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set OSD_SCRUB_ERRORS 5 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.11f is active+clean+inconsistent, acting [113,105,10] [root@ceph-mon01 osc]# OTL...
03. 해결 과정 하지만 문제되는 Object를 특정지을 수 있었음. [root@ceph-mon01 ~]# rados list-inconsistent-obj 1.11f --format=json-pretty { "epoch": 34376, "inconsistents": [ { "object": { "name": "rbd_data.39edab651c7b53.0000000000003600", "nspace": "", "locator": "",
03. 해결 과정 Object rbd_data.39edab651c7b53.0000000000003600는 고객 DB Service vm의 root filesystem volume이었음. 다행이도 DB data에는 문제가 없었고... 문제가 된 DB vm의 root filesystem을 담고 있는 RBD image를 삭제 함. 하지만 여전히 상태는 HEALTH_ERR ... [root@ceph-mon01 osc]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR 4 scrub errors Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in data: pools: 4 pools, 10240 pgs objects: 12166k objects, 46731 GB usage: 136 TB used, 98038 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+inconsistent+snaptrim_error io: client: 0 B/s rd, 351 kB/s wr, 15 op/s rd, 51 op/s wr [root@ceph-mon01 osc]# ceph health detail HEALTH_ERR 4 scrub errors; Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error pg 1.11f is active+clean+inconsistent+snaptrim_error, acting [113,105,10] [root@ceph-mon01 osc]#
03. 해결 과정 - 문제되는 object의 snapshot id 54(0x36)이 문제가 되어서 error가 발생중임. - ??? 이미 지웠는데?? 2018-11-16 18:45:00.163319 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 10: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) 2018-11-16 18:45:00.163330 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 105: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) 2018-11-16 18:45:00.163333 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 113: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) $ printf "%dn" 0x36 54 [root@ceph-osd08 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-113 --pgid 1.11f --op list | grep 39edab651c7b53 Error getting attr on : 1.11f_head,#-3:f8800000:::scrub_1.11f:head#, (61) No data available ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":54,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}] ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":63,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}] ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":-2,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}]
03. 해결 과정 - 문제되는 object를 갖고 있는 rbd image를 찾아보자! [root@ceph-mon01 osc]# rbd info volume-13076ffc-6520-4db8-b238-1ba6108bfe52 -p volumes rbd image 'volume-13076ffc-6520-4db8-b238-1ba6108bfe52': size 53248 MB in 13312 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.62cb510d494de format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: [root@ceph-mon01 osc]# [root@ceph-mon01 osc]# cat rbd-info.sh #!/bin/bash for i in `rbd list -p volumes` do rbd info volumes/$i |grep rbd_data.39edab651c7b53 echo --- $i done ---- done [root@ceph-mon01 osc]# bash rbd-info.sh rbd info에서 object의 prefix를 볼 수 있다. 모든 rbd image에서 문제되는 object를 찾는 script [root@ceph-mon01 osc]# bash rbd-info.sh --- rbdtest done ---- --- volume-00b0de1a-bfab-40e0-a444-b6c2d0de3905 done ---- --- volume-02d9c884-fc30-4700-87fd-950855ae361d done ---- ... [root@ceph-mon01 osc]# 결과는 ... 역시나 없음...
03. 해결 과정 - 해당 snapshot을 갖고 있는 volume이 삭제 되었으니 오류에 대한 조건이 더이상 존재하지 않음. - repair를 다시 해보라고 함. [root@ceph-mon01 ~]# date ; ceph pg repair 1.11f Wed Nov 28 18:16:25 KST 2018 instructing pg 1.11f on osd.113 to repair [root@ceph-mon01 ~]# ceph health detail HEALTH_ERR noscrub,nodeep-scrub flag(s) set; Possible data damage: 1 pg repair OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set PG_DAMAGED Possible data damage: 1 pg repair pg 1.11f is active+clean+scrubbing+deep+repair, acting [113,105,10] [root@ceph-mon01 ~]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR noscrub,nodeep-scrub flag(s) set Possible data damage: 1 pg repair services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in flags noscrub,nodeep-scrub data: pools: 4 pools, 10240 pgs objects: 12321k objects, 47365 GB usage: 138 TB used, 96138 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+scrubbing+deep+repair io: client: 598 kB/s rd, 1145 kB/s wr, 18 op/s rd, 63 op/s wr pg 1.11f를 repair중
03. 해결 과정 - ceph log를 확인. [root@ceph-mon01 ~]# ceph -w ... 2018-11-28 18:21:26.654955 osd.113 [ERR] 1.11f repair stat mismatch, got 3310/3312 objects, 91/92 clones, 3243/3245 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 67/68 whiteouts, 13579894784/13584089088 bytes, 0/0 hit_set_archive bytes. 2018-11-28 18:21:26.655657 osd.113 [ERR] 1.11f repair 1 errors, 1 fixed 2018-11-28 18:19:28.979704 mon.ceph-mon01 [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg repair) 2018-11-28 18:20:30.652593 mon.ceph-mon01 [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2018-11-28 18:20:35.394445 mon.ceph-mon01 [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2018-11-28 18:20:35.394457 mon.ceph-mon01 [INF] Cluster is now healthy 어..?! fixed???
03. 해결 과정 - HEALTH_OK [root@ceph-mon01 ~]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in data: pools: 4 pools, 10240 pgs objects: 12321k objects, 47366 GB usage: 138 TB used, 96138 GB / 232 TB avail pgs: 10216 active+clean 24 active+clean+scrubbing+deep io: client: 424 kB/s rd, 766 kB/s wr, 18 op/s rd, 72 op/s wr
Q&A 오픈소스컨설팅 이 영주 ( yjlee@osci.kr )
Thank you 감사합니다 Cloud & Collaboration T. 02-516-0711 E. sales@osci.kr 서울시강남구 테헤란로83길32,5층(삼성동, 나라키움삼성동A빌딩) www.osci.kr

More Related Content

PDF
[오픈소스컨설팅]RHEL7/CentOS7 Pacemaker기반-HA시스템구성-v1.0
byJi-Woong Choi
 
PDF
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
byJo Hoon
 
PDF
[발표자료] 오픈소스 기반 클라우드 네이티브 애플리케이션 구축 방안 (feat. Kubernetes)
byOpen Source Consulting
 
PPTX
클라우드 환경을 위한 네트워크 가상화와 NSX(기초편)
byLaehyoung Kim
 
PDF
[2018] 오픈스택 5년 운영의 경험
byNHN FORWARD
 
PDF
[오픈소스컨설팅]레드햇계열리눅스7 운영자가이드 - 기초편
byJi-Woong Choi
 
PPTX
[NDC17] Kubernetes로 개발서버 간단히 찍어내기
bySeungYong Oh
 
PDF
[232] 성능어디까지쥐어짜봤니 송태웅
byNAVER D2
 
[오픈소스컨설팅]RHEL7/CentOS7 Pacemaker기반-HA시스템구성-v1.0
byJi-Woong Choi
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
byJo Hoon
 
[발표자료] 오픈소스 기반 클라우드 네이티브 애플리케이션 구축 방안 (feat. Kubernetes)
byOpen Source Consulting
 
클라우드 환경을 위한 네트워크 가상화와 NSX(기초편)
byLaehyoung Kim
 
[2018] 오픈스택 5년 운영의 경험
byNHN FORWARD
 
[오픈소스컨설팅]레드햇계열리눅스7 운영자가이드 - 기초편
byJi-Woong Choi
 
[NDC17] Kubernetes로 개발서버 간단히 찍어내기
bySeungYong Oh
 
[232] 성능어디까지쥐어짜봤니 송태웅
byNAVER D2
 

What's hot

PDF
[OpenInfra Days Korea 2018] Day 2 - CEPH 운영자를 위한 Object Storage Performance T...
byOpenStack Korea Community
 
PDF
BlueStore, A New Storage Backend for Ceph, One Year In
bySage Weil
 
PDF
Ceph scale testing with 10 Billion Objects
byKaran Singh
 
PDF
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
byOpenStack Korea Community
 
PDF
Ceph Object Storage Reference Architecture Performance and Sizing Guide
byKaran Singh
 
PDF
2021.02 new in Ceph Pacific Dashboard
byCeph Community
 
PDF
Nick Fisk - low latency Ceph
byShapeBlue
 
PDF
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
byOdinot Stanislas
 
PDF
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
byOpen Source Consulting
 
PPTX
Your 1st Ceph cluster
byMirantis
 
PDF
A crash course in CRUSH
bySage Weil
 
PDF
Ceph
byHien Nguyen Van
 
PDF
Ceph RBD Update - June 2021
byCeph Community
 
PDF
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
byJi-Woong Choi
 
PDF
Crimson: Ceph for the Age of NVMe and Persistent Memory
byScyllaDB
 
PPTX
Ceph Introduction 2017
byKaran Singh
 
PDF
2019.06.27 Intro to Ceph
byCeph Community
 
PDF
Ceph and RocksDB
bySage Weil
 
PDF
Kvm performance optimization for ubuntu
bySim Janghoon
 
PPTX
OVN - Basics and deep dive
byTrinath Somanchi
 
[OpenInfra Days Korea 2018] Day 2 - CEPH 운영자를 위한 Object Storage Performance T...
byOpenStack Korea Community
 
BlueStore, A New Storage Backend for Ceph, One Year In
bySage Weil
 
Ceph scale testing with 10 Billion Objects
byKaran Singh
 
[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화
byOpenStack Korea Community
 
Ceph Object Storage Reference Architecture Performance and Sizing Guide
byKaran Singh
 
2021.02 new in Ceph Pacific Dashboard
byCeph Community
 
Nick Fisk - low latency Ceph
byShapeBlue
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
byOdinot Stanislas
 
[오픈소스컨설팅] 쿠버네티스와 쿠버네티스 on 오픈스택 비교 및 구축 방법
byOpen Source Consulting
 
Your 1st Ceph cluster
byMirantis
 
A crash course in CRUSH
bySage Weil
 
Ceph
byHien Nguyen Van
 
Ceph RBD Update - June 2021
byCeph Community
 
[오픈소스컨설팅] Open Stack Ceph, Neutron, HA, Multi-Region
byJi-Woong Choi
 
Crimson: Ceph for the Age of NVMe and Persistent Memory
byScyllaDB
 
Ceph Introduction 2017
byKaran Singh
 
2019.06.27 Intro to Ceph
byCeph Community
 
Ceph and RocksDB
bySage Weil
 
Kvm performance optimization for ubuntu
bySim Janghoon
 
OVN - Basics and deep dive
byTrinath Somanchi
 

Similar to Ceph issue 해결 사례

PDF
[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"
byOpenStack Korea Community
 
PDF
Red Hat OpenStack 17 저자직강+스터디그룹_4주차
byNalee Jang
 
PPTX
Exadata db node update
bypat2001
 
PPTX
Exadata cell update
bypat2001
 
PPTX
Docker practice
bywonyong hwang
 
PPTX
Docker on openstack by OpenSource Consulting
byOpen Source Consulting
 
PDF
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-CRUI
byTommy Lee
 
PDF
On demand-block-storage-for-docker
byJangseon Ryu
 
PDF
[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service
byOracle Korea
 
[OpenInfra Days Korea 2018] Day 1 - T4-7: "Ceph 스토리지, PaaS로 서비스 운영하기"
byOpenStack Korea Community
 
Red Hat OpenStack 17 저자직강+스터디그룹_4주차
byNalee Jang
 
Exadata db node update
bypat2001
 
Exadata cell update
bypat2001
 
Docker practice
bywonyong hwang
 
Docker on openstack by OpenSource Consulting
byOpen Source Consulting
 
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-CRUI
byTommy Lee
 
On demand-block-storage-for-docker
byJangseon Ryu
 
[Demo session] 관리형 Kafka 서비스 - Oracle Event Hub Service
byOracle Korea
 

More from Open Source Consulting

PDF
20250821_일잘러페스타_SPARK conference_AI를 통한 업무 방식 혁신
byOpen Source Consulting
 
PDF
Playce WASup | Apache/Tomcat은 그대로, 운영은 더 강력하게
byOpen Source Consulting
 
PDF
[The Future of IT] 1. AI 시대의 일하는 방법_김대일 고문
byOpen Source Consulting
 
PDF
[Atlassian System of Work] 모든 협업을 연결하는 아틀라시안의 프레임워크
byOpen Source Consulting
 
PDF
Playce Cloud 소개자료 | 글로벌 표준과 로컬라이제이션을 위한 오픈소스 패키지
byOpen Source Consulting
 
PDF
[The Future of IT] 3. AI 시대의 인프라_오픈소스를 활용한 인프라 구축 및 GPU as a Service 구현사례_김호진 상무
byOpen Source Consulting
 
PDF
AI 동료와 협업하는 업무 시스템 활용 | Atlassian AI와 monday.com AI로 경험하는 새로운 협업 방식
byOpen Source Consulting
 
PDF
[오픈소스컨설팅] 오픈소스로 확장하는 NKP 기반 Kubernetes 환경
byOpen Source Consulting
 
PDF
[오픈소스컨설팅] 먼데이닷컴 소개서
byOpen Source Consulting
 
PDF
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 04. Rovo가 적용된 ITSM 시스템 ᄃ...
byOpen Source Consulting
 
PDF
[Recap] Atlassian Team25 - Progress is impossible alone
byOpen Source Consulting
 
PDF
[The Future of IT] 2. 조직 목표 Align 및 전사 협업 확장_한진규 부사장
byOpen Source Consulting
 
PDF
[Team on Tour 2025] Atlassian의 System of Work
byOpen Source Consulting
 
PDF
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 02. 클라우드 기반 ITSM 최적화를...
byOpen Source Consulting
 
PDF
[워크숍] Get to know AI, Meet your new teammate!
byOpen Source Consulting
 
PDF
[202412 SAFe Meetup] SAFe Transfomration Journey
byOpen Source Consulting
 
PDF
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 01. Gen Next ITSM_김윤희.pdf
byOpen Source Consulting
 
PDF
[발표 자료] 시각화로 전략을 실현하는 스마트 워크플로우 with 먼데이닷컴.pdf
byOpen Source Consulting
 
PDF
Good-bye Digital Transformation, Hello AI-first Transformation
byOpen Source Consulting
 
PDF
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 03. ITSM 구현과 고객 사례_신철ᄆ...
byOpen Source Consulting
 
20250821_일잘러페스타_SPARK conference_AI를 통한 업무 방식 혁신
byOpen Source Consulting
 
Playce WASup | Apache/Tomcat은 그대로, 운영은 더 강력하게
byOpen Source Consulting
 
[The Future of IT] 1. AI 시대의 일하는 방법_김대일 고문
byOpen Source Consulting
 
[Atlassian System of Work] 모든 협업을 연결하는 아틀라시안의 프레임워크
byOpen Source Consulting
 
Playce Cloud 소개자료 | 글로벌 표준과 로컬라이제이션을 위한 오픈소스 패키지
byOpen Source Consulting
 
[The Future of IT] 3. AI 시대의 인프라_오픈소스를 활용한 인프라 구축 및 GPU as a Service 구현사례_김호진 상무
byOpen Source Consulting
 
AI 동료와 협업하는 업무 시스템 활용 | Atlassian AI와 monday.com AI로 경험하는 새로운 협업 방식
byOpen Source Consulting
 
[오픈소스컨설팅] 오픈소스로 확장하는 NKP 기반 Kubernetes 환경
byOpen Source Consulting
 
[오픈소스컨설팅] 먼데이닷컴 소개서
byOpen Source Consulting
 
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 04. Rovo가 적용된 ITSM 시스템 ᄃ...
byOpen Source Consulting
 
[Recap] Atlassian Team25 - Progress is impossible alone
byOpen Source Consulting
 
[The Future of IT] 2. 조직 목표 Align 및 전사 협업 확장_한진규 부사장
byOpen Source Consulting
 
[Team on Tour 2025] Atlassian의 System of Work
byOpen Source Consulting
 
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 02. 클라우드 기반 ITSM 최적화를...
byOpen Source Consulting
 
[워크숍] Get to know AI, Meet your new teammate!
byOpen Source Consulting
 
[202412 SAFe Meetup] SAFe Transfomration Journey
byOpen Source Consulting
 
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 01. Gen Next ITSM_김윤희.pdf
byOpen Source Consulting
 
[발표 자료] 시각화로 전략을 실현하는 스마트 워크플로우 with 먼데이닷컴.pdf
byOpen Source Consulting
 
Good-bye Digital Transformation, Hello AI-first Transformation
byOpen Source Consulting
 
[Next-Gen ITSM | Atlassian이 제시하는 ITSM 혁신] 03. ITSM 구현과 고객 사례_신철ᄆ...
byOpen Source Consulting
 

Recently uploaded

PDF
The Ultimate Serial Port Detective – Baudowl
byCysinfo Cyber Security Community
 
PDF
AI/ML Infra Meetup | Bringing Data to GPUs Anywhere + Get Low-Latency on Obje...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
PDF
Combinatorial Interview Problems with Backtracking Solutions - From Imperativ...
byPhilip Schwarz
 
PPTX
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
PDF
SCORM Cloud: The 5 categories of content distribution
byRustici Software
 
PDF
DSD-INT 2025 Thermal and chemical plumes of sea cooling water from PEM platfo...
byDeltares
 
PDF
NYC ACE 12-Nov-2025-Combined Presentation.pdf
byAUGNYC
 
PDF
Startup Core Architecture: A Simple Practical Guide
byShiv Technolabs Pvt. Ltd.
 
PDF
SWEBOK: the software engineering body of knowledge (SC25 Workshop Research So...
byHironori Washizaki
 
PDF
DSD-INT 2025 From Software to Impact - Water Quality Modelling for the UN Oce...
byDeltares
 
PPTX
Short Film Script Development Process HARMONIA
byzdolezalmedia
 
PDF
DSD-INT 2025 3D Modeling of Shallow Water Dynamics Using Delft3D FM - Martins
byDeltares
 
DOCX
Open Source BI Tableau Alternatives You Can Trust.docx
byVarsha Nayak
 
PDF
The Evolution of Project Portfolio Management - What Modern PMOs Are Doing Di...
byOnePlan Solutions
 
PDF
AOX Apps - Mobile App Development Company New York
byAOX APPS
 
PDF
Threat Hunting with Garuda: From Manual Analysis to AI-Driven Hunting By Monn...
byCysinfo Cyber Security Community
 
PPTX
Enhance Workplace Safety with EHA Good Catch System
byEHA Soft Solutions
 
PPTX
BI and Reporting Software Turning Clinic Data Into Smarter, More Compassionat...
byEasy Clinic
 
PPTX
Harnessing BUSY Software for Smart Business Management.pptx
byMoving Cloud
 
The Ultimate Serial Port Detective – Baudowl
byCysinfo Cyber Security Community
 
AI/ML Infra Meetup | Bringing Data to GPUs Anywhere + Get Low-Latency on Obje...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
Combinatorial Interview Problems with Backtracking Solutions - From Imperativ...
byPhilip Schwarz
 
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
SCORM Cloud: The 5 categories of content distribution
byRustici Software
 
DSD-INT 2025 Thermal and chemical plumes of sea cooling water from PEM platfo...
byDeltares
 
NYC ACE 12-Nov-2025-Combined Presentation.pdf
byAUGNYC
 
Startup Core Architecture: A Simple Practical Guide
byShiv Technolabs Pvt. Ltd.
 
SWEBOK: the software engineering body of knowledge (SC25 Workshop Research So...
byHironori Washizaki
 
DSD-INT 2025 From Software to Impact - Water Quality Modelling for the UN Oce...
byDeltares
 
Short Film Script Development Process HARMONIA
byzdolezalmedia
 
DSD-INT 2025 3D Modeling of Shallow Water Dynamics Using Delft3D FM - Martins
byDeltares
 
Open Source BI Tableau Alternatives You Can Trust.docx
byVarsha Nayak
 
The Evolution of Project Portfolio Management - What Modern PMOs Are Doing Di...
byOnePlan Solutions
 
AOX Apps - Mobile App Development Company New York
byAOX APPS
 
Threat Hunting with Garuda: From Manual Analysis to AI-Driven Hunting By Monn...
byCysinfo Cyber Security Community
 
Enhance Workplace Safety with EHA Good Catch System
byEHA Soft Solutions
 
BI and Reporting Software Turning Clinic Data Into Smarter, More Compassionat...
byEasy Clinic
 
Harnessing BUSY Software for Smart Business Management.pptx
byMoving Cloud
 
In this document
Powered by AI

Introduction to the presentation titled 'Open Infrastructure Day: Ceph Issue Case Study' by YJ Lee. Overview includes a table of contents with sections on configuration, issue occurrence, and resolution.

Overview of the Ceph architecture, including various nodes such as Controller Node, Compute Node, and Storage Node. It also describes Object Storage Daemon (OSD) and Placement Group (PG) in data storage.

Discussion of an issue where one OSD reaches 90% capacity causing read/write failures. References Ceph community troubleshooting guidelines and error statuses from the command line.Detailed examination of the consequences of the issue, including clusters stuck inactive, being degraded, and related pool bottleneck information.

The resolution process involves upgrading Red Hat Ceph 3.1, manual upgrades, identifying problematic images, and attempts to restore services by repairing damaged object pools.

Follow-up concerns after upgrading Ceph; resolving object inconsistencies and snapshots led to restoring cluster health through repair commands.

Final segment includes a Q&A session and conclusion with contact details for further inquiries related to the presentation.

Ceph issue 해결 사례

  • 1.
  • 2.
    Contents 01. 구성도 02. Issue발생 03. 해결 과정
  • 3.
  • 4.
    01. 구성도 ● 전체 구성도 ControllerNode Compute NodeStorage Node Deploy FireWall Router
  • 5.
    5/26 01. 구성도 ● Ceph 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3
  • 6.
    6/26 01. 구성도 ● Ceph OBJ흐름 PG: Placement Group Object를 저장하기 위한 OSD의 group. 복제본 수에 맞춰 member의 수가 달라짐. OSD: Object Storage Daemon object를 최종 저장하는 곳 Monitor: ceph OSD의 변화를 monitoring 하여 crush map을 만드는 주체 주체. [root@ceph-osd01 ~]# rados ls -p vms rbd_data.1735e637a64d5.0000000000000000 rbd_header.1735e637a64d5 rbd_directory rbd_children rbd_info rbd_data.1735e637a64d5.0000000000000003 rbd_data.1735e637a64d5.0000000000000002 rbd_id.893f4f3d-f6d9-4521-997c-72caa861ac24_disk rbd_data.1735e637a64d5.0000000000000001 rbd_object_map.1735e637a64d5 [root@ceph-osd01 ~]# OBJ의 기본 크기는 크기는 주체4MBMB CRUSH: Controlled Replication Under Scalable Hashing Object를 분산 저장하기위한 알고리즘.
  • 7.
  • 8.
    02. Issue 발생 Ceph구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 OSD 중 1개가 90%가 되어 Read/Write가 안되는 주체 상태 [root@osc-ceph01 ~]# ceph pg dump |grep -i full_ratio dumped all in format plain full_ratio 0.9 nearfull_ratio 0.8 [root@osc-ceph01 ~]# ceph daemon mon.`hostname` config show |grep -i osd_full_ratio "mon_osd_full_ratio": "0.9", [root@osc-ceph01 ~]#
  • 9.
    02. Issue 발생 -Ceph community Trouble shooting guide 참조 : http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/#no-free-drive-space
  • 10.
    02. Issue 발생 Ceph구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 osd.8의 pg 1.11f를 삭제
  • 11.
    02. Issue 발생 [root@ceph-mon02~]# ceph -s cluster f5078395-0236-47fd-ad02-8a6daadc7475 health HEALTH_ERR 1 pgs are stuck inactive for more than 300 seconds 162 pgs backfill_wait 37 pgs backfilling 322 pgs degraded 1 pgs down 2 pgs peering 4 pgs recovering 119 pgs recovery_wait 1 pgs stuck inactive 322 pgs stuck unclean 199 pgs undersized recovery 592647/43243812 objects degraded (1.370%) recovery 488046/43243812 objects misplaced (1.129%) 1 mons down, quorum 0,2 ceph-mon01,ceph-mon03 monmap e1: 3 mons at {ceph-mon01=10.10.50.201:6789/0,ceph-mon02=10.10.50.202:6789/0,ceph-mon03=10.10.50.203:6789/0} election epoch 480, quorum 0,2 ceph-mon01,ceph-mon03 osdmap e27606: 128 osds: 125 up, 125 in; 198 remapped pgs flags sortbitwise pgmap v58287759: 10240 pgs, 4 pools, 54316 GB data, 14076 kobjects 157 TB used, 71440 GB / 227 TB avail 592647/43243812 objects degraded (1.370%) 488046/43243812 objects misplaced (1.129%) 9916 active+clean 162 active+undersized+degraded+remapped+wait_backfill 119 active+recovery_wait+degraded 37 active+undersized+degraded+remapped+backfilling 4 active+recovering+degraded 1 down+peering 1 peering 300초 넘게 통신이 안되는 pg가 1개 (1.11f) ... osd가 down되어 backfill을 기다리고 있는 pg가 162개 pglog의 범위를 벗어나 backfill을 진행 하고 있는 pg가 37개 3copy를 채우지 못해 성능이 떨어진 pg가 322개 문제의 down된 pg 1개... (1.11f) ) 상태를 결정중인 pg 2개 (recovery, backfill) recovery를 기다리고 있는 pg 119개 pglog를 보고 복구중인 pg 4개 (해당 pg I/O block됨) up상태의 osd가 없어서 inactive 된 pg 1개 (1.11f) 3벌 복제에 못미치는 pg가 322개 pool의 복제본 수에 못미치는 pg가 199개 Monitor 1개 죽음 pg 1.11f를 갖고 있는 OSD 3개 죽음
  • 12.
    02. Issue 발생 구성도 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 ceph-mon1 ceph-mon2 ceph-mon3 Images Pool Openstack image들이 들어가 있음. Volumes Pool pg 1.11f는 모든 openstack volume들의 정보를 조금씩 갖고 있음. pg 1개가 down되면 해당 pool의 모든 data들을 쓸 수가 없다. [root@osc-ceph01 ~]# ceph pg dump |head dumped all in format plain ... pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 1.11f 0 0 0 0 0 0 3080 3080 active+clean 2019-07-10 08:12:46.623592 921'8580 10763:10591 [8,4,7] 8 [8,4,7] 8 921'8580 2019-07-10 08:12:46.623572 921'8580 2019-07-07 19:44:32.652377 ... Primary pg가 모든 I/O를 책임진다.
  • 13.
  • 14.
    03. 해결 과정 writeout_from:30174'649621, trimmed: -1> 2018-10-24 15:28:44.487997 7fb622e2d700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 30174'593316, trimmed: 0> 2018-10-24 15:28:44.502006 7fb61de23700 -1 osd/SnapMapper.cc: In function 'void SnapMapper::add_oid(const hobject_t&, const std::set<snapid_t>&, MapCacher::Transaction<std::basic_string<char>, ceph::buffer::list>*)' thread 7fb61de23700 time 2018-10-24 15:28:44.497739 osd/SnapMapper.cc: 228: FAILED assert(r == -2) 분석 결과... 3벌 복제된 pg간 충돌이나서 해당 pg를 갖고 있는 osd가 down된다. 이것은 redhat ceph 3.1(luminous)에서 fix되었으니 upgrade를 해라!! 그러나... - Redhat Openstack 9(Mitaka)는 Redhat ceph 3.1을 지원 안함. - Redhat ceph 3.1로 upgrade하기전에 openstack을 10(Newton)까지 upgrade 필요. - Redhat openstack 9는 TripleO로 되어져 있음. (Upgrade process가 굉장히 복잡함...) - Redhat ceph upgrade 시 Error상태에서 해야 함. - 렁나ㅣ러아니ㅓㄹ아ㅣㄴ;ㅓㄹ아ㅣ;ㄴ며랴ㅓ야냋
  • 15.
    03. 해결 과정 Openstackupgrade - 실패... - 재설치 후 모든 후 모든 모든 vm 복구 Ceph 3.1 upgrade - ceph ansible을 사용하지 않고 않고 manualy upgrade 함.
  • 16.
    03. 해결 과정 ceph-osd1 ... ceph-osd2 ... ceph-osd3 ... ...osd.0osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8 vms Pool Nova에 의해 생성되는 vm image를 저장 12345_disk 기존 vm rbd 67890_disk 신규 vm rbd 신규 VM1 ID=67890 기존 VM1 ID=12345 복구 과정 - 신규 vm생성 (ID 67890) - vms pool에 있는 rbd 67890_disk삭제 - 12345_disk를 67890_disk로 이름변경 - 이걸 모든vm에 적용... [root@ceph01 ~]# rbd list -p vms 12345_disk 67890_disk [root@ceph01 ~]# rbd rm -p vms 67890_disk Removing image: 100% complete...done. [root@ceph01 ~]# [root@ceph01 ~]# rbd mv -p vms 12345_disk 67890_disk [root@ceph01 ~]# rbd ls -p vms 67890_disk
  • 17.
    03. 해결 과정 RedhatCeph 3.1 upgrade 후 ... - 비슷한 문제 발생 - pg 1.11f 를 갖고 있는 osd들이 up down을 반복 함. [root@ceph-mon01 osc]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR noscrub,nodeep-scrub flag(s) set 5 scrub errors Possible data damage: 1 pg inconsistent services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in flags noscrub,nodeep-scrub data: pools: 4 pools, 10240 pgs objects: 12200k objects, 46865 GB usage: 137 TB used, 97628 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+inconsistent io: client: 0 B/s rd, 1232 kB/s wr, 19 op/s rd, 59 op/s wr [root@ceph-mon01 osc]# ceph health detail HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 5 scrub errors; Possible data damage: 1 pg inconsistent OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set OSD_SCRUB_ERRORS 5 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent pg 1.11f is active+clean+inconsistent, acting [113,105,10] [root@ceph-mon01 osc]# OTL...
  • 18.
    03. 해결 과정 하지만문제되는 Object를 특정지을 수 있었음. [root@ceph-mon01 ~]# rados list-inconsistent-obj 1.11f --format=json-pretty { "epoch": 34376, "inconsistents": [ { "object": { "name": "rbd_data.39edab651c7b53.0000000000003600", "nspace": "", "locator": "",
  • 19.
    03. 해결 과정 Objectrbd_data.39edab651c7b53.0000000000003600는 고객 DB Service vm의 root filesystem volume이었음. 다행이도 DB data에는 문제가 없었고... 문제가 된 DB vm의 root filesystem을 담고 있는 RBD image를 삭제 함. 하지만 여전히 상태는 HEALTH_ERR ... [root@ceph-mon01 osc]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR 4 scrub errors Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in data: pools: 4 pools, 10240 pgs objects: 12166k objects, 46731 GB usage: 136 TB used, 98038 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+inconsistent+snaptrim_error io: client: 0 B/s rd, 351 kB/s wr, 15 op/s rd, 51 op/s wr [root@ceph-mon01 osc]# ceph health detail HEALTH_ERR 4 scrub errors; Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error OSD_SCRUB_ERRORS 4 scrub errors PG_DAMAGED Possible data damage: 1 pg inconsistent, 1 pg snaptrim_error pg 1.11f is active+clean+inconsistent+snaptrim_error, acting [113,105,10] [root@ceph-mon01 osc]#
  • 20.
    03. 해결 과정 -문제되는 object의 snapshot id 54(0x36)이 문제가 되어서 error가 발생중임. - ??? 이미 지웠는데?? 2018-11-16 18:45:00.163319 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 10: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) 2018-11-16 18:45:00.163330 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 105: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) 2018-11-16 18:45:00.163333 7fb827aca700 -1 log_channel(cluster) log [ERR] : 1.11f shard 113: soid 1:f886c0a3:::rbd_data.39edab651c7b53.0000000000003600:36 data_digest 0x43d61c5d != data_digest 0x86baff34 from auth oi 1:f886c0a3::: rbd_data.39edab651c7b53.0000000000003600:36(14027'236814 osd.113.0:29524 [36] dirty|data_digest|omap_digest s 4194304 uv 235954 dd 86baff34 od ffffffff alloc_hint [0 0 0]) $ printf "%dn" 0x36 54 [root@ceph-osd08 ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-113 --pgid 1.11f --op list | grep 39edab651c7b53 Error getting attr on : 1.11f_head,#-3:f8800000:::scrub_1.11f:head#, (61) No data available ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":54,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}] ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":63,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}] ["1.11f",{"oid":"rbd_data.39edab651c7b53.0000000000003600","key":"","snapid":-2,"hash":3305333023,"max":0,"pool":1,"namespace":"","max":0}]
  • 21.
    03. 해결 과정 -문제되는 object를 갖고 있는 rbd image를 찾아보자! [root@ceph-mon01 osc]# rbd info volume-13076ffc-6520-4db8-b238-1ba6108bfe52 -p volumes rbd image 'volume-13076ffc-6520-4db8-b238-1ba6108bfe52': size 53248 MB in 13312 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.62cb510d494de format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten flags: [root@ceph-mon01 osc]# [root@ceph-mon01 osc]# cat rbd-info.sh #!/bin/bash for i in `rbd list -p volumes` do rbd info volumes/$i |grep rbd_data.39edab651c7b53 echo --- $i done ---- done [root@ceph-mon01 osc]# bash rbd-info.sh rbd info에서 object의 prefix를 볼 수 있다. 모든 rbd image에서 문제되는 object를 찾는 script [root@ceph-mon01 osc]# bash rbd-info.sh --- rbdtest done ---- --- volume-00b0de1a-bfab-40e0-a444-b6c2d0de3905 done ---- --- volume-02d9c884-fc30-4700-87fd-950855ae361d done ---- ... [root@ceph-mon01 osc]# 결과는 ... 역시나 없음...
  • 22.
    03. 해결 과정 -해당 snapshot을 갖고 있는 volume이 삭제 되었으니 오류에 대한 조건이 더이상 존재하지 않음. - repair를 다시 해보라고 함. [root@ceph-mon01 ~]# date ; ceph pg repair 1.11f Wed Nov 28 18:16:25 KST 2018 instructing pg 1.11f on osd.113 to repair [root@ceph-mon01 ~]# ceph health detail HEALTH_ERR noscrub,nodeep-scrub flag(s) set; Possible data damage: 1 pg repair OSDMAP_FLAGS noscrub,nodeep-scrub flag(s) set PG_DAMAGED Possible data damage: 1 pg repair pg 1.11f is active+clean+scrubbing+deep+repair, acting [113,105,10] [root@ceph-mon01 ~]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_ERR noscrub,nodeep-scrub flag(s) set Possible data damage: 1 pg repair services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in flags noscrub,nodeep-scrub data: pools: 4 pools, 10240 pgs objects: 12321k objects, 47365 GB usage: 138 TB used, 96138 GB / 232 TB avail pgs: 10239 active+clean 1 active+clean+scrubbing+deep+repair io: client: 598 kB/s rd, 1145 kB/s wr, 18 op/s rd, 63 op/s wr pg 1.11f를 repair중
  • 23.
    03. 해결 과정 -ceph log를 확인. [root@ceph-mon01 ~]# ceph -w ... 2018-11-28 18:21:26.654955 osd.113 [ERR] 1.11f repair stat mismatch, got 3310/3312 objects, 91/92 clones, 3243/3245 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 67/68 whiteouts, 13579894784/13584089088 bytes, 0/0 hit_set_archive bytes. 2018-11-28 18:21:26.655657 osd.113 [ERR] 1.11f repair 1 errors, 1 fixed 2018-11-28 18:19:28.979704 mon.ceph-mon01 [INF] Health check cleared: PG_DAMAGED (was: Possible data damage: 1 pg repair) 2018-11-28 18:20:30.652593 mon.ceph-mon01 [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2018-11-28 18:20:35.394445 mon.ceph-mon01 [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2018-11-28 18:20:35.394457 mon.ceph-mon01 [INF] Cluster is now healthy 어..?! fixed???
  • 24.
    03. 해결 과정 -HEALTH_OK [root@ceph-mon01 ~]# ceph -s cluster: id: f5078395-0236-47fd-ad02-8a6daadc7475 health: HEALTH_OK services: mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03 mgr: ceph-mon01(active), standbys: ceph-mon02, ceph-mon03 osd: 128 osds: 128 up, 128 in data: pools: 4 pools, 10240 pgs objects: 12321k objects, 47366 GB usage: 138 TB used, 96138 GB / 232 TB avail pgs: 10216 active+clean 24 active+clean+scrubbing+deep io: client: 424 kB/s rd, 766 kB/s wr, 18 op/s rd, 72 op/s wr
  • 25.
  • 26.
    Thank you 감사합니다 Cloud &Collaboration T. 02-516-0711 E. sales@osci.kr 서울시강남구 테헤란로83길32,5층(삼성동, 나라키움삼성동A빌딩) www.osci.kr