當(dang)今社會(hui)(hui)是一個(ge)數(shu)(shu)據(ju)(ju)(ju)社會(hui)(hui),從2010 年開始,全(quan)(quan)球(qiu)數(shu)(shu)據(ju)(ju)(ju)總量已經進入ZB 時(shi)代(dai)(dai),隨(sui)著視頻監(jian)控向(xiang)高清化、智能(neng)化的發展(zhan)(zhan),隨(sui)著移動(dong)互(hu)聯網和物聯網的興起(qi),由各類(lei)傳感器(qi)產生的數(shu)(shu)據(ju)(ju)(ju)量和數(shu)(shu)據(ju)(ju)(ju)種(zhong)類(lei)正在加速增(zeng)長,根據(ju)(ju)(ju)IDC 監(jian)測,全(quan)(quan)球(qiu)數(shu)(shu)據(ju)(ju)(ju)量大(da)(da)(da)約每(mei)18 個(ge)月就(jiu)會(hui)(hui)翻一番(這(zhe)大(da)(da)(da)概也可以稱之為數(shu)(shu)據(ju)(ju)(ju)摩爾定律(lv)吧),預計到(dao)2020 年,全(quan)(quan)球(qiu)數(shu)(shu)據(ju)(ju)(ju)量將(jiang)達(da)到(dao)或超過40ZB,人類(lei)社會(hui)(hui)正處于一個(ge)數(shu)(shu)據(ju)(ju)(ju)爆炸的時(shi)代(dai)(dai),或者大(da)(da)(da)數(shu)(shu)據(ju)(ju)(ju)時(shi)代(dai)(dai),對這(zhe)些海量數(shu)(shu)據(ju)(ju)(ju)的存儲、管理和利用(yong),促進了云(yun)計算技術的發展(zhan)(zhan),而(er)云(yun)存儲作為云(yun)計算概念的一種(zhong)具體(ti)表現(xian)形式(shi),也得到(dao)了極大(da)(da)(da)的發展(zhan)(zhan)。
相(xiang)對(dui)傳統(tong)(tong)存(cun)(cun)(cun)儲(chu)(chu)而(er)(er)言(yan),云存(cun)(cun)(cun)儲(chu)(chu)改變了(le)數據垂直存(cun)(cun)(cun)儲(chu)(chu)在(zai)某一(yi)臺物理設(she)備(bei)的(de)(de)存(cun)(cun)(cun)放模式(shi)(shi),通(tong)過寬帶(dai)網絡(luo)(比如萬兆以(yi)太網或Infiniband技(ji)(ji)術(shu))集合大(da)量(liang)的(de)(de)存(cun)(cun)(cun)儲(chu)(chu)設(she)備(bei),通(tong)過存(cun)(cun)(cun)儲(chu)(chu)虛(xu)擬化、分(fen)布式(shi)(shi)文件系(xi)統(tong)(tong)等技(ji)(ji)術(shu)將(jiang)位于各單(dan)(dan)一(yi)存(cun)(cun)(cun)儲(chu)(chu)設(she)備(bei)上的(de)(de)物理存(cun)(cun)(cun)儲(chu)(chu)資源(yuan)統(tong)(tong)一(yi)管(guan)理,構(gou)成邏輯上統(tong)(tong)一(yi)的(de)(de)存(cun)(cun)(cun)儲(chu)(chu)資源(yuan)池對(dui)外提供(gong)服(fu)務(wu),從(cong)而(er)(er)在(zai)存(cun)(cun)(cun)儲(chu)(chu)容量(liang)上得以(yi)從(cong)單(dan)(dan)設(she)備(bei)PB 級擴展至(zhi)數十PB乃至(zhi)數百(bai)PB,并且(qie),由于云存(cun)(cun)(cun)儲(chu)(chu)系(xi)統(tong)(tong)中的(de)(de)各節(jie)點能夠并行提供(gong)讀寫訪問服(fu)務(wu),系(xi)統(tong)(tong)性(xing)(xing)能隨著業務(wu)節(jie)點的(de)(de)增(zeng)加而(er)(er)獲(huo)得顯著提升(sheng)。同時,通(tong)過數據冗余編碼技(ji)(ji)術(shu)、業務(wu)快速(su)接管(guan)技(ji)(ji)術(shu),提供(gong)相(xiang)對(dui)傳統(tong)(tong)存(cun)(cun)(cun)儲(chu)(chu)設(she)備(bei)更高的(de)(de)系(xi)統(tong)(tong)可(ke)用性(xing)(xing)。容量(liang)和(he)性(xing)(xing)能的(de)(de)可(ke)擴展性(xing)(xing)、系(xi)統(tong)(tong)高可(ke)用性(xing)(xing),是云存(cun)(cun)(cun)儲(chu)(chu)系(xi)統(tong)(tong)最核心的(de)(de)技(ji)(ji)術(shu)特征。
通常,從云存儲的技(ji)術實現層(ceng)次上(shang)看,從底層(ceng)向上(shang),可以分為存儲層(ceng)、管理(li)調(diao)度層(ceng)、訪問(wen)接口層(ceng)、業務應用層(ceng)等(deng)四個層(ceng)次,如上(shang)圖所示:
云存儲技術實現層次
其(qi)中存(cun)(cun)儲層(ceng)是(shi)云存(cun)(cun)儲的基(ji)礎,存(cun)(cun)儲設(she)(she)(she)備(bei)可以是(shi)FC 光纖通道(dao)設(she)(she)(she)備(bei),也(ye)可以是(shi)基(ji)于IP的ISCSI 存(cun)(cun)儲設(she)(she)(she)備(bei)或者NAS 系統,在(zai)一個(ge)云存(cun)(cun)儲系統中,底層(ceng)物理(li)存(cun)(cun)儲設(she)(she)(she)備(bei)數量(liang)龐大,而(er)且(qie)設(she)(she)(she)備(bei)形態(tai)理(li)論上(shang)也(ye)允許異構,在(zai)物理(li)存(cun)(cun)儲設(she)(she)(she)備(bei)之上(shang)是(shi)一個(ge)統一的存(cun)(cun)儲設(she)(she)(she)備(bei)管理(li)層(ceng),實現對物理(li)存(cun)(cun)儲設(she)(she)(she)備(bei)的邏(luo)輯(ji)虛擬(ni)化管理(li)、狀(zhuang)態(tai)監控和維護等功(gong)能。
管理調度層(ceng)是(shi)云(yun)存儲(chu)最核心的部(bu)分(fen),這一層(ceng)次(ci)協(xie)調各底層(ceng)存儲(chu)設備實(shi)(shi)現(xian)對數據的多副本或冗(rong)余(yu)編碼計算(suan)、分(fen)散存儲(chu)、負載均衡等功能(neng),在常見的云(yun)存儲(chu)系統中,通常采用(yong)分(fen)布式文件(jian)系統或者對象存儲(chu)來實(shi)(shi)現(xian)。不過(guo),在具體實(shi)(shi)現(xian)時,該層(ceng)的功能(neng)也(ye)可能(neng)上移,位于訪問(wen)接(jie)口層(ceng)和應(ying)用(yong)服(fu)務(wu)層(ceng)之間(jian),甚至直接(jie)嵌(qian)入到業(ye)務(wu)應(ying)用(yong)層(ceng)中,和業(ye)務(wu)應(ying)用(yong)緊密結合(he),形成業(ye)務(wu)專用(yong)云(yun)存儲(chu)。
訪(fang)問接(jie)口(kou)層(ceng)是業務應用和云(yun)存儲平臺之間的(de)一個橋梁(liang),提供應用服務所需(xu)要調用的(de)函數(shu)接(jie)口(kou),由于云(yun)存儲系(xi)統往(wang)(wang)往(wang)(wang)需(xu)要支(zhi)持多(duo)種不(bu)同的(de)業務系(xi)統,往(wang)(wang)往(wang)(wang)需(xu)要提供多(duo)種訪(fang)問接(jie)口(kou),例(li)如ISCSI、NFS、CIFT、FTP、REST 等。
業(ye)(ye)務(wu)應(ying)(ying)(ying)用(yong)(yong)層通過云存儲(chu)(chu)(chu)(chu)(chu)系統(tong)提(ti)供(gong)的(de)(de)(de)各種(zhong)訪問接口,對(dui)用(yong)(yong)戶提(ti)供(gong)豐富(fu)的(de)(de)(de)業(ye)(ye)務(wu)類型,例(li)如(ru)公有云存儲(chu)(chu)(chu)(chu)(chu)提(ti)供(gong)的(de)(de)(de)網盤(pan)服務(wu)、數據歸檔服務(wu)、高清視頻監(jian)控服務(wu)等。部分(fen)云存儲(chu)(chu)(chu)(chu)(chu)系統(tong)也會(hui)在這一層實(shi)(shi)(shi)現管(guan)理調度層的(de)(de)(de)功能,將業(ye)(ye)務(wu)數據的(de)(de)(de)冗余編碼、分(fen)散存儲(chu)(chu)(chu)(chu)(chu)、負載均(jun)衡、故障(zhang)保護等功能和(he)各種(zhong)業(ye)(ye)務(wu)的(de)(de)(de)實(shi)(shi)(shi)現緊密(mi)結合,提(ti)供(gong)非常極(ji)具特色的(de)(de)(de)業(ye)(ye)務(wu)應(ying)(ying)(ying)用(yong)(yong),和(he)云存儲(chu)(chu)(chu)(chu)(chu)系統(tong)的(de)(de)(de)核心功能在管(guan)理調度層實(shi)(shi)(shi)現的(de)(de)(de)通用(yong)(yong)云存儲(chu)(chu)(chu)(chu)(chu)相對(dui)應(ying)(ying)(ying),此類云存儲(chu)(chu)(chu)(chu)(chu)系統(tong)的(de)(de)(de)核心功能在應(ying)(ying)(ying)用(yong)(yong)層實(shi)(shi)(shi)現,可以歸類為應(ying)(ying)(ying)用(yong)(yong)云存儲(chu)(chu)(chu)(chu)(chu),例(li)如(ru)宇視公司的(de)(de)(de)視頻監(jian)控云存儲(chu)(chu)(chu)(chu)(chu)CDS(Cloud DirectStorage)解決(jue)方案,可以為用(yong)(yong)戶提(ti)供(gong)極(ji)其豐富(fu)的(de)(de)(de)視頻監(jian)控業(ye)(ye)務(wu)。
CDS 解決(jue)方(fang)案繼承了宇(yu)視(shi)科技視(shi)頻(pin)(pin)ISCSI塊直(zhi)存(cun)高(gao)性能的(de)(de)(de)優點(dian),采(cai)用宇(yu)視(shi)最(zui)先(xian)進的(de)(de)(de)裸數(shu)據存(cun)儲(chu)(chu)(chu)(chu)(chu)技術,可極(ji)大程度(du)發揮存(cun)儲(chu)(chu)(chu)(chu)(chu)設(she)備讀寫性能,實現云存(cun)儲(chu)(chu)(chu)(chu)(chu)中的(de)(de)(de)秒(miao)級檢索和回(hui)放,通過采(cai)用標準(zhun)的(de)(de)(de)IPSAN 設(she)備,通過CDV 存(cun)儲(chu)(chu)(chu)(chu)(chu)虛擬化組件,構成(cheng)全局(ju)統一的(de)(de)(de)虛擬存(cun)儲(chu)(chu)(chu)(chu)(chu)空(kong)間,系(xi)統支持海量的(de)(de)(de)云存(cun)儲(chu)(chu)(chu)(chu)(chu)節點(dian)管理能力,可管理高(gao)達(da)2048 個存(cun)儲(chu)(chu)(chu)(chu)(chu)節點(dian),支持動(dong)態擴容和縮(suo)容,通過云存(cun)儲(chu)(chu)(chu)(chu)(chu)管理服(fu)務器對(dui)物理存(cun)儲(chu)(chu)(chu)(chu)(chu)資源進行實時動(dong)態監控,支持節點(dian)間負載(zai)均衡和故障(zhang)業(ye)務重新調度(du),提(ti)供(gong)不間斷的(de)(de)(de)視(shi)頻(pin)(pin)和圖片存(cun)儲(chu)(chu)(chu)(chu)(chu)服(fu)務。同時也(ye)能提(ti)供(gong)視(shi)頻(pin)(pin)實況、視(shi)頻(pin)(pin)點(dian)播回(hui)放、報警管理、布防撤(che)防、地圖應用等多種視(shi)頻(pin)(pin)業(ye)務。
應(ying)用(yong)云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)和(he)業務(wu)深度結合,針對(dui)業務(wu)特點選(xuan)擇(ze)適(shi)合的(de)(de)(de)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)模式,往(wang)往(wang)能(neng)(neng)提供(gong)非常優異的(de)(de)(de)業務(wu)訪問性能(neng)(neng),但通常各(ge)廠家(jia)實現(xian)(xian)方(fang)式各(ge)有(you)不同,難以互通或者供(gong)第三方(fang)開發新的(de)(de)(de)數(shu)(shu)(shu)據(ju)處理業務(wu),因(yin)此,應(ying)用(yong)云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統比較適(shi)合業務(wu)種類(lei)較少,業務(wu)模型(xing)相對(dui)固(gu)定的(de)(de)(de)應(ying)用(yong)場合。而一個云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統中(zhong)(zhong),不僅會存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)海量的(de)(de)(de)數(shu)(shu)(shu)據(ju),而且需(xu)要(yao)對(dui)這(zhe)(zhe)些數(shu)(shu)(shu)據(ju)進行分(fen)析(xi)和(he)利用(yong),這(zhe)(zhe)往(wang)往(wang)需(xu)要(yao)集(ji)合多個專業廠商來進行各(ge)自(zi)擅長領域的(de)(de)(de)業務(wu)開發,因(yin)此,系(xi)(xi)(xi)統的(de)(de)(de)開放(fang)性、接(jie)口(kou)(kou)的(de)(de)(de)標(biao)準化,則(ze)成為云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統建設的(de)(de)(de)重要(yao)需(xu)求,在(zai)這(zhe)(zhe)種云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統中(zhong)(zhong),由管理調度層實現(xian)(xian)的(de)(de)(de)云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)的(de)(de)(de)核心功能(neng)(neng)則(ze)直接(jie)在(zai)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)層之上(shang)實現(xian)(xian),再通過訪問接(jie)口(kou)(kou)層為業務(wu)平臺(tai)提供(gong)通用(yong)的(de)(de)(de)訪問接(jie)口(kou)(kou),如ISCSI塊存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)訪問接(jie)口(kou)(kou)、POSIX 文(wen)件訪問接(jie)口(kou)(kou)、RESTFUL 接(jie)口(kou)(kou)等,而業務(wu)應(ying)用(yong)開發則(ze)無需(xu)再考慮數(shu)(shu)(shu)據(ju)冗(rong)余、分(fen)散存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)、負載均衡等存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)專屬特性,而集(ji)中(zhong)(zhong)在(zai)業務(wu)應(ying)用(yong)本身(shen)的(de)(de)(de)功能(neng)(neng)特性方(fang)面,例如視頻監控(kong)系(xi)(xi)(xi)統中(zhong)(zhong)的(de)(de)(de)智能(neng)(neng)識別、卡口(kou)(kou)計數(shu)(shu)(shu)等,這(zhe)(zhe)種云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統與具體的(de)(de)(de)應(ying)用(yong)耦合程度較低(di),具有(you)更好的(de)(de)(de)通用(yong)性,可以認為是一種通用(yong)云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)。目前(qian)通用(yong)云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)的(de)(de)(de)應(ying)用(yong)主要(yao)以分(fen)布式文(wen)件系(xi)(xi)(xi)統和(he)對(dui)象(xiang)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)的(de)(de)(de)方(fang)式出現(xian)(xian),由于人類(lei)發展過程中(zhong)(zhong)產生的(de)(de)(de)數(shu)(shu)(shu)據(ju)絕大(da)部分(fen)都是以文(wen)件形式存(cun)(cun)(cun)(cun)(cun)(cun)(cun)在(zai),因(yin)此云(yun)(yun)(yun)(yun)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統以分(fen)布式集(ji)群文(wen)件系(xi)(xi)(xi)統的(de)(de)(de)方(fang)式出現(xian)(xian)是一個必然的(de)(de)(de)選(xuan)擇(ze),而隨著寬帶移動互聯網(wang)、WEB 技術的(de)(de)(de)發展,對(dui)象(xiang)存(cun)(cun)(cun)(cun)(cun)(cun)(cun)儲(chu)(chu)(chu)(chu)系(xi)(xi)(xi)統也(ye)越來越多地出現(xian)(xian)在(zai)網(wang)盤、數(shu)(shu)(shu)據(ju)歸檔等應(ying)用(yong)中(zhong)(zhong)。
在一個通(tong)用云存(cun)(cun)(cun)儲系(xi)(xi)統(tong)中(zhong),都(dou)需要通(tong)過橫(heng)向擴(kuo)展(zhan)存(cun)(cun)(cun)儲節點來解決大量數(shu)(shu)據(ju)(ju)(ju)(ju)的(de)(de)(de)存(cun)(cun)(cun)儲容量和(he)讀寫訪(fang)(fang)(fang)問(wen)(wen)性(xing)(xing)(xing)(xing)能問(wen)(wen)題,而(er)其中(zhong)最(zui)關(guan)鍵的(de)(de)(de)問(wen)(wen)題,是(shi)解決記錄數(shu)(shu)據(ju)(ju)(ju)(ju)邏輯位(wei)置和(he)物理(li)位(wei)置之(zhi)間(jian)的(de)(de)(de)映(ying)像關(guan)系(xi)(xi)的(de)(de)(de)可靠(kao)性(xing)(xing)(xing)(xing)存(cun)(cun)(cun)儲和(he)高(gao)性(xing)(xing)(xing)(xing)能訪(fang)(fang)(fang)問(wen)(wen)問(wen)(wen)題,也就是(shi)元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)的(de)(de)(de)存(cun)(cun)(cun)儲與(yu)訪(fang)(fang)(fang)問(wen)(wen)問(wen)(wen)題,當然,元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)還記錄著文件(jian)或者對象的(de)(de)(de)訪(fang)(fang)(fang)問(wen)(wen)權限、屬(shu)性(xing)(xing)(xing)(xing)、訪(fang)(fang)(fang)問(wen)(wen)時(shi)間(jian)等其他(ta)信(xin)息(xi)。元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)的(de)(de)(de)處理(li)好壞,將極大地影響系(xi)(xi)統(tong)的(de)(de)(de)整(zheng)體性(xing)(xing)(xing)(xing)能和(he)可擴(kuo)展(zhan)性(xing)(xing)(xing)(xing)。一般而(er)言,根據(ju)(ju)(ju)(ju)對元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)的(de)(de)(de)管理(li)模(mo)型,可以將通(tong)用云存(cun)(cun)(cun)儲系(xi)(xi)統(tong)分為三種(zhong)類型,即集中(zhong)式元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)、分布式元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)和(he)無元(yuan)數(shu)(shu)據(ju)(ju)(ju)(ju)三種(zhong)類型的(de)(de)(de)系(xi)(xi)統(tong)。
集中(zhong)式元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)云存(cun)(cun)(cun)(cun)儲(chu)(chu)系(xi)(xi)(xi)統(tong)是一(yi)種典型(xing)的非對(dui)(dui)稱式系(xi)(xi)(xi)統(tong),在(zai)系(xi)(xi)(xi)統(tong)中(zhong),通常具(ju)有(you)一(yi)個(ge)中(zhong)央元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)管(guan)(guan)理(li)服(fu)務(wu)(wu)器(qi),負責元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)的存(cun)(cun)(cun)(cun)儲(chu)(chu)和處理(li)查(cha)(cha)詢與(yu)修改請求,例如,在(zai)HDFS 系(xi)(xi)(xi)統(tong)中(zhong),該元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)管(guan)(guan)理(li)服(fu)務(wu)(wu)器(qi)即為名字節(jie)點Namenode,同時(shi),存(cun)(cun)(cun)(cun)在(zai)大量的數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)存(cun)(cun)(cun)(cun)儲(chu)(chu)節(jie)點提供客(ke)(ke)戶I/O數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)的并行(xing)(xing)存(cun)(cun)(cun)(cun)儲(chu)(chu)與(yu)訪(fang)問。這(zhe)種架構中(zhong),客(ke)(ke)戶端(duan)每次對(dui)(dui)數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)流(liu)的I/O 操作(zuo),都(dou)需要(yao)先向元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)管(guan)(guan)理(li)服(fu)務(wu)(wu)器(qi)進行(xing)(xing)元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)查(cha)(cha)詢,客(ke)(ke)戶端(duan)在(zai)獲(huo)得需要(yao)讀(du)寫的數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)塊物理(li)位置(zhi)等信(xin)息后,對(dui)(dui)于數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)的I/O 操作(zuo)則直接在(zai)客(ke)(ke)戶端(duan)和數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)存(cun)(cun)(cun)(cun)儲(chu)(chu)節(jie)點之間(jian)進行(xing)(xing)。相對(dui)(dui)傳統(tong)存(cun)(cun)(cun)(cun)儲(chu)(chu)系(xi)(xi)(xi)統(tong),集中(zhong)元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)云存(cun)(cun)(cun)(cun)儲(chu)(chu)系(xi)(xi)(xi)統(tong)將控制流(liu)和數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)流(liu)進行(xing)(xing)了(le)(le)分離,系(xi)(xi)(xi)統(tong)在(zai)擴(kuo)展性(xing)和處理(li)性(xing)能方面(mian)獲(huo)得了(le)(le)較(jiao)大的提升(sheng),同時(shi),由(you)于元數(shu)(shu)(shu)(shu)據(ju)(ju)(ju)(ju)集中(zhong)在(zai)一(yi)臺服(fu)務(wu)(wu)器(qi)上進行(xing)(xing)管(guan)(guan)理(li),整個(ge)系(xi)(xi)(xi)統(tong)架構比(bi)較(jiao)簡(jian)單,降低了(le)(le)系(xi)(xi)(xi)統(tong)設計的復雜性(xing),目前業界采用這(zhe)種架構的系(xi)(xi)(xi)統(tong)主要(yao)有(you)GFS、HDFS、Lustre 等。
不難看出,雖然集中元數據云存儲系統架構簡單,但會存在兩個主要的問題:
性(xing)(xing)能(neng)(neng)瓶(ping)頸問(wen)(wen)(wen)題(ti)。元(yuan)數(shu)(shu)(shu)據的(de)基本特(te)性(xing)(xing)要(yao)求任何時(shi)候對(dui)(dui)(dui)用(yong)(yong)(yong)戶數(shu)(shu)(shu)據的(de)訪(fang)(fang)(fang)問(wen)(wen)(wen),都(dou)(dou)需(xu)(xu)要(yao)同步(bu)地修改元(yuan)數(shu)(shu)(shu)據,例(li)如(ru),即使(shi)是對(dui)(dui)(dui)文(wen)件(jian)(jian)進行(xing)(xing)(xing)讀操(cao)作(zuo)(zuo)或者列(lie)出目錄操(cao)作(zuo)(zuo),都(dou)(dou)需(xu)(xu)要(yao)更新(xin)元(yuan)數(shu)(shu)(shu)據中的(de)文(wen)件(jian)(jian)訪(fang)(fang)(fang)問(wen)(wen)(wen)時(shi)間(jian)屬(shu)性(xing)(xing)。由于每次I/O 訪(fang)(fang)(fang)問(wen)(wen)(wen)都(dou)(dou)需(xu)(xu)要(yao)首先訪(fang)(fang)(fang)問(wen)(wen)(wen)元(yuan)數(shu)(shu)(shu)據服務器,以便實現對(dui)(dui)(dui)系統命名空(kong)間(jian)進行(xing)(xing)(xing)解析、數(shu)(shu)(shu)據定位和(he)訪(fang)(fang)(fang)問(wen)(wen)(wen)對(dui)(dui)(dui)象權限(xian)控制等,然(ran)后才能(neng)(neng)夠(gou)(gou)對(dui)(dui)(dui)用(yong)(yong)(yong)戶數(shu)(shu)(shu)據進行(xing)(xing)(xing)I/O 操(cao)作(zuo)(zuo),隨(sui)著系統規模不斷擴大(da),需(xu)(xu)要(yao)管(guan)理的(de)存儲節點、文(wen)件(jian)(jian)數(shu)(shu)(shu)量、I/O 操(cao)作(zuo)(zuo)數(shu)(shu)(shu)量等都(dou)(dou)會急劇增(zeng)加(jia)(jia),而(er)對(dui)(dui)(dui)元(yuan)數(shu)(shu)(shu)據進行(xing)(xing)(xing)管(guan)理的(de)物理服務器性(xing)(xing)能(neng)(neng)有限(xian),從而(er)形(xing)成性(xing)(xing)能(neng)(neng)瓶(ping)頸,這種性(xing)(xing)能(neng)(neng)瓶(ping)頸在(zai)大(da)量小文(wen)件(jian)(jian)訪(fang)(fang)(fang)問(wen)(wen)(wen)時(shi)會更為(wei)(wei)突出。為(wei)(wei)解決這一問(wen)(wen)(wen)題(ti),人(ren)們通常會采用(yong)(yong)(yong)更高(gao)性(xing)(xing)能(neng)(neng)的(de)CPU,并(bing)且采用(yong)(yong)(yong)SSD 來加(jia)(jia)速(su)對(dui)(dui)(dui)元(yuan)數(shu)(shu)(shu)據的(de)訪(fang)(fang)(fang)問(wen)(wen)(wen),雖然(ran)能(neng)(neng)夠(gou)(gou)在(zai)一定程度(du)上提(ti)升(sheng)元(yuan)數(shu)(shu)(shu)據訪(fang)(fang)(fang)問(wen)(wen)(wen)性(xing)(xing)能(neng)(neng),但往往成本高(gao)昂,且提(ti)升(sheng)效果有限(xian),系統性(xing)(xing)能(neng)(neng)隨(sui)存儲節點數(shu)(shu)(shu)量增(zeng)加(jia)(jia)只能(neng)(neng)呈現對(dui)(dui)(dui)數(shu)(shu)(shu)方式提(ti)升(sheng)而(er)無法線性(xing)(xing)提(ti)升(sheng)。
元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)單點(dian)故(gu)(gu)(gu)障(zhang)(zhang)(zhang)問(wen)題。在(zai)集(ji)中元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)云存(cun)儲系統(tong)中,整(zheng)個(ge)系統(tong)的(de)性(xing)能(neng)和可靠(kao)性(xing)完全依(yi)賴于(yu)元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi),一(yi)(yi)旦元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)故(gu)(gu)(gu)障(zhang)(zhang)(zhang),系統(tong)將無法提供(gong)任何服(fu)(fu)務(wu)(wu)(wu)(wu),因此,元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)就是整(zheng)個(ge)系統(tong)中的(de)潛在(zai)單點(dian)故(gu)(gu)(gu)障(zhang)(zhang)(zhang)點(dian)。為(wei)解決(jue)這一(yi)(yi)問(wen)題,通常對元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)采用(yong)備(bei)機形成HA 解決(jue)方(fang)案來提供(gong)更高的(de)系統(tong)可用(yong)性(xing),一(yi)(yi)旦主(zhu)(zhu)用(yong)元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)故(gu)(gu)(gu)障(zhang)(zhang)(zhang),備(bei)用(yong)元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)可以替代(dai)主(zhu)(zhu)用(yong)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)提供(gong)對元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)的(de)訪問(wen)操作(zuo),主(zhu)(zhu)用(yong)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)和備(bei)用(yong)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)之間(jian)的(de)元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)必須隨時同步,否則(ze)一(yi)(yi)旦主(zhu)(zhu)用(yong)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)故(gu)(gu)(gu)障(zhang)(zhang)(zhang),則(ze)可能(neng)導致數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)不一(yi)(yi)致問(wen)題,但元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)同步操作(zuo)會(hui)進一(yi)(yi)步加重了元(yuan)(yuan)(yuan)數(shu)(shu)(shu)據(ju)(ju)(ju)(ju)服(fu)(fu)務(wu)(wu)(wu)(wu)器(qi)(qi)的(de)性(xing)能(neng)負擔,導致整(zheng)個(ge)系統(tong)的(de)訪問(wen)性(xing)能(neng)受到拖累。
為了解(jie)決集(ji)(ji)中(zhong)元(yuan)(yuan)數(shu)(shu)據(ju)(ju)系統中(zhong)的性能瓶頸和單(dan)點故障問題,一種改(gai)進后的分(fen)布(bu)(bu)式元(yuan)(yuan)數(shu)(shu)據(ju)(ju)云存(cun)儲系統得以(yi)出(chu)現,這種系統采用多臺(tai)元(yuan)(yuan)數(shu)(shu)據(ju)(ju)服(fu)(fu)務器(qi)形(xing)成(cheng)集(ji)(ji)群工作的方式提供元(yuan)(yuan)數(shu)(shu)據(ju)(ju)訪(fang)問服(fu)(fu)務,集(ji)(ji)群中(zhong)的每一臺(tai)設備都可以(yi)提供元(yuan)(yuan)數(shu)(shu)據(ju)(ju)訪(fang)問,從而提高整體(ti)訪(fang)問性能,并且解(jie)決了元(yuan)(yuan)數(shu)(shu)據(ju)(ju)服(fu)(fu)務器(qi)單(dan)點故障問題。在實(shi)際設計中(zhong),分(fen)布(bu)(bu)式元(yuan)(yuan)數(shu)(shu)據(ju)(ju)服(fu)(fu)務器(qi)通常和數(shu)(shu)據(ju)(ju)存(cun)儲節點集(ji)(ji)成(cheng)到一起,形(xing)成(cheng)全對稱分(fen)布(bu)(bu)式架構,EMC 公司(si)的Isilon OneFS 是這一架構的典型(xing)代表。
分布式(shi)云(yun)存(cun)(cun)儲系統需要在所有(you)元(yuan)(yuan)數(shu)(shu)(shu)據(ju)存(cun)(cun)儲節點之間進(jin)行元(yuan)(yuan)數(shu)(shu)(shu)據(ju)同(tong)步操(cao)作(zuo),這大大增加了系統設計的復雜性,如(ru)果元(yuan)(yuan)數(shu)(shu)(shu)據(ju)沒有(you)及時得到同(tong)步,或(huo)(huo)者(zhe)遭(zao)到意外(wai)破壞,則(ze)會出現同(tong)一(yi)(yi)文件或(huo)(huo)者(zhe)對象的元(yuan)(yuan)數(shu)(shu)(shu)據(ju)不一(yi)(yi)致,進(jin)一(yi)(yi)步導(dao)致上層應用在通過不同(tong)物理(li)服務器(qi)訪(fang)問文件或(huo)(huo)對象數(shu)(shu)(shu)據(ju)時出現數(shu)(shu)(shu)據(ju)不一(yi)(yi)致或(huo)(huo)者(zhe)讀寫錯誤的問題,這種風險隨著云(yun)存(cun)(cun)儲系統規模的擴(kuo)大而大幅增加。
此外,由于分(fen)布式元(yuan)數(shu)據系統(tong)需要(yao)在(zai)大量節點(dian)之(zhi)間(jian)進(jin)(jin)行(xing)(xing)元(yuan)數(shu)據同(tong)步,同(tong)步過(guo)程需要(yao)采用(yong)(yong)各種加(jia)(jia)鎖(suo)機(ji)制,以(yi)確保數(shu)據的一(yi)致(zhi)性(xing)(xing),加(jia)(jia)鎖(suo)機(ji)制的存在(zai)導(dao)致(zhi)很(hen)多任務處理無(wu)法并行(xing)(xing)進(jin)(jin)行(xing)(xing),元(yuan)數(shu)據同(tong)步也會對后端帶(dai)寬帶(dai)來(lai)開(kai)銷,二者都會帶(dai)來(lai)系統(tong)性(xing)(xing)能(neng)(neng)上的額外開(kai)銷,部分(fen)抵消由于采用(yong)(yong)分(fen)布式元(yuan)數(shu)據存取帶(dai)來(lai)的性(xing)(xing)能(neng)(neng)提(ti)升,且(qie)系統(tong)規模越大,性(xing)(xing)能(neng)(neng)開(kai)銷越顯(xian)著。為了(le)降低性(xing)(xing)能(neng)(neng)開(kai)銷,通常需要(yao)為節點(dian)配置高性(xing)(xing)能(neng)(neng)CPU、SSD或者高性(xing)(xing)能(neng)(neng)SAS 或FC 硬(ying)盤陣列(lie)來(lai)提(ti)供元(yuan)數(shu)據訪(fang)問,并且(qie)采用(yong)(yong)分(fen)布式共享(xiang)內存和高性(xing)(xing)能(neng)(neng)后端網絡(如Infiniband 或萬兆以(yi)太(tai)網)來(lai)加(jia)(jia)速元(yuan)數(shu)據同(tong)步速度,不過(guo)這(zhe)樣就顯(xian)著提(ti)升了(le)系統(tong)的硬(ying)件成(cheng)本。
既然(ran)元(yuan)數據(ju)(ju)總(zong)會(hui)帶來性(xing)能開(kai)銷和可(ke)靠(kao)性(xing)問題,第(di)三種云(yun)存(cun)(cun)儲(chu)系(xi)(xi)統(tong)則(ze)徹(che)底拋棄元(yuan)數據(ju)(ju),而(er)是采用(yong)算(suan)法(fa)來對文(wen)件或(huo)對象進行(xing)定位(wei),并將該算(suan)法(fa)集成在(zai)每(mei)一(yi)(yi)個存(cun)(cun)儲(chu)節點(dian)上,客戶端從(cong)(cong)任何(he)一(yi)(yi)個存(cun)(cun)儲(chu)節點(dian)進行(xing)數據(ju)(ju)訪問都會(hui)獲得同樣(yang)的(de)結(jie)果,云(yun)存(cun)(cun)儲(chu)系(xi)(xi)統(tong)中的(de)每(mei)一(yi)(yi)個存(cun)(cun)儲(chu)節點(dian)都可(ke)以獨立、并行(xing)地對外提(ti)供(gong)服務,從(cong)(cong)而(er)真正實現性(xing)能隨節點(dian)數增加(jia)而(er)線性(xing)擴展,由于無(wu)需在(zai)節點(dian)間進行(xing)元(yuan)數據(ju)(ju)的(de)同步操作,極大地提(ti)高了(le)系(xi)(xi)統(tong)的(de)穩定性(xing)和可(ke)靠(kao)性(xing)。
宇視科(ke)技的(de)(de)UCS(Unified Cloud Storage)統(tong)(tong)一云存(cun)儲(chu)系(xi)(xi)統(tong)(tong)是在視頻(pin)應用(yong)云存(cun)儲(chu)CDS 解決方案之外(wai)提供(gong)的(de)(de)更為通用(yong)的(de)(de)云存(cun)儲(chu)解決方案,它是一種(zhong)采用(yong)無(wu)(wu)元數(shu)據(ju)設(she)計的(de)(de)全(quan)對(dui)(dui)稱(cheng)分(fen)布(bu)式存(cun)儲(chu)系(xi)(xi)統(tong)(tong),通過UVFS 分(fen)布(bu)式文件系(xi)(xi)統(tong)(tong)實(shi)現對(dui)(dui)存(cun)儲(chu)資源的(de)(de)虛擬化統(tong)(tong)一管理(li),對(dui)(dui)外(wai)提供(gong)各種(zhong)標準軟(ruan)件接口,無(wu)(wu)需(xu)集成(cheng)特殊軟(ruan)件客(ke)戶端(duan)到前(qian)端(duan)設(she)備(例如(ru)IPC),也無(wu)(wu)需(xu)前(qian)端(duan)應用(yong)設(she)備在數(shu)據(ju)保護方面做額外(wai)的(de)(de)開(kai)發工作,對(dui)(dui)于數(shu)據(ju)的(de)(de)保護功能完全(quan)在存(cun)儲(chu)卷層面實(shi)現,具(ju)有架構簡單(dan)、高性能、高擴展性、高可用(yong)性、開(kai)放性等(deng)特點(dian),如(ru)下(xia)圖所(suo)示:
宇視全對稱分布式UCS 通用云存儲
具有以下顯著特征:
■全局統一的命名空間
■無元數據全對稱分布式設計,智能哈希算法數據定位,并行任務處理,系統性能隨節點數增加而線性提升
■靈活、高效的系統擴展能力,最大可橫向擴展256 個存儲節點,管理數十PB 存儲空間
■支持節點間故障保護、負載均衡,任一節點故障,剩余節點均可繼續提供客戶端無感知的數據讀寫服務
■基于卷提供多種數據分布和保護策略,如N+M 糾刪碼、多副本、高性能分布式存儲等
■繼承宇視在傳統存儲上的長期積累,支持多種RAID 類型,提供更高性能的塊級底層數據保護
■存儲節點支持掉電數據保護功能,已寫入文件系統buffer 的數據在掉電時能得到妥善保存
■支持ISCSI、NFS、CIFS、FTP、HTTP、REST 等多種(zhong)標準訪(fang)問接口
UCS 統一云(yun)存(cun)儲解(jie)決方案可(ke)廣泛(fan)用于高清視頻監控系(xi)(xi)統、警用執法(fa)儀在線存(cun)儲、警情視頻庫歸檔(dang)系(xi)(xi)統等,為智慧城市(shi)、平安城市(shi)建設提供大數(shu)據、云(yun)存(cun)儲解(jie)決方案。