分布式文件系统之MogileFS

MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。目前使用MogileFS 的公司非常多，如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等，分别为所在的组织或公司管理着海量的图片。

MogileFS由3个部分组成：
(1)server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。
(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。
(3)客户端API：MogileFS的客户端API很多，例如Perl、PHP、Java、Python等，用这个模块可以编写客户端程序，实现文件的备份管理功能等。

现在我们在做一个mogilefs集群分布式存储,节点设计如下:

192.168.42.150 node1 [Nginx,Tracker, Storage,DataBase]
192.168.42.151 node2 [Tracker, Storage]
192.168.42.152 node3 [Tracker, Storage]

1.准备这三台机器,初始化工作,时间同步,分别在这三台机器上安装mogilefs,我们这里用rpm安装

cd /usr/local/src  
#下载RPM安装包
wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-2.46-2.el6.noarch.rpm 
wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
wget http://192.168.42.26/install_package/down/mogilefs/MogileFS-Utils-2.19-1.el6.noarch.rpm
wget http://192.168.42.26/install_package/down/mogilefs/perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm
wget http://192.168.42.26/install_package/down/mogilefs/perl-MogileFS-Client-1.14-1.el6.noarch.rpm
wget http://192.168.42.26/install_package/down/mogilefs/perl-Perlbal-1.78-1.el6.noarch.rpm

#安装依赖
yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y


#安装rpm
yum install *.rpm -y

2.在node1节点上部署mariadb

yum install mariadb-server -y
vim /etc/my.cnf.d/server.cnf
[mysqld]
skip_name_resolve=1
innodb_file_per_table = 1

#启动mysql,做以下操作
systemctl start mariadb
mysql

MariaDB [(none)]> GRANT ALL ON *.* TO 'root'@'192.168.42.%' IDENTIFIED BY 'mageedu';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'moguser'@'192.168.42.%' IDENTIFIED BY 'mogpass' WITH GRANT OPTION;   
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH   PRIVILEGES; 
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> CREATE DATABASE mogilefs;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> exit
Bye

#设定数据库,生成数据表
mogdbsetup --dbhost=192.168.42.150  --dbrootpass='mageedu'  --dbuser='moguser' --dbpass='mogpass'  --dbname=mogilefs  

MariaDB [(none)]> use mogilefs;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mogilefs]> show tables;
+----------------------+
| Tables_in_mogilefs   |
+----------------------+
| checksum             |
| class                |
| device               |
| domain               |
| file                 |
| file_on              |
| file_on_corrupt      |
| file_to_delete       |
| file_to_delete2      |
| file_to_delete_later |
| file_to_queue        |
| file_to_replicate    |
| fsck_log             |
| host                 |
| server_settings      |
| tempfile             |
| unreachable_fids     |
+----------------------+
17 rows in set (0.00 sec)

3.配置mogilefsd

vim /etc/mogilefs/mogilefsd.conf

db_dsn = DBI:mysql:mogilefs:host=192.168.42.150
db_user = moguser
db_pass = mogpass
listen = 0.0.0.0:7001
conf_port = 7001

4.启动mogilefsd

service mogilefsd start

[root@node1 my.cnf.d]# service mogilefsd start
Starting mogilefsd (via systemctl):                        [  OK  ]
[root@node1 my.cnf.d]# ss -tnl
State      Recv-Q Send-Q  Local Address:Port                 Peer Address:Port              
LISTEN     0      128                 *:22                              *:*                  
LISTEN     0      1024                *:7001                            *:*                  
LISTEN     0      100         127.0.0.1:25                              *:*                  
LISTEN     0      50                  *:3306                            *:*                  
LISTEN     0      128                :::22                             :::*                  
LISTEN     0      100               ::1:25                             :::*

5.配置storage(可以是所有节点)

mkdir -pv /data/mogilefs/dev1 #创建设备目录
chown -R mogilefs.mogilefs /data/mogilefs/dev1 #权限
vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs/

修改脚本
lockfile下添加
pidfile=/var/run/mogilefsd/mogstored.pid

start()修改
[ $RETVAL = 0 ] && success && touch ${lockfile} && echo $(pidof mogstored) > ${pidfile} || failure

stop()修改
[ $RETVAL = 0 ] && success && rm -f ${lockfile} ${pidfile} || failure

#启动mogstored
[root@node1 init.d]# service mogstored start
Starting mogstored (via systemctl):                        [  OK  ]
[root@node1 init.d]# ss -tnl
State      Recv-Q Send-Q  Local Address:Port                 Peer Address:Port              
LISTEN     0      1024                *:7500                            *:*                  
LISTEN     0      1024                *:7501                            *:*                  
LISTEN     0      128                 *:22                              *:*                  
LISTEN     0      1024                *:7001                            *:*                  
LISTEN     0      100         127.0.0.1:25                              *:*                  
LISTEN     0      50                  *:3306                            *:*                  
LISTEN     0      128                :::22                             :::*                  
LISTEN     0      100               ::1:25                             :::*

6.node2,node3上分别创建设备目录和分发配置文件,脚本文件

node2上操作:
mkdir -pv /data/mogilefs/dev2 
chown -R mogilefs.mogilefs /data/mogilefs/dev2 

node3上操作:
mkdir -pv /data/mogilefs/dev3 
chown -R mogilefs.mogilefs /data/mogilefs/dev3


#分发脚本和配置文件
scp /etc/rc.d/init.d/mogstored     root@192.168.42.151:/etc/rc.d/init.d/ 
scp /etc/mogilefs/mogstored.conf   root@192.168.42.151:/etc/mogilefs/

scp /etc/rc.d/init.d/mogstored     root@192.168.42.152:/etc/rc.d/init.d/ 
scp /etc/mogilefs/mogstored.conf   root@192.168.42.152:/etc/mogilefs/

在node2,和node3启动mogstored
service mogstored start

7.node1节点上需要添加

#查看帮助
mogadm -help


[root@node1 init.d]# mogadm check
Checking trackers...
  127.0.0.1:7001 ... OK
Checking hosts...
No devices found on tracker(s).

我们还可以使用mogadm --trackers=192.168.42.150:7001 check 得到同样的结果


#添加主机
mogadm --trackers=192.168.42.150:7001 host add 192.168.42.150 --ip=192.168.42.150  --status=alive
mogadm --trackers=192.168.42.150:7001 host add 192.168.42.151 --ip=192.168.42.151  --status=alive
mogadm --trackers=192.168.42.150:7001 host add 192.168.42.152 --ip=192.168.42.152  --status=alive

[root@node1 init.d]# mogadm host list
192.168.42.150 [1]: alive
  IP:       192.168.42.150:7500

192.168.42.151 [2]: alive
  IP:       192.168.42.151:7500

192.168.42.152 [3]: alive
  IP:       192.168.42.152:7500

添加设备

(1).没有添加设备之前查看,是没有结果的

[root@node1 init.d]# mogadm device list
192.168.42.150 [1]: alive
                    used(G)    free(G)   total(G)  weight(%)

192.168.42.151 [2]: alive
                    used(G)    free(G)   total(G)  weight(%)

192.168.42.152 [3]: alive
                    used(G)    free(G)   total(G)  weight(%)

(2)添加设备的操作命令

mogadm device add 192.168.42.150 1
mogadm device add 192.168.42.151 2
mogadm device add 192.168.42.152 3

(3)此时查看有结果了

[root@node1 init.d]# mogadm device list 
192.168.42.150 [1]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev1:   alive      1.688     15.298     16.986        100

192.168.42.151 [2]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev2:   alive      1.557     15.430     16.986        100

192.168.42.152 [3]: alive
                    used(G)    free(G)   total(G)  weight(%)
   dev3:   alive      1.557     15.430     16.986        100

(4)此时查看mogadm check

[root@node1 init.d]# mogadm check
Checking trackers...
  127.0.0.1:7001 ... OK

Checking hosts...
  [ 1] 192.168.42.150 ... OK
  [ 2] 192.168.42.151 ... OK
  [ 3] 192.168.42.152 ... OK

Checking devices...
  host device         size(G)    used(G)    free(G)   use%   ob state   I/O%
  ---- ------------ ---------- ---------- ---------- ------ ---------- -----
  [ 1] dev1            16.986      1.689     15.297   9.95%  writeable   0.0
  [ 2] dev2            16.986      1.557     15.429   9.17%  writeable   0.0
  [ 3] dev3            16.986      1.557     15.429   9.17%  writeable   0.0
  ---- ------------ ---------- ---------- ---------- ------
             total:    50.959      4.803     46.156   9.43%

添加域

(1)查看没有添加域的状态
[root@node1 init.d]# mogadm domain list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
(2)添加域操作
mogadm domain add files
mogadm domain add images

(3)再次查看列出
[root@node1 init.d]# mogadm domain list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 files                default                   2        MultipleHosts() NONE   

 images               default                   2        MultipleHosts() NONE

添加class

(1)查看一下class列表
[root@node1 init.d]# mogadm class list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 files                default                   2        MultipleHosts() NONE   

 images               default                   2        MultipleHosts() NONE   

(2).添加class操作(不知为什么这一步会导致只有一个副本)
需要这样添加 mogadm class add images jpeg --mindevcount=2 --replpolicy="MultipleHosts(2)"

mogadm class add images jpeg 
mogadm class add images png
mogadm class add images gif

(3).再次查看列表
[root@node1 init.d]# mogadm class  list
 domain               class                mindevcount   replpolicy   hashtype
-------------------- -------------------- ------------- ------------ -------
 files                default                   2        MultipleHosts() NONE   

 images               default                   2        MultipleHosts() NONE   
 images               gif                       2        MultipleHosts() NONE   
 images               jpeg                      2        MultipleHosts() NONE   
 images               png                       2        MultipleHosts() NONE

上传图片测试

(1)上传图片
mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.gif' --file='/tmp/83e.gif' --class=gif

mogupload --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' --file='/tmp/zhuoku194.jpg' --class=jpeg
mogupload --tracker=192.168.42.150:7001 --domain=images --key='2.jpg' --file='/tmp/1.jpg' --class=jpeg

(2)查看图片信息
mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif' 
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.gif' 
- file: /1.gif
     class:                  gif
  devcount:                    1
    domain:               images
       fid:                    2
       key:               /1.gif
    length:               499461
 - http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid

mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' 
[root@node1 init.d]# mogfileinfo --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' 
- file: /1.jpg
     class:                 jpeg
  devcount:                    1
    domain:               images
       fid:                    3
       key:               /1.jpg
    length:               374408
 - http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid


(3)在浏览器访问
http://192.168.42.151:7500/dev2/0/000/000/0000000002.fid
http://192.168.42.150:7500/dev1/0/000/000/0000000003.fid
都是ok的


删除文件:
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.jpg' 
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='2.jpg' 
mogdelete --tracker=192.168.42.150:7001 --domain=images --key='1.gif' 
删除class:
mogadm class delete images jpeg
mogadm class delete images gitj
mogadm class delete images png
删除域:
mogadm domain delete files
mogadm domain delete images


#配置Nginx

nginx 的mogilefs模块的官方文档地址：
http://www.grid.net.ru/nginx/mogilefs.en.html
wget http://www.grid.net.ru/nginx/download/nginx_mogilefs_module-1.0.4.tar.gz

tar xvf nginx_mogilefs_module-1.0.4.tar.gz 
mv nginx_mogilefs_module-1.0.4 /usr/local/

cd /usr/local/src
wget http://192.168.42.26/nginx-1.11.8.tar.gz
tar xvf nginx-1.11.8.tar.gz
cd nginx-1.11.8
yum groupinstall "Development Tools" "Server Platform Developments" 
yum install pcre-devel openssl-devel  zlib-devel -y
groupadd -r nginx
useradd -r -g nginx nginx

我的编译安装方法
./configure --with-pcre  --user=nginx --group=nginx  --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module  --with-http_realip_module --add-module=/usr/local/nginx_mogilefs_module-1.0.4

make && make install

编译会可能报错:
cc1: all warnings being treated as errors
make[1]: *** [objs/addon/nginx_mogilefs_module-1.0.4/ngx_http_mogilefs_module.o] Error 1
make[1]: Leaving directory `/usr/local/src/nginx-1.11.8'
make: *** [build] Error 2

解决办法:
我们这里打开
vim  objs/Makefile
#去掉警告
-Werror 

#编辑nginx配置文件

user   nginx;
worker_processes  1;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;

    keepalive_timeout  65;
    upstream trackers {
           server 192.168.42.150:7001 weight=1;
           server 192.168.42.151:7001 weight=1;
        server 192.168.42.152:7001 weight=1;
        check interval=1000 rise=2 fall=5 timeout=1000;
    }


    server {
        listen       80;
        server_name  localhost;

        location / {
            root   html;
            index  index.html index.htm;
        }

       location  /images/ {
          mogilefs_tracker trackers;
          mogilefs_domain images;
          mogilefs_methods GET;
          mogilefs_noverify on;

          mogilefs_pass {
                  proxy_pass $mogilefs_path;
                  proxy_hide_header Content-Type;
                  proxy_buffering off;
          }
      }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }


    }


}

浏览器访问:

http://192.168.42.150/images/2.jpg
http://192.168.42.150/images/1.gif
都OK

此时我们的trackers其实只有一个,但我们在nginx配置文件中写了三个,所以我们需要把另外两个节点也要安装tracker

scp /etc/mogilefs/mogilefsd.conf 192.168.42.151:/etc/mogilefs/
scp /etc/mogilefs/mogilefsd.conf 192.168.42.152:/etc/mogilefs/

启动mogilefsd:(两台节点)
service mogilefsd start

三个节点都能查看内容列表

[root@node1 conf]# moglistkeys --trackers=192.168.42.150:7001 --domain=images
1.gif
1.jpg
2.jpg
[root@node1 conf]# moglistkeys --trackers=192.168.42.151:7001 --domain=images
1.gif
1.jpg
2.jpg
[root@node1 conf]# moglistkeys --trackers=192.168.42.152:7001 --domain=images
1.gif
1.jpg
2.jpg

原创文章，作者：srayban，如若转载，请注明出处：http://www.178linux.com/78103