fastdfs+nginx+tracker搭建互联网电商分布式图片服务器过程

文章由LinuxBoy分享于2019-03-25 10:03:10热评（297）

fastdfs+nginx+tracker搭建互联网电商分布式图片服务器过程

创业型的互联网公司，所以用开源软件自己搭建图片服务器，用来上传下载以及nginx转发负载均衡

一、在安装FastDFS之前必须先安装libevent，安装libevent步骤如下：

1.下载libevent：

wget https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz

`2.解压libevent：`

tar -zxvpf libevent-2.0.21-stable.tar.gz

`3.进入解压目录：`

cd libevent-2.0.21-stable

`4.指定安装目录：`

yum install -y libtool* yum install -y autoconf* yum install -y automake ./autogen.sh ./configure –prefix=/usr/local/libevent-2.0.21

`5.安装libevent：`

make make install

`二、安装FastDFS步骤如下：`

`1.下载FastDFS：`

wget http://fastdfs.googlecode.com/files/FastDFS_v4.06.tar.gz

`2.解压FastDFS：`

tar -xvf FastDFS_v4.06.tar.gz

`3.进入解压目录：`

cd FastDFS

`4.修改FastDFS的安装文件：`

vim make.sh TARGET_PREFIX=/usr/local/fastdfs —安装路径 TARGET_CONF_PATH=/etc/fdfs —配置文件路径 WITH_LINUX_SERVICE=1 —是否为linux服务

`5.指定libevent的lib目录：`

./make.sh C_INCLUDE_PATH=/usr/local/libevent-2.0.21/include LIBRARY_PATH=/usr/local/libevent-2.0.21/lib

`6.安装FastDFS：`

./make.sh install

`7.设置软连接(否则编译nginx时会出错)`

[root@zhitianstatic2 FastDFS]# ln -sf /usr/local/fastdfs/lib/libfastcommon.so /usr/local/lib/libfastcommon.so [root@zhitianstatic2 FastDFS]# ln -sf /usr/local/fastdfs/lib/libfastcommon.so /usr/lib/libfastcommon.so [root@zhitianstatic2 FastDFS]# ln -sf /usr/local/fastdfs/lib/libfastcommon.so /usr/lib64/libfastcommon.so

[root@zhitianstatic2 FastDFS]# ln -s /usr/local/fastdfs/lib/libfdfsclient.so /usr/local/lib/libfdfsclient.so [root@zhitianstatic2 FastDFS]# ln -s /usr/local/fastdfs/lib/libfdfsclient.so /usr/lib/libfdfsclient.so [root@zhitianstatic2 FastDFS]# ln -s /usr/local/fastdfs/lib/libfdfsclient.so /usr/lib64/libfdfsclient.so [root@zhitianstatic2 FastDFS]#

`三，安装依赖组件包`

yum install -y gcc gcc-c++ autoconf automake bzip2-devel ncurses-devel
yum install -y openssl libtool* zlib-devel libxml-devel libjpeg-devel ligpng-devel libtiff-devel
yum install -y openssl fontconfig-devel freetype-devel libXpm-devel gettext-devel curl curl-devel pam-devel e2fsprogs-devel krb5-devel libidn libidn-devel

`四，开始安装nginx Nginx和fastdfs-nginx-module`

`1，安装pcre`

yum install -y pcre-devel PS：有些yum安装成功，有些失败，如果失败了，在下一步骤直接源码方式安装pcre

准备组件 fastdfs-nginx-module tar -xvf fastdfs-nginx-module_v1.15.tar.gz

`2，安装nginx`

yum install -y openssl* tar -xvf pcre-8.31.tar.gz -C /usr/local/ tar -xvf fastdfs-nginx-module_v1.15.tar.gz tar -xvf nginx-1.4.2.tar.gz cd nginx-1.4.2 ./configure –with-http_stub_status_module –with-http_realip_module –with-http_ssl_module –prefix=/usr/local/nginx –add-module=/soft/fastdfs-nginx-module/src –with-pcre=/usr/local/pcre-8.31 或者 ./configure –prefix=/usr/local/nginx –conf-path=/usr/local/nginx/conf/nginx.conf –add-module=/soft/fastdfs-nginx-module/src

make make install

PS: –with-pcre=/usr/local/pcre-8.31，加这个会报错 make[1]: * [objs/nginx] Error 1

原文章地址：http://blog.csdn.net/mchdba/article/details/50933405，未经过原作者mchdba（黄杉）允许，谢绝转载。

`3，nginx编译包报错：`

/soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c: In function ‘ngx_http_fastdfs_handler’: /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:842: error: ‘struct fdfs_http_context’ has no member named ‘arg’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:843: error: ‘struct fdfs_http_context’ has no member named ‘header_only’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:844: error: ‘struct fdfs_http_context’ has no member named ‘url’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:845: error: ‘struct fdfs_http_context’ has no member named ‘output_headers’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:846: error: ‘struct fdfs_http_context’ has no member named ‘send_file’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:847: error: ‘struct fdfs_http_context’ has no member named ‘send_reply_chunk’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:847: error: ‘fdfs_send_reply_chunk’ undeclared (first use in this function) /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:848: error: ‘struct fdfs_http_context’ has no member named ‘proxy_handler’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:854: error: ‘struct fdfs_http_context’ has no member named ‘if_modified_since’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:857: error: ‘struct fdfs_http_context’ has no member named ‘if_modified_since’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:887: error: ‘struct fdfs_http_context’ has no member named ‘range’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:893: error: ‘struct fdfs_http_context’ has no member named ‘if_range’ /soft/fastdfs-nginx-module/src/ngx_http_fastdfs_module.c:893: error: ‘true’ undeclared (first use in this function) make[1]: * [objs/addon/src/ngx_http_fastdfs_module.o] Error 1 make[1]: Leaving directory `/soft/nginx-1.4.2’ make: * [install] Error 2

解决方案：

cd fastdfs-nginx-module/src
vi config
去掉/local  
CORE_INCS="$CORE_INCS /usr/local/fastdfs/include/fastdfs /usr/local/fastdfs/include/fastcommon/"
CORE_LIBS="$CORE_LIBS -L/usr/lib -lfastcommon -lfdfsclient"

改成这样是因为我们在安装fastdfs的时候，指定的目录是/usr/local/fastdfs而不是/usr/local/，所以要修改为我们安装的fastdfs的目录/usr/local/fastdfs/include/fastdfs

或者软连接： ln -s /usr/local/fastdfs/lib/libfastcommon.so /usr/lib64/libfastcommon.so ln -s /usr/local/fastdfs/lib/libfastcommon.so /usr/lib/libfastcommon.so ln -s /usr/local/fastdfs/lib/libfdfsclient.so /usr/lib64/libfdfsclient.so ln -s /usr/local/fastdfs/lib/libfdfsclient.so /usr/lib/libfdfsclient.so

然后重新进行解压缩编译问题就解决了。

`4，去启动nginx`

将libfastcommon.so、libfastcommon.so.1、libfdfsclient.so、libfdfsclient.so.1复制到/lib64/，否则启动会报错启动：/usr/local/nginx/sbin/nginx 启动报错： [root@zhitianstatic1 nginx-1.4.2]# /usr/local/nginx/sbin/nginx /usr/local/nginx/sbin/nginx: error while loading shared libraries: libfastcommon.so: cannot open shared object file: No such file or directory [root@zhitianstatic1 nginx-1.4.2]#

访问：在浏览器中输入nginx服务器ip地址即可。

`5，配置nginx.conf`

    location /group1/M00 {
        root   /home/storage/fastdfs/data;
        ngx_fastdfs_module;
    }

`6，启动nginx`

[root@mall_hc_244 nginx-1.4.2]# /usr/local/nginx/sbin/nginx ngx_http_fastdfs_set pid=15554 [root@mall_hc_244 nginx-1.4.2]#

`五，配置tracker server 在static2 10.250.11.32上面：`

`1. vim /etc/fdfs/tracker.conf`

bind_addr=192.168.227.128 base_path=/home/tracker/fastdfs

`2.配置防火墙：`

_ iptables -I INPUT -p tcp -m state –state NEW -m tcp –dport 22122 -j ACCEPT

`3.保存配置：`

/etc/init.d/iptables save

启动 tracker

/usr/local/fastdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf

在启动时报错：

/usr/local/bin/fdfs_trackerd: error while loading shared libraries: libevent-2.0.so.5: cannot open shared object file: No such file or directory

执行下面的命令即可正常启动：

echo ‘/usr/local/libevent-2.0.21/include/’ >> /etc/ld.so.conf echo ‘/usr/local/libevent-2.0.21/lib/’ >> /etc/ld.so.conf ldconfig [root@mall_hc_244 nginx-1.4.2]# echo ‘/usr/local/libevent-2.0.21/include/’ >> /etc/ld.so.conf [root@mall_hc_244 nginx-1.4.2]# echo ‘/usr/local/libevent-2.0.21/lib/’ >> /etc/ld.so.conf [root@mall_hc_244 nginx-1.4.2]# ldconfig [root@mall_hc_244 nginx-1.4.2]# /usr/local/fastdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf [2016-01-22 07:10:35] ERROR - file: tracker_func.c, line: 191, “/home/tracker/fastdfs” can’t be accessed, error info: No such file or directory [2016-01-22 07:10:35] INFO - local_host_ip_count: 2, 127.0.0.1 192.168.121.244 [2016-01-22 07:10:35] CRIT - exit abnormally! [root@mall_hc_244 nginx-1.4.2]# mkdir -p /home/tracker/fastdfs [root@mall_hc_244 nginx-1.4.2]#

`5. 查看监听程序：`

netstat -ntpl | grep fdfs

`6.添加开机启动命令行：`

vim /etc/rc.d/rc.local /usr/local/fastdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

`六、配置 storage server 在static1 10.250.11.31上面`

`1. vim /etc/fdfs/storage.conf`

group_name=group1 bind_addr=10.250.11.31 base_path=/home/storage/fastdfs store_path0=/home/storage/fastdfs tracker_server=10.250.11.32:22122 http.server.port=8080

`2. 复制一份`

[root@mall_hc_244 nginx-1.4.2]# cp /soft/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/ [root@mall_hc_244 nginx-1.4.2]#

`3. vim /etc/fdfs/mod_fastdfs.conf`

base_path=/home/storage/fastdfs group_name=group1

store_path0=/home/storage/fastdfs

tracker_server=10.250.11.32:22122：

url_have_group_name = true group_count = 0

`4. vim /usr/local/nginx/conf/nginx.conf`

location /group1/M00 { root /home/storage/fastdfs/data; ngx_fastdfs_module; }

`5.配置防火墙：`

iptables -I INPUT -p tcp -m state –state NEW -m tcp –dport 23000 -j ACCEPT

`6.保存配置：`

/etc/init.d/iptables save

`7.启动 storage`

/usr/local/fastdfs/bin/fdfs_storaged /etc/fdfs/storage.conf
[root@mall_hc_244 nginx-1.4.2]# /usr/local/fastdfs/bin/fdfs_storaged /etc/fdfs/storage.conf
[2016-01-22 07:21:41] ERROR - file: trunk_mgr/trunk_shared.c, line: 177, "No such file or directory" can't be accessed, error info: /home/storage/fastdfs
[2016-01-22 07:21:41] CRIT - exit abnormally!

[root@mall_hc_244 nginx-1.4.2]# ll /home/storage/fastdfs
ls: cannot access /home/storage/fastdfs: No such file or directory
[root@mall_hc_244 nginx-1.4.2]# mkdir -p /home/storage/fastdfs
[root@mall_hc_244 nginx-1.4.2]# /usr/local/fastdfs/bin/fdfs_storaged /etc/fdfs/storage.conf
data path: /home/storage/fastdfs/data, mkdir sub dir...
mkdir data path: 00 ...
mkdir data path: 01 ...
mkdir data path: 02 ...
mkdir data path: 03 ...
mkdir data path: 04 ...
mkdir data path: 05 ...
mkdir data path: 06 ...
mkdir data path: 07 ...
mkdir data path: 08 ...
mkdir data path: 09 ...
mkdir data path: 0A ...
mkdir data path: 0B ...
mkdir data path: 0C ...
.......................................................
mkdir data path: FD ...
mkdir data path: FE ...
mkdir data path: FF ...
data path: /home/storage/fastdfs/data, mkdir sub dir done.
[root@mall_hc_244 nginx-1.4.2]#

`8. 查看监听程序：`

netstat -ntpl | grep fdfs

查看日志文件：

cat /home/storage/fastdfs/logs/storaged.log [root@mall_hc_244 nginx-1.4.2]# cat /home/storage/fastdfs/logs/storaged.log [2016-01-22 07:22:04] INFO - FastDFS v4.06, base_path=/home/storage/fastdfs, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=192.168.121.244, client_bind=1, max_connections=256, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=80, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2016-01-22 07:22:09] INFO - file: storage_param_getter.c, line: 187, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, store_slave_file_use_link=0 [2016-01-22 07:22:09] INFO - file: storage_func.c, line: 174, tracker_client_ip: 192.168.121.244, my_server_id_str: 192.168.121.244, g_server_id_in_filename: -193353536 [2016-01-22 07:22:09] INFO - local_host_ip_count: 2, 127.0.0.1 192.168.121.244 [2016-01-22 07:22:09] INFO - file: tracker_client_thread.c, line: 308, successfully connect to tracker server 192.168.121.244:22122, as a tracker client, my ip is 192.168.121.244

`七、测试：`

[root@mall_hc_244 nginx-1.4.2]# /usr/local/fastdfs/bin/fdfs_test /etc/fdfs/client.conf upload /etc/passwd This is FastDFS client test program v4.06

FastDFS may be copied only under the terms of the GNU General Public License V3, which may be found in the FastDFS source kit. Please visit the FastDFS Home Page http://www.csource.org/ for more detail.

[2016-01-22 07:23:18] ERROR - file: ../client/client_func.c, line: 257, “/home/yuqing/fastdfs” can’t be accessed, error info: No such file or directory [root@mall_hc_244 nginx-1.4.2]#

修改配置文件:/etc/fdfs/client.conf： base_path=/home/storage/fastdfs tracker_server=192.168.121.244:22122

关闭防火墙： iptables -I INPUT -p tcp -m state –state NEW -m tcp –dport 8080 -j ACCEPT

再次开始测试： [root@mall_hc_244 nginx-1.4.2]# /usr/local/fastdfs/bin/fdfs_test /etc/fdfs/client.conf upload /tmp/1.png This is FastDFS client test program v4.06

[2016-01-22 07:27:29] DEBUG - base_path=/home/storage/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

tracker_query_storage_store_list_without_group: server 1. group_name=, ip_addr=192.168.121.244, port=23000

group_name=group1, ip_addr=192.168.121.244, port=23000 storage_upload_by_filename group_name=group1, remote_filename=M00/00/00/wKh59FahaWGAQIobAAADWBdhqfs1165318 source ip address: 192.168.121.244 file timestamp=2016-01-22 07:27:29 file size=856 file crc32=392276475 file url: http://192.168.121.244:8080/group1/M00/00/00/wKh59FahaWGAQIobAAADWBdhqfs1165318 storage_upload_slave_by_filename group_name=group1, remote_filename=M00/00/00/wKh59FahaWGAQIobAAADWBdhqfs1165318_big source ip address: 192.168.121.244 file timestamp=2016-01-22 07:27:29 file size=856 file crc32=392276475 file url: http://192.168.121.244:8080/group1/M00/00/00/wKh59FahaWGAQIobAAADWBdhqfs1165318_big [root@mall_hc_244 nginx-1.4.2]#

启动命令： /usr/local/fastdfs/bin/restart.sh /usr/local/fastdfs/bin/fdfs_trackerd /etc/fdfs/tracker.conf /usr/local/fastdfs/bin/restart.sh /usr/local/fastdfs/bin/fdfs_storaged /etc/fdfs/storage.conf /usr/local/nginx/sbin/nginx -s reload

测试图片上传分发： /usr/local/fastdfs/bin/fdfs_test /etc/fdfs/client.conf upload /tmp/1.png

做软连接： ln -s /home/storage/fastdfs/data /home/storage/fastdfs/data/M00

`八，同步开发的svn的静态资源到nginx`

安装svn客户端：yum install -y svn 然后开始同步：svn checkout https://192.168.121.206/svn/Fusion/trunk/yueworld_static /usr/local/nginx/yueworld_static/ 这个要输入p同意checkout然后输入连接svn的用户名密码 [root@mall_hc_244 yueworld_static]# svn checkout https://192.168.121.206/svn/Fusion/trunk/yueworld_static /usr/local/nginx/yueworld_static/

`九，问题`

`1，问题1：`

libtool: compile: unrecognized option `-DHAVE_CONFIG_H'
libtool: compile: Try `libtool --help' for more information.
make[1]: *** [pcrecpp.lo] Error 1
make[1]: Leaving directory `/home/guangbo/work/pcre-8.12'
make: *** [all] Error 2

缺少gcc-c++和libtool，也就是c++编译包解决方法：需要先安装libtool和gcc-c++ #yum -y install libtool #yum -y install gcc-c++

`2，问题2error log:`

2016/01/23 04:20:49 [error] 7875#0: *4 open() “/usr/local/nginx/html/group1/M00/00/00/wKh59FaijDCAF92yAAAWHOdjXIg765_big.png” failed (2: No such file or directory), client: 192.168.120.249, server: localhost, request: “GET /group1/M00/00/00/wKh59FaijDCAF92yAAAWHOdjXIg765_big.png HTTP/1.1”, host: “192.168.121.244”

access log: 192.168.120.249 - - [23/Jan/2016:04:20:49 +0800] “GET /group1/M00/00/00/wKh59FaijDCAF92yAAAWHOdjXIg765_big.png HTTP/1.1” 404 570 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36”

有如下2个原因：

1)，软连接没有创建创建M00目录的软连接 [root@storage1]# ln -s /home/storage/fastdfs/data /home/storage/fastdfs/data/M00

2)，nginx没有生效，需要重新安装

`十，配置nginx.conf`

  worker_processes  8;

    error_log /usr/local/nginx/logs/nginx_error.log crit;
    pid /usr/local/nginx/nginx.pid;

    worker_rlimit_nofile 65535;

    events {
         use epoll;
         worker_connections  65535;
    }
    http {
        include mime.types;
        default_type application/octet-stream;

         #charset gb2312;

         server_names_hash_bucket_size 128;  
         client_header_buffer_size 32k;  
         large_client_header_buffers 4 32k;  
         client_max_body_size 8m;

         sendfile        on;
         tcp_nopush     on;

         keepalive_timeout  60;

         tcp_nodelay on;  

         fastcgi_connect_timeout 300;  
         fastcgi_send_timeout 300;  
         fastcgi_read_timeout 300;  
         fastcgi_buffer_size 64k;  
         fastcgi_buffers 4 64k;  
         fastcgi_busy_buffers_size 128k;  
         fastcgi_temp_file_write_size 128k;  

         gzip on;  
         gzip_min_length  1k;  
         gzip_buffers     4 16k;  
         gzip_http_version 1.0;  
         gzip_comp_level 2;  
         gzip_types  text/plain application/x-javascript text/css application/xml;  
         gzip_vary on;  

         #limit_zone  crawler  $binary_remote_addr  10m;

        server {
            listen       80;
            server_name  www.nginxtest.com 192.168.121.102;
            index index.html index.htm index.jsp index.do;
            root /usr/local/nginx/html/myloan;
           #配置Nginx动静分离，定义的静态页面直接从Nginx发布目录读取。
            location ~ .*\.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ 
             { 
                  root /usr/local/nginx/html/myloan; 
                  #expires定义用户浏览器缓存的时间为7天，如果静态页面不常更新，可以设置更长，这样可以节省带宽和缓解服务器的压力
                   expires      7d; 
               } 
          #所有jsp、do的动态请求都交给后面的tomcat处理 
          location ~ (\.jsp)|(\.do)$ 
         { 

              #tomcat地址
              proxy_pass http://192.168.121.102:8080;  
              proxy_redirect off;  
              proxy_set_header HOST $host;  
              proxy_set_header X-Real-IP $remote_addr;  
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  
              client_max_body_size 10m;  
              client_body_buffer_size 128k;  
              proxy_connect_timeout 90;  
              proxy_send_timeout 90;  
              proxy_read_timeout 90;  
              proxy_buffer_size 4k;  
              proxy_buffers 4 32k;  
              proxy_busy_buffers_size 64k;  
              proxy_temp_file_write_size 64k;  
       }      

         log_format  access  '$remote_addr - $remote_user [$time_local] "$request" '  
              '$status $body_bytes_sent "$http_referer" '  
              '"$http_user_agent" $http_x_forwarded_for';  
         access_log  /usr/local/nginx/logs/access.log  access;  

     }
     server  
      {  
          listen  80;  
          server_name  www.server.nginxtest.com;  

          location / {  
          stub_status on;  
           access_log   off;  
      }  
   }
}

推荐文章：

fastdfs+nginx+tracker搭建互联网电商分布式图片服务器过程