happyfish100/fastdfs

FastDFS is an open source high performance distributed file system. Its major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and video sharing site.

FastDFS Features

• grouped storage servers, simple and flexible
• peer-to-peer structure, no single point of failure
• The file ID is generated by FastDFS and serves as a credential for file access FastDFS does not require the traditional name server or meta server
• support large, medium, and small files well; supports the merged storage for small files and can store a massive number of small files
• the storage server supports multiple disks and supports data recovery for the single disk
• provide a nginx extension module that can seamlessly integrate with nginx
• support IPv6, NAT network and cross data center or hybrid cloud deployment
• support read-write separation and cross data center disaster backup
• provide truncate, append and modify APIs for appender files
• supports multi-threaded file upload and download, support resume from a broken point

Architecture Interpretation

FastDFS has two roles: tracker and storage. The tracker takes charge of scheduling and load balancing for file access. The storage store files and it's function is file management including: file storing, file syncing, providing file access interface. It also manage the meta data which are attributes representing as key value pair of the file. For example: width=1024, the key is "width" and the value is "1024".

The tracker and storage contain one or more servers. The servers in the tracker or storage cluster can be added to or removed from the cluster by any time without affecting the online services. The servers in the tracker cluster are peer to peer.

The storage servers organizing by the file group to obtain high capacity. The storage system contains one or more groups whose files are independent among these groups. The capacity of the whole storage system equals to the sum of all groups' capacity. A file group contains one or more storage servers whose files are same among these servers. The servers in a file group backup each other, and all these servers are load balancing. When adding a storage server to a group, files already existing in this group are replicated to this new server automatically, and when this replication done, system will switch this server online to providing storage services.

When the whole storage capacity is insufficient, you can add one or more groups to expand the storage capacity. To do this, you need to add one or more storage servers.

The identification of a file is composed of two parts: the group name and the file name.

Client test code use client library please refer to the directory: client/test.

Client SDK

• C Language: client/ subdir under FastDFS source code, header files default installation path is /usr/include/fastdfs/
• PHP extension: php_client/ subdir under FastDFS source code
• Java SDK: https://github.com/happyfish100/fastdfs-client-java
• Go SDK: https://github.com/qifengzhang007/fastdfs_client_go

For FastDFS installation and configuration documentation, please refer to the Wiki; For more FastDFS related articles, please subscribe the Wechat/Weixin public account (Chinese Language): fastdfs

See also

FastDFS is a lightweight object storage solution. If you need a general distributed file system for databases, K8s and virtual machines (such as KVM), you can learn about FastCFS which achieves strong data consistency and high performance.

We provide technical support service and customized development. Welcome to use WeChat or email for discuss.

email: 384681(at)qq(dot)com