Prerequisites:
- Three or more NAS devices
- Each NAS has dual network ports
- Each NAS has similar physical disk sizes
- NAS BIOS supports power-on auto-start
- NAS system disk is SSD or NVMe, and data disk is HDD, or you can distinguish between system and data disks to avoid mistakes during high availability testing
A simple home network diagram is shown below:
The optical modem is connected to NAS network port 1 through a gigabit switch.
The network ports 2 of the three NAS devices are connected to a 2.5G switch to form an internal network.
Ensure that each NAS has a separate vmbr0 and vmbr1 set in Proxmox.
The bound network ports should also be consistent on each NAS.
vmbr0 is shared with virtual machines, so it is for LAN.
vmbr1 is for WAN, connected to the optical modem.
Ensure that the LAN IP set on each NAS is in the same network segment to form a cluster.
After forming the cluster, choose one node to create an ikuai virtual machine.
Add LAN and WAN, with LAN on top (which is vmbr0).
The virtual machine disk can be set to a smaller size, such as 8GB, and can be placed on any of the three NAS devices.
Install ikuai and after installation, set the LAN network segment to be consistent with the NAS, and then dial up. After dialing up, all three NAS devices should have internet access. Update the system on each device and then restart.
The above steps are only to ensure that there is internet access when installing Ceph.
After forming the cluster, follow the steps to install Ceph in the data center's Ceph section. It is recommended to choose:
osd_pool_default_min_size = 2
osd_pool_default_size = 2
Add the administrator and monitor to all three nodes.
After installing large-capacity mechanical disks on each node, initialize them.
Add them as OSDs.
Create a Ceph FS.
In the data center storage, create an RBD.
Another important point to note is that the time of each node in the Ceph storage needs to be synchronized regularly to avoid inconsistent clocks. Therefore, you need to install systemd-timesyncd on each node in the system page.
Log in to each node and run crontab -e.
*/10 * * * * /usr/sbin/ntpdate ntp.aliyun.com
This will synchronize the time every ten minutes.
Now, you can enjoy playing with hyper-convergence. For example, for the ikuai dial-up virtual machine we just created, we can move its disk from the local storage to the newly created distributed storage Ceph, and then delete the source and restart.
Due to the consistent NAS models and network ports, a magical effect can be achieved. My ikuai dial-up virtual machine can achieve seamless switching between physical nodes in seconds (other virtual machines can also do this). The main point is to migrate the data in memory, and since the storage is on the distributed storage Ceph, there is no need to migrate it. For LXC in template mode, it requires a restart after migration due to limitations. This can be considered as achieving seamless live migration.
Configure High Availability (HA)
Next, it's simple. You can freely experiment, such as creating a virtual machine (with both the virtual machine disk and data disk on the Ceph RBD) and setting up SMB sharing.
Test High Availability:
Randomly unplug a data disk, and you will find that your SMB sharing still works without any issues. The virtual machines run without any issues, and you can still read and write data through SMB. In the Ceph tab, there will be a warning indicating that an OSD is offline. After a while, the data will be written and when you plug the data disk back in, the OSD will refresh its status as available and automatically balance the data that was written.
Ultimate Test:
Randomly turn off one NAS device to simulate a failure. If HA is configured and there are virtual machines or LXC containers on the turned-off device, they will automatically migrate to the surviving nodes and start running after a brief retry. Your SMB sharing will also work normally and you can read and write data. After a while, when you restart the turned-off NAS device, Ceph will automatically start balancing.
Of course, more extreme situations, such as all three NAS devices failing or two failing at the same time, are not within our consideration. In that case, a higher level of remote multi-activity is required.
In a home environment, if NAS is used to store important data such as code, family photos, blog articles, and keys, relying on a single machine's hardware RAID or software RAID carries certain risks. Even if it can be successfully rebuilt, the rebuilding time can be quite long for today's common mechanical hard drives of 10TB or more. During this time, your storage service will be completely unavailable, and whether the rebuild can be successful is also a concern. This situation is obviously unacceptable.
Therefore, distributed storage systems like Ceph have great potential in the home market. It is expected that manufacturers can try multi-node solutions to meet the needs of geek users. Adopting a multi-node solution can also provide greater scalability, as the disk slots of a single node are limited.
Even for the most geeky users, in a home environment, they should not rely solely on RAID. RAID is just a fault-tolerant mechanism, not a backup. The enterprise-level solution is to use distributed storage, even with just three nodes, it can be part of a distributed system.