Node failures

Prepare the replacement nodeCopy Anchor Link

• Hardware Make sure that the replacement node's hardware (CPU, memory, network, and drive type) roughly matches the failed node. Higher-spec hardware does not hurt cluster performance. Run the same operating system, kernel, and library versions as the other nodes so the replacement does not behave differently from the rest of the cluster.

• Exclusive drive access As with physical drive operations, RustFS requires exclusive access to its storage volumes. Do not let other processes or scripts modify data on these volumes directly, as this can corrupt data or reduce redundancy.

Update hostname and network resolutionCopy Anchor Link

• DNS or /etc/hosts If the replacement node's IP address differs from the failed node, point the old node's hostname (for example, rustfs-node-2.example.net) to the new node's IP address so that cluster nodes can still reach each other by the same name.

# Example: Add or modify line in /etc/hosts
192.168.1.12 rustfs-node-2.example.net

Use ping or nslookup to verify that the hostname resolves to the new node.

Deploy and configure RustFS servicesCopy Anchor Link

• Install the same version Install the same RustFS version as the existing nodes, using the same installation method and directory. Make sure that startup scripts, environment variables, and configuration files (such as /etc/default/rustfs) are identical to those on the other nodes.

• Check the configuration

• Check that RUSTFS_VOLUMES in /etc/default/rustfs is byte-for-byte identical to the other nodes, and that it covers the replacement node's hostname and port.

• Make sure that all nodes use the same access keys and permission settings; otherwise, the new node fails to authenticate and cannot join the cluster.

Rejoin the cluster and trigger data healingCopy Anchor Link

• Start the service

systemctl start rustfs

Alternatively, start RustFS with your own startup script. Then run journalctl -u rustfs -f to confirm that the new node has detected the other nodes and started healing.

• Watch healing Healing runs in the background once the node rejoins; no manual command is required. Watch its progress through the service logs on the replacement node, and check node and disk status in the RustFS Console:

journalctl -u rustfs -f

• Community observations In community testing, when a node goes offline and rejoins, RustFS heals only that node instead of rebalancing the entire cluster. This avoids unnecessary network and I/O spikes.

Follow-up checks and best practicesCopy Anchor Link

• Monitoring and Alerts

• During healing, monitor disk and network load to make sure the cluster still meets its read, write, and bandwidth requirements.

• Set up alerts to notify operations teams when node healing fails or progress stalls beyond thresholds.

• Regular failure testing Run regular node failure drills and practice the full recovery procedure so your team is familiar with the commands and incident response process.

• Find the cause Run detailed hardware diagnostics (for example, SMART data and BIOS logs) on nodes or drives that fail repeatedly, and put a preventive maintenance plan in place.

• Ask for help For in-depth troubleshooting and recovery help, contact the RustFS team or the community.

These steps replace a failed node and heal its data. Adapt the commands to your environment, and keep the configuration identical across nodes.