→Home Lab | →Back | →Gateway

Network Wide Power Fail Infrastructure W Nut

Smart Home Edge Systems

(smart-home-edge-systems.us)

Figure 1: Your Home in the Clouds!

Configuring a Network Wide Power Fail Infrastructure using Network UPS Tool.

For a Network UPS Tools (NUT) deployment where the UPS is physically connected via USB, to the main Promox server you have two primary options within your Proxmox VE topology.

Because the UPS is a critical infrastructure component that dictates the shutdown sequence of your entire physical host (including all your VMs and LXCs), the ideal architecture is to install the NUT server directly on the Proxmox Host (PVE Host, 192.168.86.250).

UPS Direct Connection to the PVE Host Overview

In this configuration, the physical USB cable goes from the UPS into the Kamrui Mini PC, and the bare-metal Proxmox Debian layer handles the driver (usbhid-ups).

  • Fail-Safe Shutdowns: If the battery runs critically low, the host has direct control to gracefully stop your LXCs/VMs and then safely power itself down. If the NUT server is inside a virtualized container and that container crashes or hangs, the host won't receive the shutdown signal.

  • Network Independence: If your internal virtual switch (vmbr0) or local routing encounters an issue during a power event, the host can still talk to the UPS locally over the USB bus without needing network layers to stay up just to read its own power status.

  • Server-Client Topology: The PVE host acts as the NUT Master (Server). It can then easily serve data across 192.168.86.0/24 to NUT Slaves (Clients) like your Home Assistant OS instance (VM 101) for dashboards and automations.

  • Compute Isolation: Keeping Proxmox alone on UPS 1 eliminates heavy load draw (spinning drives, PoE switches), maximizing runtime so your core logic stays alive as a command node to coordinate everything else.

  • Circuit Segmentation: Spreading 4 UPS units across independent wall outlets prevents a sudden high-power draw (like a GPU spike on UPS 3) from tripping a breaker that takes down your network or server infrastructure.

  • Storage Protection: Enforcing a soft shutdown on the Synology NAS units at 60% battery preserves Btrfs/EXT4 file systems and prevents degraded RAID arrays or NFS stale-handle deadlocks.

  • Continuous Network Fabric: Network gear on UPS 2 remains powered long enough to let all remote nodes (Proxmox, Windows, ESP32, NAS) communicate telemetry until their final shutdown phase.

Figure 2: System Level UPS Setup

Architectural Weaknesses & Edge Cases

Table 1: UPS Power Edge Cases
CRITICAL OUTAGE RISKS (Edge Cases)
Edge Case Resulting System Vulnerability
1. Synology "Safe Mode" Loop NAS halts services/unmounts volumes but stays powered. If grid returns, NAS won't auto-reboot.
2. Partial Outage / Bounce RPi/NAS shut down at 60%, grid restores at 50%. Proxmox never hit 20%, so system is half-dead.
3. Master-Client Dependency Synology NAS 2 on UPS 2 relies on Proxmox on UPS 1 over Ethernet on UPS 2 to receive triggers.

Synology DSM by default enters Safe Mode (unmounts volumes and shuts down services to protect data) but leaves the motherboard powered on until the UPS physically cuts input power.

  • The Problem: If the grid returns after the NAS enters Safe Mode, but before UPS 2 dies completely, UPS 2 never power-cycles its outlets.

  • Result: The Synology units remain powered on in a permanently halted state. They will not reboot automatically when grid power normalizes until someone manually presses the physical power buttons.

Suppose a storm drops utility power. UPS 2 drains to 60%, triggering the Synology NAS units and ESP32 to shut down. Then power comes back on while UPS 1 is still at 40% (meaning Proxmox never received its 20% shutdown threshold and stayed running).

  • The Problem: Proxmox never shut down, but its NFS storage backend (Synology NAS 2) did shut down.

  • Result: Proxmox VM storage drops offline, throwing I/O errors and corrupting running virtual machines because the host never rebooted to trigger storage re-initialization.

Synology NAS units are plugged into UPS 2, but they monitor UPS 1 via Proxmox (192.168.86.250) over the network switch (powered by UPS 2).

  • If UPS 2 fails prematurely (or its network switch fails), the Synology units lose connection to Proxmox. Without explicit deadtime watchdog settings in upsmon.conf, the Synology NAS units won't know whether power is fine or if Proxmox died, potentially missing their shutdown window.

To make this setup self-healing, the reboot sequence must mirror the shutdown sequence in reverse, driven by hardware power-cycling.

Code 2: UPS Power Ideal Recovery Sequence

IDEAL AUTOMATED RECOVERY SEQUENCE

[ Utility Power Restored ]
│
▼
1. Mains Voltage Stabilizes ──► [ UPS 2 (Network) ] & [ UPS 1 (Compute) ]
│
├─► UPS 2 powers on outlets ──► Router, Switches & Pi-Hole boot up
│ (Network Fabric Restored)
│
├─► UPS 1 delays output 5-10s ─► Kamrui Mini PC auto-boots (BIOS "Restore AC Power")
│ (Proxmox Host / NUT Master comes online)
│
└─► Synology NAS Units ─────────► Controlled via Wake-on-LAN (WOL) or

  1. Enable "Auto-Reboot when Power Restores" on Synology: In DSM → Control Panel → Hardware & Power, check "Restart automatically when power supply issue is fixed". Additionally, check "Shut down UPS when system enters Safe Mode". This forces Synology to tell UPS 2 to kill power to its outlets once Safe Mode is reached, ensuring a complete power cycle when grid power returns.

  2. Configure BIOS AC Power Loss Settings: Set the Kamrui Mini PC (Proxmox), Raspberry Pi, and Win 11 PC BIOS setting State After G3 / AC Power Loss to Power On (or Last State).

  3. Proxmox Storage Watchdog Script: Add an automated systemd timer on Proxmox that periodically pings/checks the Synology NFS share. If the NFS mount goes stale during a partial outage bounce, the watchdog script cleanly restarts Proxmox storage services or triggers a self-reboot once the NAS comes back online.

Since Proxmox (UPS 1) stays online during a partial outage where UPS 2 drops to 60% and shuts down the Synology units, Proxmox can act as an automated recovery orchestrator.

Because your Synology NAS units are in Safe Mode/halted while plugged into live power, they cannot turn themselves back on automatically. Proxmox can monitor their availability and issue Wake-on-LAN (WOL) magic packets over the restored network to wake them up without requiring a physical power cycle or button press.

Proxmox runs a lightweight background script that continuously monitors your infrastructure state:

Code 3: UPS Power Edge Case Recovery Sequence

EDGE CASE AUTOMATED RECOVERY FLOW

[ UPS 2 Hits 60% ] ──► Synology NAS Units Safe-Mode & Halt
│
▼
[ Grid Power Returns ] ──► Network Switches & Gateway Boot Up
│
▼
[ Proxmox Watchdog ] ──► Detects Network Gateway is reachabile AGAIN
├─► Detects Synology NFS/IP is DOWN
├─► Issues Wake-on-LAN (WOL) Magic Packets to NAS MACs
└─► Waits 5-10 mins for Synology Boot ──► Remounts NFS Storage

Because Proxmox virtual machines might throw I/O errors if their underlying NFS storage disappears during the NAS reboot window, you can refine how Proxmox handles storage mounts:

  1. Pause/Suspend VMs on NFS: When Proxmox detects the 60% trigger, have it suspend running NFS-dependent VMs to RAM rather than hard-stopping them.

  2. Disable NFS Storage Plugin: Temporarily mark the Synology NFS storage as disabled in Proxmox (pvesm set <storage> --disable 1). This prevents Proxmox from locking up while waiting for stale NFS socket timeouts.

  3. Re-Enable Post-WOL: Once the script sends the WOL packet and sleeps for the 5–10 minute boot duration, it re-enables the storage (pvesm set <storage> --disable 0), mounts the NFS shares cleanly, and resumes the paused VMs.

Textual and Structural Layout

The complete textual and structural layout of your multi-UPS infrastructure.

This design separates your core computing layer (Proxmox) from your network and storage dependencies, ensuring your infrastructure stays alive and manageable for as long as possible during a power event.

  • Dedicated to: Proxmox VE Host (Kamrui Mini PC)

  • Physical Data Link: A USB communication cable connects directly from UPS 1 into a USB port on the Proxmox Host.

  • Role: This UPS only powers the low-draw Mini PC. Because it isn't burdened by spinning hard drives or power-hungry PoE cameras, it provides a maximum runtime buffer for the Proxmox host to execute its network watchdog scripts and manage the system shutdown.

  • Control Signaling:

    • The Proxmox Host runs the NUT Master (Server) daemon, monitoring the battery status of UPS 1 over the USB connection.

    • It shares this power status out over the local network via Ethernet at 192.168.86.250.

  • Dedicated to: All supporting equipment and network appliances.

  • Power Connections:

    • Internet Modem (Plugged into the SmartThings Z-Wave Smart Plug ──► UPS 2 Battery Outlets)

    • Wi-Fi Router / EERO Nest Primary Node

    • Core Network Switch

    • PoE Network Switches (Powering IP surveillance cameras)

    • Synology NAS 1 (Surveillance Station storage)

    • Synology NAS 2 (NFS/Data storage)

    • SmartThings Hub

    • Raspberry Pi (Backup Pi-hole)

  • Role: Keeps your entire local network fabric, Wi-Fi mesh, smart home mesh, and data storage fully active.

  • Control Signaling:

    • The Synology NAS units are configured as NUT Secondary (Slave) clients.

    • They continuously poll the Proxmox Host (192.168.86.250) over the local Ethernet network.

    • Even though they are plugged into UPS 2 for power, they listen to the battery thresholds and shutdown triggers broadcast by Proxmox. When the designated threshold is reached (e.g., UPS 2 charge drops to 60%), the Synologies execute their countdown to enter Safe Mode, saving their disk volumes from corruption long

  • Dedicated to: Windows 11 Desktop

  • Physical Data Link: A direct USB communication cable connects UPS 3 to a USB port on the Windows 11 PC.

  • Role: Powers your primary workstation independently. Isolating the desktop onto its own outlet and battery prevents high-power components (like GPU power spikes) from draining or tripping the core infrastructure battery banks.

  • Control Signaling:

    • Runs WinNUT-Client (or local CyberPower PowerPanel / NUT Windows service) locally over USB.

    • Operates as an isolated node that triggers a graceful Windows shutdown when UPS 3 battery runtime drops below safety thresholds, or listes for network heartbeat signals from Proxmox.

  • Dedicated to: Main Living Room TV, Nvidia Shield, and Secondary Mesh Network

  • Power Connections:

    • Wall Screen LCD TV

    • Nvidia Shield Media Player

    • Secondary EERO Mesh Router Node

  • Physical Data Link: A USB communication cable connects directly into an ESP32-S3 Microcontroller running an embedded NUT daemon / custom USB host code.

  • Role: Keeps your streaming media and secondary Wi-Fi mesh node online through short flickers and brownouts without risking filesystem loss on storage servers.

  • Control Signaling:

    • The ESP32 Node acts as a lightweight network telemetry server over Wi-Fi, streaming battery health metrics to Home Assistant or Proxmox via MQTT / HTTP sockets.

    • When UPS 2 broadcasts a 60% battery threshold signal across the network, the ESP32 receives the trigger to cleanly shut down media sessions or signal power-down events to connected home entertainment gear.

Code 4: UPS Power Distribution

========================= INDEPENDENT CIRCUIT & TELEMETRY MATRIX =========================

[ Outlet 1 ] ──► [ UPS 1: Compute ] ──(USB)──► Kamrui Mini PC (Proxmox / NUT Master)
│ │
├─► Kamrui Mini PC ├─► Primary PiHole LXC/VM
├─► SmartThings Hub (Backup) └─► Exports upsd @ 192.168.86.250
└─► PDU / Power Strip


[ Outlet 2 ] ──► [ UPS 2: Network ] ──(USB)──► Raspberry Pi (Backup PiHole / NUT Server)
│ │
├─► Modem, Router & Core Switch └─► Local USB upsd server
├─► EERO Primary Mesh & PoE Switches
├─► SmartThings Primary Hub
├─► Synology NAS (Surveillance) ──(NUT Client pointing to Pi/Proxmox)
├─► Synology NAS (NFS Data) ──(NUT Client pointing to Pi/Proxmox)
└─► Raspberry Pi (Backup PiHole)


[ Outlet 3 ] ──► [ UPS 3: Workstation ] ──(USB)──► Windows 11 Desktop
│ │
└─► Win 11 PC └─► WinNUT / USB monitoring


[ Outlet 4 ] ──► [ UPS 4: Entertainment ] ──(USB)──► ESP32 Node
│ │
├─► Secondary EERO Mesh └─► WiFi Telemetry / NUT Exporter
├─► Wall Screen LCD TV
└─► Nvidia Shield

Tiered Outage Execution Timeline

Table 5: Tiered Outage Execution Timeline
Trigger Monitoring Node Targets Automated Action
UPS 2 @ 60% Raspberry Pi (Backup Pi-hole) Both Synology NAS units Triggers graceful volume unmount and system halt via upsc / upsmon network signals.
UPS 2 @ 60% Raspberry Pi (Backup Pi-hole) ESP32 Node (UPS 4) Sends custom network payload to ESP32 to trigger optional soft-shutdown on TV/Shield node.
UPS 2 @ 40% Raspberry Pi (Backup Pi-hole) RPi Host & UPS 4 hardware Initiates OS shutdown for the Raspberry Pi itself; sends final power-off signal to UPS 4.
UPS 1 @ 20% Proxmox Host (192.168.86.250) Kamrui Mini PC & VMs Triggers graceful guest VM/LXC shutdowns followed by Proxmox host OS shutdown.
UPS 1 @ 10% Proxmox Host (192.168.86.250) UPS 1 Hardware Issue upsdrvctl shutdown / load.off command to kill output outlets and preserve battery safety margin.

Final setup procedures

Here is the step-by-step setup guide to implement your 4-UPS architecture, percentage-based shutdown triggers, Wake-on-LAN recovery, and client node configurations.

The Kamrui Mini PC monitors UPS 1 via USB host and acts as the central upsd server (192.168.86.250) for remote clients.

Run the following in the Proxmox shell:

Code 6: UPS Proxmox Nut Install

Bash
apt update && apt install -y nut wakeonlan

Set NUT to operate as a netserver:

Code 7: UPS Proxmox Nut Configure

Ini, TOML
MODE=netserver

Add the CyberPower configuration with forced low-battery overrides:

Code 8: UPS Proxmox Nut Driver

Ini, TOML
[ups1]
driver = usbhid-ups
port = auto
desc = "Compute UPS 1 - Kamrui Mini PC"
override.battery.charge.low = 20
override.battery.charge.warning = 30

Listen on all interfaces (or specifically 192.168.86.250):

Code 9: UPS Proxmox Nut Driver Binding

Ini, TOML
LISTEN 0.0.0.0 3493

Configure authentication credentials for remote clients:

Code 10: UPS Proxmox Nut Users

Ini, TOML
[admin]
password = SuperSecretPassword
actions = SET
instcmds = ALL

[monuser]
password = SecretClientPassword
upsmon master

Edit the /etc/nut/upsmon.conf with the following contents.

Code 11: UPS Proxmox Nut Local Monitor

Ini, TOML
MONITOR ups1@localhost 1 monuser SecretClientPassword master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h now"
POWERDOWNFLAG /etc/killpower

Execute the following commands on the Promox Console.

Code 12: UPS Proxmox Nut Start Services

Bash
systemctl restart nut-server nut-client
systemctl enable nut-server nut-client

The Raspberry Pi monitors UPS 2 directly via USB and executes the 60% and 40% threshold triggers.

Run the following on the Raspberry Pi.

Code 13: UPS PiHole Nut Install

Bash
sudo apt update && sudo apt install -y nut

Edit the /etc/nut/ups.conf & /etc/nut/nut.conf and restart the daemon as follows:

Code 14: UPS PiHole Nut Configure

In /etc/nut/nut.conf:
Ini, TOML
MODE=netserver

In /etc/nut/ups.conf:
Ini, TOML
[ups2]
driver = usbhid-ups
port = auto
desc = "Infrastructure UPS 2 - Network & Storage"

In /etc/nut/upsd.conf:
Ini, TOML
LISTEN 0.0.0.0 3493

In /etc/nut/upsd.users:
Ini, TOML
[monuser]
password = SecretClientPassword
upsmon master

Restart services:
Bash
sudo systemctl restart nut-server

Create the custom trigger script:

Code 15: UPS PiHole Nut Monitoring Script

Bash
#!/bin/bash
CHARGE=$(upsc ups2@localhost battery.charge 2>/dev/null)
STATUS=$(upsc ups2@localhost ups.status 2>/dev/null)

# Execute only if running on battery power
if [[ "$STATUS" == *"OB"* ]]; then
if [ "$CHARGE" -le 60 ] && [ ! -f /tmp/ups2_60_triggered ]; then
# Signal Synology NAS units via SSH or Safe Mode command
ssh -o StrictHostKeyChecking=no admin@192.168.86.X "synopoweroff"
ssh -o StrictHostKeyChecking=no admin@192.168.86.Y "synopoweroff"
# Optional: Send HTTP / MQTT call to ESP32 (UPS 4)
curl -s "http://192.168.86.Z/shutdown" || true
touch /tmp/ups2_60_triggered
fi

if [ "$CHARGE" -le 40 ] && [ ! -f /tmp/ups2_40_triggered ]; then
touch /tmp/ups2_40_triggered
sudo shutdown -h now
fi
fi

# Reset flags if AC power returns
if [[ "$STATUS" == *"OL"* ]]; then
rm -f /tmp/ups2_60_triggered /tmp/ups2_40_triggered
fi

Make it executable:
Bash
sudo chmod +x /usr/local/bin/ups2_monitor.sh

Add to /etc/crontab:

Code 16: UPS PiHole Nut Crontab

Code snippet
* * * * * root /usr/local/bin/ups2_monitor.sh
* * * * * root sleep 30; /usr/local/bin/ups2_monitor.sh

Configure both NAS units to connect to NUT server signals and process Safe Mode commands cleanly.

  1. Log in to DSM on both NAS units.

  2. Go to Control Panel → Hardware & Power → UPS.

  3. Check Enable UPS support.

  4. Select Synology UPS Server or Network NUT Server.

  5. Set Network UPS Server IP address:

    • Point to 192.168.86.250 (Proxmox) or 192.168.86.W (Raspberry Pi UPS 2 server).

  6. Set Time before Synology NAS enters Safe Mode: Choose Customize time (e.g., 2 minutes) or set it to respond to low battery signals.

  1. Go to Control Panel → Hardware & Power → General.

  2. Under Power Recovery, check Enable Wake on LAN (WOL).

  3. Check Restart automatically when power supply issue is fixed.

  4. Note the Ethernet MAC Addresses (found under Control Panel → Info Center → Network).

This script runs on Proxmox (UPS 1) to wake up the Synology NAS units via Wake-on-LAN if UPS 1 never failed, but UPS 2 dropped to 60% and restored power.

Edit the file /usr/local/bin/synology-recovery.sh and make it executable

Code 17: UPS Proxmox Auto Recovery Script

Bash
#!/bin/bash

GATEWAY_IP="192.168.86.1" # Core Router IP
NAS1_IP="192.168.86.X" # Synology Surveillance IP
NAS2_IP="192.168.86.Y" # Synology NFS IP
NAS1_MAC="AA:BB:CC:DD:EE:01" # Synology 1 MAC
NAS2_MAC="AA:BB:CC:DD:EE:02" # Synology 2 MAC

FLAG_FILE="/tmp/syno_shutdown_active"

# Check if Gateway is reachable
if ping -c 1 -W 2 "$GATEWAY_IP" > /dev/null 2>&1; then
if [ -f "$FLAG_FILE" ]; then
echo "Network power restored. Issuing WOL packets to Synology units..."
wakeonlan "$NAS1_MAC"
wakeonlan "$NAS2_MAC"

# Remove trigger flag
rm -f "$FLAG_FILE"

# Wait 8 minutes for NAS array to re-initialize before un-freezing mounts
sleep 480
pvesm set nfs-storage-name --disable 0
fi
else
# Gateway dropped (Network power down)
touch "$FLAG_FILE"
fi

Make it executable:
Bash
chmod +x /usr/local/bin/synology-recovery.sh

Bash

Add to /etc/crontab on Proxmox:

Code 18: UPS Proxmox Crontab

Code snippet
* * * * * root /usr/local/bin/synology-recovery.sh >/dev/null 2>&1

  1. Download and install WinNUT-Client (or CyberPower PowerPanel Personal).

  2. Open WinNUT-Client configuration.

  • UPS Name: ups3 (or cyberpower)

  • NUT Server IP: 127.0.0.1 (if USB is monitored via local WinNUT runner) or 192.168.86.250 (if monitoring central signals).

  • Port: 3493

  • Select Shutdown System on ONBATT + LOWBATT or set battery runtime remaining to < 5 minutes.

The ESP32 reads battery data over USB Host or Serial and reports telemetry to Home Assistant or Proxmox via Wi-Fi.

If using ESPHome with USB Host HID / Serial component:

Code 19: UPS ESP32 Nut Custom Firmware

YAML
esphome:
name: esp32-ups4-node

esp32:
board: esp32-s3-devkitc-1
framework:
type: esp-idf

wifi:
ssid: "YourNetworkSSID"
password: "YourWiFiPassword"
manual_ip:
static_ip: 192.168.86.Z
gateway: 192.168.86.1
subnet: 255.255.255.0

api:
encryption:
key: "YourGeneratedAPIKey"

web_server:
port: 80

# Configure custom local endpoint to listen for network shutdown calls from Raspberry Pi
http_request:

switch:
- platform: template
name: "Entertainment Display Relay"
turn_off_action:
- logger.log: "Executing TV/Shield Soft Off Trigger"

Operational Validation Checklist

  1. Verify Proxmox Server Status: Run upsc ups1@localhost on Proxmox to ensure battery telemetry is broadcasting properly.

  2. Verify Network Client Connectivity: From the Raspberry Pi, run upsc ups1@192.168.86.250 to confirm cross-network socket connectivity.

  3. Simulate Power Outage: Unplug Outlet 2 (Network UPS). Verify that:

    • The Raspberry Pi detects the drop.

    • At 60% battery, Synology units receive the safe mode command.

    • Restoring power to Outlet 2 causes Proxmox to automatically issue Wake-on-LAN packets and resume normal operation.