It seems to that it works. I don't get any web-scrapers hitting anything but my main domain. I can't find any of my subdomains on google.
Please tell me how you believe that it works. Maybe i overlooked something...
I've got a Sever running Homeassistant with the ESPHome Addon. The Lights got a custom PCB in them using a ESP32 and a 4 channel warm/cold white led strip driver. But you can also build them using of-the-shelf parts. They are mains powered without a switch, instead i wired the switches to a sensor input. This allows me to control the light either via the switch, or Homeassistant. They even got some buttons directly on them to force them on/off if my server is down. I also got a radar in there for presence detection. Basically the same as an infrared motion sensor, but it doesn't turn the light off while im on the toilet. Thanks to using Homeassistant, I can change the color temperature and brightness of the lights depending on the time of day. It's really nice to have some dim and warm lights in the evening before going to bed.
But ESPHome isn't limited to some custom build stuff. Anything that uses an ESP32* chip can be flashed to run ESPHome instead of whatever it came with. I got some sonoff relays that control my shutters and an Emporia Vue 2 to measure my power usage. Depending on the device you might be able to flash it either via Wifi or you have to disassemble it to get to the programming pins. The nice thing about the ESP32 is that a vendor cannot lock the firmware. You can always flash something custom.
ESPHome isn't limited to Homeassistant however. You can also have each device run a web-server to control it, or connect it to MQTT.
Also i should mention some alternatives:
I use good ol' obscurity. My reverse proxy requires that the correct subdomain is used to access any service that I host and my domain has a wildcard entry. So if you access asdf.example.com you get an error, the same for directly accessing my ip, but going to jellyfin.example.com works. And since i don't post my valid urls anywhere no web-scraper can find them. This filters out 99% of bots and the rest are handled using authelia and crowdsec
That reminds me of PayPal. I can't use my hardware sevurity key to log in on mobile... Unless I enable desktop mode in firefox. Then it works