Yesterday, I had the pleasure of finding out that there was an invisible/missing/rogue VM on one of the servers in our pool.
We had a “dead-beef” issue a few weeks prior and had attempted to migrate our VMs off to other members of the pool before rebooting the member. With that attempt, it caused all of the VMs to go into a paused state from which a forced shutdown was required. We rebooted the server at that point because it was the only way we could get the “dead-beef” issue recovered.
After the pool member reboot, the VMs would not start, so we cloned them and booted them up. Everything seemed fine in the next few weeks, until I decided to delete the originals as the replacements appeared to be functioning fine.
One of those servers had apparently been running in the background even though XenServer showed it as off.
We could ping it, see a MAC address for it that didn’t match the server it should have been, etc. Though, it didn’t show up in any VM list, be it from the XenCenter console, or through the “xe vm-list” command.
Later, we discovered that there was an extra domain (command: list_domains) on one of the servers. It’s possible that it could have been fixed by destroying that domain, but we ended up rebooting the entire pool. I started with that pool member after shutting down / migrating the existing VMs and placing it into Maintenance Mode.
The rogue server stopped responding pretty much immediately at this point and when it came back up it was no longer an issue.
TL;DR – Apparently, sometimes a server can be running in XenServer with nearly no record of it’s existence after being deleted in a “powered off” state.