Skip to content

Hypervisor (VMWare ESXi)

Proxmox explodes (package manager with updating in a HYPERVISOR) and hardware passthrough is basically impossible. so I use ESXi.

Virtual Machine Map

  • There are a number of uninteresting machines and services not listed here.

  • Minecraft is given its own VM along with a bare metal nvme drive. The servers I run are so IO intensive they actually manage to overload the VMWare virtual disk buffer, which results in crashing. I don't feel like tuning ESXi's paramaters for this.

  • Owncloud and Mkdocs share similar isolation-oriented security practices.

Dreadnought_ESXiMedia_VMServarr_VMLibrary_VMTrueNAS_VMPfSense_VMCloud_VMMinecraft_VMCaddyJellyfinOwncloudNavidromeCrucibleCauldronCloudWireguardMullvadMkDocsHTTP/s PortsVideoMusicDocsBackups & DataPublished Site
Dreadnought_ESXiMedia_VMServarr_VMLibrary_VMTrueNAS_VMPfSense_VMCloud_VMMinecraft_VMCaddyJellyfinOwncloudNavidromeCrucibleCauldronCloudWireguardMullvadMkDocsHTTP/s PortsVideoMusicDocsBackups & DataPublished Site

Installation

ESXi 8 (current version as of writing) is as I understand it no longer available to the general public, so it is tricky to source an ISO. (EDIT: nobody knows what the fuck broadcom is doing anymore..). Here's a place that seems to keep an archive of ESXI ISO files: https://mirror.mahanserver.net/VMware/ESXI/80u3/

I also keep the latest ISO in my archive. You want the VMVisor Installer ISO.

REDACTED Use Rufus to flash it and install normally.

Quirks

I've found that problems start when memory usage gets closer to the total available system memory. This threshold is about 182/192G or 10G free memory. It causes performance issues and also random VM crashes.

  • control+q if the terminal is not responding to typing (in a linux VM where you didn't install the desktop gui)

VM Creation and Management

Small Notes

  • Don't forget to configure autostart on your VMs!
  • Under VM advanced options you can set the latency acceptability to "normal". "High" requires reserving 100% of allocated CPU, which defeats the purpose of me min-maxing dynamic allocation.

Enable Clipboard for VMs

to enable copy-paste functionality:

  • navigate to vm settings -> options -> advanced -> add configuration
  • add the following values:
  • isolation.tools.copy.disable FALSE
  • isolation.tools.paste.disable FALSE
  • isolation.tools.setGUIOptions.enable TRUE

CPU Notes (NUMA Nodes etc.)

"Cores per socket" should be equal to the cores per chiplet if you have a multi-chiplet CPU. This is because each chiplet's L3 cache is only available to its own chiplet. Setting this up properly will let your VM's OS manage processes correctly. You should use the PHYSICAL cores as your guide, not threads, as this could lead to numa misalignments.

Networking

I map each physical NIC to its own virtual switch and port group. There are a couple reasons for doing this:

  • Virtual Switches are single threaded, and if you're using 10g networking like I am you are already hitting the limit of what can be passed around through that.
    • From online resources and testing I've found that virtual connections seem to cap out at ~20gbit/sec
  • I do not want to start creating VLANs. They introduce a risk if I mess up configuration or anything changes ever and traffic starts leaking where it shouldn't.

A PfSense VM is used to glue everything together. All existing port groups are added to PfSense so it can manage everything.

Memory Management

Critical/Core service VMs are given hard locked reserved memory to prevent crashes or other problems if I mess up ram provisioning at any point. Atow I generally just hard lock RAM on every VM since I have an abundance of memory.

The UI for RAM assigning is buggy and you may have to flick back and forth between MB GB TB to get the right control flow to pop back to saying everything is okay.

Hardware Passthrough

GPU

Graphics cards need to not be in use to be available for passthrough, otherwise the Hypervisor will just gank it to send video out. I don't like the option where you can add in a script to gank the card at boot, because at that point you should just not have anything plugged into it. (if it is enabled and says it needs to reboot, disable and re-enable it)

Unsupported Hardware

You can pass through quite a lot in actuality. If a device does not show as capable of passthrough you can edit ESXi's passthru file and add the device ID of the thing you're passing through. I haven't done this in a while so this section is blank because I'm not gonna figure out how to do it again atm (d3d0).