Labbing Virtualization Part 2

Sidebar: The host pictured above was a previous rendition of my virtual labbing environment, back from the BX. It still amazes me just how much you can get out of a physical host that you piece together and build it yourself... Or with your brother...

So, just to remind you of the setup for this post, let’s recap what we went over in Part 1 with regards to our environment. I have a single Virtual Host (Dell Server) running VMWare as its OS (Operating System) with various Host resources available for consumption by the Guest Virtual Machines (VMs) to utilize. In my particular case with VMWare (OS) as the Hypervisor, I have multiple Guest VMs (VMs) running concurrently. Confused yet? Good. That was our quick review. Let’s trek forward here. In our example from the last post we had various Guest VMs we were referencing. We had a Windows Box, a WordPress Box and EVE-NG that I horribly drew out. Remember this odd picture?

I’m going to take a quick pause here and just throw a PSA out there for all of my not-exactly-full-tilt-technically-savvy-folks who happen to be reading this post; we will be diving a little further below the surface to talk about some of the “techie things” of “whatever it is [you] I do and talk about all day,” so just be aware of that as this article continues.

Let’s start by having you wrap your head around what I refer to as a “Hypervisor.” The first logical question that hopefully most people at this point might be asking is, “Well, what is a Hypervisor?” I’m so glad you asked. A Hypervisor can be either hardware or software (or an Operating System) that is capable of deploying, building and/or running Virtual Machines (VMs). So referencing our previous post and example, VMWare (the OS currently running on the Host) would be our Hypervisor as it is the Host’s (Dell) OS that is actually running the various Guest VMs (Windows, WordPress and EVE-NG) from our last example. Feel free to pause here to just let that sink in. Sidebar: I may use the following terms interchangeably going forward. OS, VMWare, ESXi, Hypervisor, just be aware that I am referencing the same thing.

So let’s zero in on the EVE-NG Guest VM that’s running and what it actually is with regards to VMWare. What is its actual function? And more importantly, why do I care? So to the Hosts (Dell) Hypervisor (VMWare), EVE-NG is just another Guest VM under its control. Granted in this case, this specific Guest VM (EVE-NG) has quite a bit more resources at its disposal (96 Gigs of RAM and 12 CPU Sockets) when compared to that of a more “typical server” that I have deployed in my environment. You may be asking, “so why is that?” Great question. Let’s see why. Don’t forget, that we have the ability to run multiple concurrent VMs (Virtual Machines) within the EVE-NG Guest VM; including (but not limited to) Windows and Linux Machines, Routers, Switches, Firewalls, Application Servers, and much more inside of this VM. So according to what we defined earlier: a Hypervisor can be either hardware or software (or an Operating System) that is capable of deploying, building and/or running Virtual Machines (VMs), EVE-NG seems to be able to foot the bill as a Hypervisor. It’s just a Hypervisor that’s running inside of another Hypervisor that’s running on a Virtual Host (Dell). Inception much? Well... Almost, but I'm not exactly trying to hire people to wake me up at this point of my life. (Inception movie reference).

Now, when we think about the resources that are available to a given “Hypervisor” (whether this is from the Host to its Hypervisor or it’s from the Hypervisor into another “Nested Hypervisor” inside top level Host’s Hypervisor), we should probably “walk the path” mentally, so to speak, so we don’t go around in circles and get frustrated or confused. Think about an easier scenario. We have a Host (Dell) running VMWare as its Hypervisor (OS). The Host (Dell) has resources physically available to it (eg. the Host’s Memory, CPU, Storage, PCIe Devices, USB Devices, etc) which we can make available to its Hypervisor. Once available to the Host’s Hypervisor, the Hypervisor can then make those resources available to a VM to do our bidding and help take over the world or enhance our labbing experiences (6 of a dozen and half of another I think the saying goes). OK, so let’s kick the tires a bit here. When we pass a device or resource from the Host (Dell) into its OS (VMWare) at a basic level, what do we need in order for that device/resource to actually function? Well, for starters, the Host needs to be able to recognize what you are actually plugging into it, and typically this is resolved by providing it with some drivers to control the hardware. Then the Hypervisor (VMWare) will also need to know how to control this device/resource. Hopefully that makes sense.

The following couple of lines are just a friendly reminder regarding hardware compatibility in general. If you’re reading this article as one of those who are actually “full-tilt-technically-savvy-folks,” you probably are already aware that you need to confirm compatibility between all of your devices (Host, Hypervisor and the Virtual Machine the device/resource is destined for). Nevertheless, I’d like to mention the following couple of lines for clarity. I know that this may sound obvious, “pero let me tell you something…” I’ve been known to notoriously assume compatibility when it comes to server parts. Sometimes, I am correct in my assumptions, but sometimes, well I’m not. So my advice is, don’t assume it’s going to work. Take 30-90 seconds of time and go ahead and use Google or DuckDuckGo, or whatever happens to be your search engine of choice, to confirm compatiblilty. They can probably put you in touch with someone or reference some article/KB that can answer your question on compatibility with your particular Host.

Now that we have that squared away, let’s reel it back in here and continue our discussion. Let’s use a USB 3.0 PCIe card, located on a riser card of my host, for this “pass through” example. So let’s assume you have confirmed by boot-up, that the Host recognizes this card. OK, cool; the first barrier has been crossed. Visual Aid below of the actual card (red circle located above the dual port NIC card above the onboard VGA and COM ports) we are attempting to working with, because I like pictures.

The next “hurdle” so to speak, is the USB 3.0 controller driver. Luckily, VMWare has a lot of baked in drivers and my card is compatible with both the Hypervisor (VMWare) as well as the Host (Dell). OK, so barrier 2 is done. (In some cases, you may need an actual hardware driver for the device on the USB card. I will not be covering that, but it's just a heads up.)

So the PCIe card is plugged into the Host (Dell) machine and the Host understands it. Great. The Host (Dell) then makes the PCIe USB 3.0 card available to VMWare as a resource for consumption by the Guest VMs. So now that you’re probably thoroughly confused, or you’re wondering why I keep repeating myself, or just maybe you’re wondering when and if this article will ever end or get to it’s point, let’s just throw one more piece at you. Let’s get to pass this actual USB Controller into EVE-NG’s environment.

Another disclaimer: I am quite green with regards to Linux. So, just like VMWare, I have learned the beginnings of Linux out of necessity... Like, what's the IP Address being assigned to my eth0 card? Or, how do I see if there's any USB stuff?

Just for posterity… (maybe EVE-NG will get a voicemail with a date and location? - Tenet movie reference) Let’s see what EVE-NG sees USB-wise before passing anything through to it:

As expected, and as illustrated above, EVE-NG has no knowledge of USB Controllers or USB Devices of any form within my current setup, which should make sense, hopefully, to you reading this article… Why would I spend time writing 2 articles, so far, and begin a 3rd post on this topic if this was working without any manual intervention?

Bear in mind that the output of the above command for you may differ from mine. In this example/post/article I have concocted, EVE-NG is a VM running within another Hypervisor aka “Nested Hypervisor.” Your EVE-NG instance may be sitting on your Host’s bare metal and actually list a USB Controllers and Devices at this point. I am fully aware that you may not like mental torture, you also may not be trying to push the limits of your patience like me, performing Virtualization a Hypervisor inside another Hypervisor. Truth be told, chances are, if your EVE-NG installation is on bare metal, Linux will more than likely pickup your USB Controller and USB Devices and you may see output some. However, as previously stated, this example/post/article is for peeling back some Nested Hypervisor, leveraging hardware and software virtualization. This may be why you’re seeing a different output than mine above… Now, back to our regular scheduled program…

Are you still awake? I hope so. I talk to myself enough, so I don’t need to actually write to myself about this stuff… So full disclosure, when I was studying for my CCIE Security lab in 2019, I originally had QUITE A FEW issues passing through the USB Controller as a USB 3.0 controller to EVE-NG. It could have been a combination of anything, from of my lack of knowledge of Linux, my lackluster passable knowledge of VMWare, the Version of VMWare I was previously running last year (I think it was 6.0) or the version of EVE-NG I was running. After wrestling with it for quite some time we, (my brother and I), got the USB 3.0 controller working. So let’s add this USB Controller to the EVE-NG VM already!

Hop onto vCenter and then edit your VM. “Add a New Device.” Select USB Controller from the list. At the sake of sounding like a broken record, we need to add this to EVE-NG VM because in my scenario, this is a Nested Hypervisor. We want VMWare to pass the USB controller that it is aware of to the EVE-NG VM so then we can assign and utilize the USB Controller and it’s USB devices onto nodes in the lab topologies… For those of you dying for a picture to break up this wall of technical text, “your savior is here…” – Loki

Once you add the controller, make sure you select USB 3.0 and hit OK on the Edit Settings page in VMWare.

Then you’ll see it start doing some work in your Recent Tasks to pass this USB Controller down to your EVE-NG VM

Once VMWare finishes doing its magic, let’s hop back on EVE-NG and see if anything changed. Did it actually get the controller? Did it work? Why am I doing this again?

Awww SNAP! There’s the 4th wall break. Well, technically the Third wall (Host –> Hypervisor –> Hypervisor 2), the 4th wall will be in the next post, I promise. So, we have successfully passed through the USB Controller to EVE-NG, but where are the actual USB Device(s) attached to this USB Controller? After all, I did leave a USB device attached to this card the last time I was at the rack, didn’t I? I sure hope so…

Let’s head back to the “Edit Settings” option of the EVE-NG VM in vCenter. However, this time around, since we already have the USB Controller installed, we now will add a “Host USB Device” aka what’s connected to a USB port to the EVE-NG VM.

After selecting “Host USB Device,” VMWare should present you with a list of “what’s connected to this PCIe card.” Essentially, it’s asking you “so, what do you want me to make available to EVE-NG?” In my case, I have a USB Hard Drive as well as this other device…. What is that other device? Hmmm… Let’s peep what it is and add it.

Let’s hit OK one more time on the Edit Settings screen, and VMWare will then go to work again passing through this “Ralink” thing…

Ok. So now we have passed through this “Ralink” devaaaaace (device) to EVE-NG, does it understand what we just passed through? Let’s find out… Hop over to EVE-NG and let’s give it the old “lsusb” command

Well, well, well… What do we have here? Isn’t that a sight for sore eyes. Now why in the nested-virtual-hypervising-world would we need to accomplish something like? Oh I don’t know… Maybe we can now pass this wireless adapter through to a VM/Node running inside of EVE-NG and have it pickup Wifi? If you have stuck with me this far, I’m both honored and impressed. I know, there’s a lot of information to keep straight here, so I commend you on your effort and your stamina.

Stay tuned for the third post, coming to a web browser near you, for how to accomplish assigning this USB Device to a node running inside of EVE-NG. Then, stick around to see what else you can have passed through to your lab nodes. Maybe it will spark some ideas of your own on how to grow your lab topologies, or lab up some other items that you may not be familiar with.

OK. One final thought. Just for end to end clarity, my current setup and versioning of the lab environment (VMWare ESXi 6.7, EVE-NG (3.0.1-17-PRO), Dell System Firmware Versions) seem to be much more stable than when I had all of the issues I mentioned over a year ago when I was studying for the CCIE Security and trying to pass through both the USB Controller and USB Devices. This time around, I was able to “pass through” both the USB Controller and USB Device, both as USB2.0, as well as, USB3.0. Pics below. We will test this on a node on a lab topology in our next post.

– 51406 Out…