HomeLab Stage L: 100GbE

This is my #HomeLab Blog Post number 50. Amazing to see the journey so far….

In the last episode I mentioned the awesome integrations from Dell into the VMware cosmos. Now it is time for a huge announcement:

I have successfully implemented 100GbE network inside my HomeLab environment. Yes you read it correct, 100 Gigabit at Home 🙂

Why? Because I wanted it… and I would like to max out my vMotion network.

After 1GbE, 10GbE and 40GbE

I am using my Ubiquiti network stuff since several years now and I am still very happy with it. Gigabit and 10GbE are working fine, the access points running without any problems. I really like their solution, but what about high performance when you need more than 10GbE?

In the past I invested into one Dell S6000 switch, which has 32 ports that supports 40GbE. That was a big improvment and I used it mainly for the vSAN and vMotion traffic.

Do you know that feeling when you want more????

It´s time for 100GbE

My entire network runs totally fine, so why change it? I really wanted to create the ultimate HomeLab…

Server side:

Starting point: 4 x Mellanox ConnectX-5 Dual Port 100GbE PCIe 3.0 x16

My new main vSAN cluster is equipped with 4 x Dell XR2 servers. Each server has 2 x PCIe x16 slots. One slot is used by my Nvidia T4 GPUs and the other one with these dual port NICs.

I ordered 8 x Mellanox QSFP28 cables supporting 100GbE for the servers.

Network side:

After some research I found my 100GbE canidate: Mellanox (Nvidia Networking) SN-2100 Switch. This tiny little beast is absolute amazing!

16 x 100GbE ports that can be splitted (later more on that topic)

Power consumption of the Mellanox SN2100 is 86 watts running in “production”. I also ordered the Rackmount Kit:

Web-UI:

The Mellanox switch got a Web UI and of course an CLI access. My switch is operating on the Mellanox Onyx Operating system.

I really like the UI, but for several tasks you need the CLI

QSFP-Split Mode:

For the switch uplink and some 10GbE ESXi connections I have configured some of the ports into split-mode for Breakout cables. This can be done via CLI on the port itself. First shutdown the port, enter module-type qsfp-split-4 and confirm. Now you have splitted your 100GbE port into 4 smaller one. They can operate at 1, 10 or 25GbE. Very flexible….

Jumbo Frames:

I have enabled Jumbo Frames for several reasons: vMotion, vSAN and of course: NSX-T

LACP:

I have configured an LACP portchannel as an uplink to my Ubiquiti US-16XG switch. Very easy and simple to configure:

VLANs:

I have configured all my required VLANs on the Mellanox SN2100 switch and setup the ports:

Routing:

Main routing is done on the Ubiquiti USG-XG-8 machine, NSX-T TEP and NSX-T Edge TEP routing is done on the Mellanox SN2100. Why? Because Ubiquiti is not able to route Jumbo Frames between VLANs…..

Using 100GbE

The new Dell XR2 cluster is attached to the SN2100 via Dual 10GbE and Dual 100GbE connections.

My new Monster Workstation is attached via 40GbE, also my Custom All Flash NAS system. I love that powerful performance inside my network.

vSAN is using the 100GbE vDS portgroup without any modification.

vMotion is tuned to also support the 100GbE connection via 6 streams. This is done via an advanced setting inside ESXi:

The Dell S6000 switch is now only for test enviroments and setups. I can control each of my devices at both datacenters via IP Power / PDU.

Dynamic Routing

My next steps are the implementation of dynamic routing protocols around my entire network, including Mellanox, Ubiquiti and Velocloud. I had OSPF running in the past mainly for NSX-V. Maybe I will implement BGP or once again OSPF. We will see….

Mellanox supports both:

Ubiquiti´s USG-XG-8 can also do both (needs to be configured via CLI)

And finally my Veloclouds also supports both:

I wanted to create the ultimate HomeLab / HomeDC. What do you think?

But I am not finished, I think I will never be finished….. So stay tuned for the next episodes…..

Here is the next one: HomeLab Stage LI: Datacenter Resizing