Skip to content

Latest commit

 

History

History
231 lines (166 loc) · 20.8 KB

README.md

File metadata and controls

231 lines (166 loc) · 20.8 KB
Raw

terraform-equinix-metal-vcf

run-pre-commit-hooks generate-terraform-docs

terraform-equinix-metal-vcf is a minimal Terraform module that utilizes the Terraform provider for Equinix to provision, configure and setup an Equinix Metal hardware environment with the pre-requisites for a VCF 5.1.1 installation.

Target Metal Architecture featuring Metal VRF for Underlay routing

Usage

This project is supported by the user community. Equinix does not provide support for this project.

This project may be forked, cloned, or downloaded and modified as needed as the base in your integrations and deployments.

This project may also be used as a Terraform module.

To use this module in a new project, create a file such as: examples/vcf_management_domain/main.tf.

Pre-Requisites

You are responsible for the following

Physical Network

  • DHCP with an appropriate scope size (one IP per physical NIC per host) is configured for the ESXi Host Overlay (TEP) network. Providing static IP pool is also supported but some Day-N operations like stretching a cluster will not be allowed if static IPs are used.

Physical Hardware and ESXi Host

  • Hardware and firmware (including HBA and BIOS) is configured for vSAN.

    • Note: The Equinix Support team can assist with ensuring that BIOS configuration is brought into compliance with vSAN recommendations should it be discovered that this is not already the case.

  • Physical hardware health status is 'healthy' without any errors.

    • Note: The Equinix Support team can assist with ensuring hardware is brought into a healthy state should it be discovered otherwise.

  • All hosts are in synchronization with a central time server (NTP).

    • Note: While this module does configure the user provided NTP server details, the provided NTP server IP must be reachable by the Metal Instances through VRF Interconnection.

Supporting Infrastructure

  • DNS server for name resolution. Management IP of hosts is registered and queryable as both a forward (hostname-to-IP), and reverse (IP-to-Hostname) entry.
    • Note: While this module does configure the user provided DNS server details, the provided DNS server IP must be reachable by the Metal Instances through VRF Interconnection.

This module does provide for the following VCF Infrastructure requirements as required by Cloud Builder

Network

  • Top of Rack switches are configured. Each host and NIC in the management domain must have the same network configuration. No ethernet link aggregation technology (LAG/VPC/LACP) is being used.

  • IP ranges, subnet mask, and a reliable L3 (default) gateway for each VLAN are provided.

  • Jumbo Frames (MTU 9000) are recommended on all VLANs. At a minimum, MTU of 1600 is required on the NSX Host Overlay VLAN and must be enabled end to end through your environment.

  • VLANs for management, vMotion, vSAN and NSX Host Overlay networks are created and tagged to all host ports. Each VLAN is 802.1q tagged.

  • Management IP is VLAN backed and configured on the host. vMotion & vSAN IP ranges are configured during the bring-up process.

Hardware and ESXi Hosts

  • All servers are vSAN compliant and certified on the VMware Hardware Compatibility Guide, including but not limited to BIOS, HBA, SSD, HDD, etc.

  • Identical hardware (CPU, Memory, NICs, SSD/HDD, etc.) within the management cluster is highly recommended. Refer to vSAN documentation for minimal configuration.

  • One physical NIC is configured and connected to the vSphere Standard switch. The second physical NIC is not configured.

  • ESXi is freshly installed on each host. The ESXi version matches the build listed in the Cloud Foundation Bill of Materials.

  • All hosts are configured with a central time server (NTP). NTP service policy set to 'Start and stop with host'.

  • Each ESXi host is running a non-expired license - initial evaluation license is accepted. The bring-up process will configure the permanent license provided.

Other Infrastructure

  • All hosts are configured with a DNS server for name resolution.

Custom root password

Generating custom root password

To generate a password hash of your desired ESXi root password run the 'mkpasswd' command on a Linux system with the 'whois' package installed as follows

mkpasswd --method=SHA-512

You'll be prompted to enter the desired password sting you wish to hash, then press enter.

Alt text

The output will be the string you need to use in the esxi_password variable near the end of the terraform.tfvars.example file

Terraform Deployment Workflow

Preparation

  • Download the vcf-ems-deployment-parameter_X.X.X spreadsheet for the VCF version you're deploying from VMware and fill it out for your environment. For more about the "Cloud Builder Deployment Parameter Guide" spreadsheet file and its configuration, see About the Deployment Parameter Workbook on docs.vmware.com (requires a login and entitlements).
  • Copy the terraform.tfvars.example file to terraform.tfvars and fill it with the same values you used in the vcf-ems-deployment-parameter_X.X.X spreadsheet.

Deployment

  • Deploy this Terraform module by running terraform init -upgrade and terraform apply.
  • Note the following values that you'll need later:
    • The public IP address of the bastion host: terraform output -raw bastion_public_ip
    • The private IP address of the bastion host: terraform output -raw bastion_private_ip
    • The public IP address of the management host: terraform output -raw management_public_ip
    • The password for the management host: terraform output -raw management_password
    • The DNS address of the ESX01 host: terraform output -raw esx01_address
  • The following steps should be run on the management host.
    • RDP to the public IP of management host you noted earlier.
      • Username: SYSTEM\Admin
      • Password: Use the password you noted earlier.
    • Download the Cloudbuilder OVA from VMware
    • Log in to one of the ESXi hosts
      • Use the DNS address of the ESX01 host you noted earlier. (Our example uses: https://sfo01-m01-esx01.sfo.rainpole.io)
      • Username: root
      • Password: the custom root password you used to generate the hash earlier.
    • Deploy Cloudbuilder OVA on ESXi, we recommend following VMware's documentation for this. https://docs.vmware.com/en/VMware-Cloud-Foundation/5.1/vcf-deploy/GUID-78EEF782-CF21-4228-97E0-37B8D2165B81.html
      • You will need to use the bastion host private IP you noted earlier as the DNS and NTP server during the OVA deployment.
    • Login to cloudbuilder at the address you installed it at using the username/password you chose during the OVA deployment.
    • Upload the vcf-ems-deployment-parameter spreadsheet to cloudbuilder when it asks for it.
    • Fix issues cloudbuilder finds.
    • Push deploy button and wait while VCF deploys.

Requirements

Name Version
terraform >= 1.5
equinix >= 1.35
random >= 3

Providers

Name Version
equinix >= 1.35
random >= 3

Modules

Name Source Version
metal_vrf ./modules/metal_vrf_w_interconnection_service_tokens n/a
metal_vrf_gateways_w_dynamic_neighbor ./modules/metal_vrf_gateway_w_dynamic_neighbor n/a
ssh ./modules/ssh/ n/a
vcf_metal_devices ./modules/vcf_metal_device n/a

Resources

Name Type
equinix_metal_device.bastion resource
equinix_metal_device.management resource
equinix_metal_port.bastion_bond0 resource
equinix_metal_port.management_bond0 resource
random_password.management resource

Inputs

Name Description Type Default Required
bastion_ip IP address for the Bastion host string n/a yes
cloudbuilder_ip IP address for the Cloudbuilder appliance string n/a yes
esxi_devices Map containing individual ESXi device details for each Metal Instance 
map(object({
    name           = string               # Short form hostname of system (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > I6:L6)
    mgmt_ip        = string               # Management Network IP address for VMK0 (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > I7:L7)
    reservation_id = optional(string, "") # Hardware reservation IDs to use for the VCF nodes. Each item can be a reservation UUID or next-available.
  }))
 n/a yes
esxi_management_gateway Management Network Gateway for ESXi default TCP/IP Stack (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > F8) string n/a yes
esxi_management_subnet Management Network Subnet Mask for VMK0 (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > E8) string n/a yes
esxi_network_space Overall Network space for the VCF project string n/a yes
esxi_password mkpasswd Pre-hashed root password to be set for ESXi instances (Hash the password from vcf-ems-deployment-parameter.xlsx > Credentials Sheet > C8 using 'mkpasswd --method=SHA-512' from Linux whois package) string n/a yes
esxi_plan Slug for target hardware plan type. The only officially supported server plan for ESXi/VCF is the 'n3.xlarge.opt-m4s2' https://deploy.equinix.com/product/servers/n3-xlarge-opt-m4s2/ string n/a yes
esxi_version_slug Slug for ESXi OS version to be deployed on Metal Instances https://github.com/equinixmetal-images/changelog/blob/main/vmware-esxi/x86_64/8.md string n/a yes
metal_auth_token API Token for Equinix Metal API interaction https://deploy.equinix.com/developers/docs/metal/identity-access-management/api-keys/ string n/a yes
metal_project_id Equinix Metal Project UUID, can be found in the General Tab of the Organization Settings https://deploy.equinix.com/developers/docs/metal/identity-access-management/organizations/#organization-settings-and-roles string n/a yes
metal_vrf_asn ASN to be used for Metal VRF https://deploy.equinix.com/developers/docs/metal/networking/vrf/ string n/a yes
metro Equinix Metal Metro where Metal resources are going to be deployed https://deploy.equinix.com/developers/docs/metal/locations/metros/#metros-quick-reference string n/a yes
nsx_devices Map containing NSX Cluster host and IP details 
map(object({
    name = string # Short form hostname of system (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > I6:L6)
    ip   = string # Management Network IP address for VMK0 (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > I7:L7)
  }))
 n/a yes
sddc_manager_ip IP address for the SDDC Manager string n/a yes
sddc_manager_name Hostname for the SDDC Manager string n/a yes
vcenter_ip IP address for the vCenter Server string n/a yes
vcenter_name Hostname for the vCenter Server string n/a yes
vcf_vrf_networks Map of Objects representing configuration specifics for various network segments required for VCF Management and Underlay Networking 
map(object({
    vlan_id        = string                # (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > C7:C10) 802.1q VLAN number
    vlan_name      = string                # (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > D7:D10) Preferred Description of Metal VLAN
    subnet         = string                # (vcf-ems-deployment-parameter.xlsx > Hosts and Networks Sheet > E7:E10) CIDR Subnet to be used within this Metal VLAN
    enable_dyn_nei = optional(bool, false) # Whether or not to configure BGP Dynamic Neighbor functionality on the gateway, only use for NSX-t Edge uplink VLANs if NSX-t will peer with Metal VRF
    dyn_nei_range  = optional(string, "")  # CIDR Range of IPs that the Metal VRF should expect BGP Peering from
    dyn_nei_asn    = optional(string, "")  # ASN that the Metal VRF should expect BGP Peering from
  }))
 n/a yes
vrf_bgp_customer_peer_ip_pri IP of BGP Neighbor on Primary Interconnection that Metal VRF should expect to peer with string n/a yes
vrf_bgp_customer_peer_ip_sec IP of BGP Neighbor on Secondary Interconnection that Metal VRF should expect to peer with string n/a yes
vrf_bgp_md5_pri MD5 Shared Password for BGP session authentication string n/a yes
vrf_bgp_md5_sec MD5 Shared Password for BGP session authentication string n/a yes
vrf_bgp_metal_peer_ip_pri IP of Metal VRF on Primary Interconnection for peering with BGP Neighbor string n/a yes
vrf_bgp_metal_peer_ip_sec IP of Metal VRF on Secondary Interconnection for peering with BGP Neighbor string n/a yes
vrf_peer_asn ASN that will establish BGP Peering with the Metal VRF across the interconnections string n/a yes
vrf_peer_subnet Subnet used for both Metal VRF interconnections (/29 or larger) string n/a yes
vrf_peer_subnet_pri Subnet used for point to point Metal VRF BGP Neighbor connection across the Primary interconnection string n/a yes
vrf_peer_subnet_sec Subnet used for point to point Metal VRF BGP Neighbor connection across the Secondary interconnection string n/a yes
windows_management_ip IP address for the Windows management host string n/a yes
zone_name DNS Zone name to use for deployment (vcf-ems-deployment-parameter.xlsx > Deploy Parameters Sheet > J6:K6) string n/a yes
bastion_name Hostname for the Bastion host string "bastion" no
bastion_plan Which plan to use for the ubuntu based bastion host. string "m3.small.x86" no
cloudbuilder_name Hostname for the Cloudbuilder appliance string "cloudbuilder" no
windows_management_name Hostname for the Windows management host string "management" no
windows_management_plan Which plan to use for the windows management host. string "m3.small.x86" no

Outputs

Name Description
bastion_public_ip The public IP address of the bastion host. Used for troubleshooting.
cloudbuilder_default_gateway Cloudbuilder Default Gateway to use during OVA deployment.
cloudbuilder_hostname Cloudbuilder Hostname to use during OVA deployment.
cloudbuilder_ip Cloudbuilder IP to use during OVA deployment.
cloudbuilder_subnet_mask Cloudbuilder Subnet Mask to use during OVA deployment.
cloudbuilder_web_address Cloudbuilder Web Address
dns_domain_name DNS Domain Name to use during OVA deployment.
dns_domain_search_paths DNS Domain Search Paths to use during OVA deployment.
dns_server DNS Server to use during OVA deployment.
esx01_web_address The web address of the first ESXi host to use in a browser on the management host.
ntp_server NTP Server to use during OVA deployment.
ssh_private_key Path to the SSH Private key to use to connect to bastion and management hosts over SSH.
windows_management_password Randomly generated password used for the Admin accounts on the management host.
windows_management_rdp_address The public IP address of the windows management host.

Contributing

If you would like to contribute to this module, see CONTRIBUTING page.

License

Apache License, Version 2.0. See LICENSE.