Microsoft Multi-Site Failover Cluster for DR & Business Continuity

Not every organisation looses millions of dollar per second but some does. An organisation may not loose millions of dollar per second but consider customer service and reputation are number one priority. These type of business wants their workflow to be seamless and downtime free. This article is for them who consider business continuity equals money well spent. Here is how it is done:

Multi-Site Failover Cluster

Microsoft Multi-Site Failover Cluster is a group of Clustered Nodes distribution through multiple sites in a region or separate region connected with low latency network and storage. As per the diagram illustrated below, Data Center A Cluster Nodes are connected to a local SAN Storage, while replicated to a SAN Storage on the Data Center B. Replication is taken care by a identical software defined storage on each site.  Software defined storage will replicate volumes or Logical Unit Number (LUN) from primary site in this example Data Center A to Disaster Recovery Site B. Microsoft Failover cluster is configured with pass-through storage i.e. volumes and these volumes are replication to DR site. In the Primary and DR sites, physical network is configured using Cisco Nexus 7000. Data network and virtual machine network are logically segregated in Microsoft System Center VMM and physical switch using virtual local area network or VLAN.  A separate Storage Area Network (SAN) is created in each site with low latency storage. Volumes of pass-through storage are replicated to DR site using identical size of volumes.

image

                                     Figure: Highly Available Multi-site Cluster

image

                           Figure: Software Defined Storage in Each Site

 Design Components of Storage:

  • SAN to SAN replication must be configured correctly
  • Initial must be complete before Failover Cluster is configured
  • MPIO software must be installed on the cluster Nodes (N1, N2…N6)
  • Physical and logical multipathing must be configured
  • If Storage is presented directly to virtual machines or cluster nodes then NPIV must configured on the Fabric Zones.
  • All Storage and Fabric Firmware must up to date with manufacturer latest software
  • An identical software defined storage must be used on the both sites 
  • If a third party software is used to replicate storage between sites then storage vendor must be consulted before the replication. 

Further Reading:

Understanding Software Defined Storage (SDS)

How to configure SAN replication between IBM Storwize V3700 systems

Install and Configure IBM V3700, Brocade 300B Fabric and ESXi Host Step by Step

Application Scale-out File Systems

Design Components of Network:

  • Isolate management, virtual and data network using VLAN
  • Use a reliable IPVPN or Fibre optic provider for the replication over the network
  • Eliminate all single point of failure from all network components
  • Consider stretched VLAN for multiple sites 

Further Reading:

Understanding Network Virtualization in SCVMM 2012 R2

Understanding VLAN, Trunk, NIC Teaming, Virtual Switch Configuration in Hyper-v Server 2012 R2

Design failover Cluster Quorum

  • Use Node & File Share Witness (FSW) Quorum for even number of Cluster Nodes
  • Connect File Share Witness on to the third Site
  • Do not host File Share Witness on a virtual machine on same site
  • Alternatively use Dynamic Quorum

Further Reading:

Understanding Dynamic Quorum in a Microsoft Failover Cluster

Design of Compute

  • Use reputed vendor to supply compute hardware compatible with Microsoft Hyper-v
  • Make sure all latest firmware updates are applied to Hyper-v host
  • Make manufacture provide you with latest HBA software to be installed on Hyper-v host

Further Reading:

Windows Server 2012: Failover Clustering Deep Dive Part II

Implementing a Multi-Site Failover Cluster

Step1: Prepare Network, Storage and Compute

Understanding Network Virtualization in SCVMM 2012 R2

Understanding VLAN, Trunk, NIC Teaming, Virtual Switch Configuration in Hyper-v Server 2012 R2

Install and Configure IBM V3700, Brocade 300B Fabric and ESXi Host Step by Step

Step2: Configure Failover Cluster on Each Site

Windows Server 2012: Failover Clustering Deep Dive Part II

Understanding Dynamic Quorum in a Microsoft Failover Cluster

Multi-Site Clustering & Disaster Recovery

Step3: Replicate Volumes

How to configure SAN replication between IBM Storwize V3700 systems

How to create a Microsoft Multi-Site cluster with IBM Storwize replication

Use Cases:

Use case can be determined by current workloads and future workloads plus business continuity. Deploy Veeam One to determine current workloads on your infrastructure and propose a future workload plus business continuity.  Here is a list of use cases of multi-site cluster.

  • Scale-Out File Server for application data-  To store server application data, such as Hyper-V virtual machine files, on file shares, and obtain a similar level of reliability, availability, manageability, and high performance that you would expect from a storage area network. All file shares are simultaneously online on all nodes. File shares associated with this type of clustered file server are called scale-out file shares. This is sometimes referred to as active-active.

  • File Server for general use – This type of clustered file server, and therefore all the shares associated with the clustered file server, is online on one node at a time. This is sometimes referred to as active-passive or dual-active. File shares associated with this type of clustered file server are called clustered file shares.

  • Business Continuity Plan

  • Disaster Recovery Plan

  • DFS Replication Namespace for Unstructured Data i.e. user profile, home drive, Citrix profile

  • Highly Available File Server Replication 

Multi-Site Hyper-v Cluster for High Availability and Disaster Recovery

In most of the SMB customer, the nodes of the cluster that reside at their primary data center provide access to the clustered service or application, with failover occurring only between clustered nodes. However for an enterprise customer, failure of a business critical application is not an option. In this case, disaster recovery and high availability are bonded together so that when both/all nodes at the primary site are lost, the nodes at the secondary site begin providing service automatically, or with minimal intervention.

The maximum availability of any services or application depends on how you design your platform that hosts these services. It is important to follow best practices in Compute, Network and Storage infrastructure to maximize uptime and maintain SLA.

The following diagram shows a multi-site failover cluster that uses four nodes and supports a clustered service or application.

 

image

 

The following rack diagram shows the identical compute, storage and networking infrastructure in both site.

image

Physical Infrastructure

  • Primary and Secondary sites are connected via 2x10Gbps dark fibre
  • Storage vendor specific replication software such as EMC recovery point
  • Storage must have redundant storage processor
  • There must be redundant Switches for networking and storage
  • Each server must be connected to redundant switches with redundant NIC for each purpose
  • Each Hyper-v server must have minimum dual Host Bus Adapter (HBA) port connected to redundant MDS switches
  • Each network must be connected to dual NIC from server to switches
  • Only iLO/DRAC will have a single connection
  • Each site must have redundant power supply.

Storage Requirements

Since I am talking about highly available and redundant systems design, this sort of design must consist of replicated or clustered storage presented to multi-site Hyper-v cluster nodes. Replication of data between sites is very important in a multi-site cluster, and is accomplished in different ways by different hardware vendors. You will achieve high performance through hardware or block level replication instead of software. You should contact your storage vendor to come up with solutions that provide replicated or clustered storage.

Network Requirements:

A multi-site cluster running Windows Server 2008 can contain nodes that are in different subnet however as a best practice, you must configure Hyper-v cluster in same subnet. You applications and services can reside in separate subnets. To avoid any conflict, you should use dark fibre connection or MPLS network between multi-sites that allows VLANs.

Note that you must configure Hyper-v with static IP. In a multi-site cluster, you might want to tune the “heartbeat” settings, see http://go.microsoft.com/fwlink/?LinkId=130588 for details.

Network Configuration Spread Sheet

Network

VLAN ID

NICs and Switch Ports speed

iLO/DRAC

10

1Gbps

MGMT

20

2x1Gbps

Live Migration

30

2x10Gbps

Storage Migration

40

2x10Gbps

Virtual Machine

50,60

4x10Gbps

iSCSI Network

70

4x10Gbps

Heartbeat network

80

2x1Gbps

Storage Replication

(Separate from Hyper-v)

90

Dark Fibre

2x10Gbps

Note that iSCSI network is only required if you are using IP Storage instead of Fibre Channel (FC) storage.

Cluster Selection: Node and File Share Majority (For Cluster with Special Configurations)

Quorum Selection: Since you will be configuring Node and File Share Majority cluster, you will have the option to place quorum files to shared folder. Where do you place this shared folder? Since we are talking about fully redundant and highly available Hyper-v Cluster, we have several options to place quorum shared folder.

Option1: Secondary Site

Option 2: Third Site

Visit http://technet.microsoft.com/en-us/library/cc770620%28WS.10%29.aspx for more details on quorum.

Hyper-v Cluster Configuration:

Visit https://araihan.wordpress.com/2013/06/04/windows-server-2012-failover-clustering-deep-dive/ for detailed cluster configuration guide.

Why Managed vCenter Provider cannot be called Cloud Provider?

Before I answer the question of the title of this article, let’s start with what is public cloud and how a public cloud can be defined.

In cloud computing, the word cloud (also phrased as “the cloud”) is used as a metaphor for “the Internet,” so the phrase cloud computing means “a type of Internet-based computing,” where different services and applications are delivered to an organization through the Internet.

Cloud computing is a method of computing that relies on sharing computing resources rather than having own dedicated local resources to handle workloads such as an application. In this type of computing, unused resources are released back to the pool of resources and reutilised when resources are in demand.

There may be differences in service and application offered by cloud service provider but almost all cloud service provider offer some common services, automation, compliance and utilities to tenant. Almost all major service providers have common characteristics and some has enhanced characterises when comes to cloud computing:

Example: Microsoft Azure Platform or Amazon Web Services

Common Features:

Shared Hardware: By definition public cloud is a multi-tenant environment, resources are shared among clients. Multiple clients are hosted on the same hardware, storage and network devices as the other tenants in the cloud.

Cost effective: Public clouds bring together greater levels of resource and so can benefit from the largest economies of scale. The centralised operation and management of the underlying resources is shared across all of the subsequent cloud services whilst components, such as servers, require less bespoke configuration. Some mass market propositions can even be free to the client, relying on advertising for their revenue.

Ownership and proprietary obligations: You may curious to know who owns what between your cloud provider and you. Your cloud provider owns the layer of physical hardware which you don’t have any control or say what they buy or replace. But you have the ownership of your data, intellectual properties, virtual servers and application. As long as you pay the bills and you do lawful business on hosted environment, your service provider has no rights to switch off or being regular outage on the virtual servers and application without you being notified or compensated.

Enhanced Features:

Self-management: with the high volume, utility model, self-managed systems are required for this business model to make sense. Advantage here for the tech savvy tenants that like to setup and manage the details of their own domain, servers and application by themselves. In this type of cloud based solution provides client with control of their own data and intellectual properties. Self-service is also provide a sense of ownership to a tenant who is willing to manage their own uses of the service and application and keep track of bills and data they own.

Security: Since public cloud is a multi-tenant environment, physical and logical securities are in place to protect a tenant being visible by another tenant. Security is not just placing a Cisco ASA or Juniper firewall in the front of internet and having some virtual switches in vCenter. This is the security that complies with corporate policies and regulations of each territories, the tenant resides.

Hardware Performance: In the public cloud, you cannot select the physical hardware such as compute, cache, network or storage devices. Your virtual server is placed on whatever hardware and network, the cloud provider designates for you. But you have the choice to buy various types of compute, network, load balancer, virtual IP address and storage based on your requirement such as specific IOPS and latency requirement by your application. You can chose to place virtual server with high IOPS capability and very low latency storage. Off course there will extra cost involve by doing so but you will be at least guaranteed performance of the virtual machine. Example: Azure Storage Classification

Network: Even though public cloud is a shared model but you have the choice to procure a dedicated high bandwidth secure network within the shared network guaranteed by the provider. The service provider also guarantees you the security of this network your company procured from the service provider.Example: Azure ExpressRoute

Utility Model: Public Clouds typically deliver a pay-as-you-go model, where you pay by the hour for the compute resources you use. This is an economical way to go if you’re spinning up & tearing down development servers on a regular basis.

No Contracts: Along with the utility model, you’re only paying by the hour – if you want to shut down your server after only 2 hours of use, there is no contract requiring your ongoing use of the server.

Reliability: The sheer number of servers and networks involved in creating a public cloud and the redundancy configurations mean that should one physical component fail, the cloud service would still run unaffected on the remaining components. In some cases, where clouds draw resource from multiple data centres, an entire data centre could go offline and individual cloud services would suffer no ill effect. There is, in other words, no single point of failure which would make a public cloud service vulnerable

Flexibility: There are many IaaS, PaaS and SaaS services available on the market which follow the public cloud model and that are ready to be accessed as a service from any internet enabled device. These services can fulfil most computing requirements and can deliver their benefits to private and enterprise clients alike. Businesses can even integrate their public cloud services with private clouds, where they need to perform sensitive business functions, to create hybrid clouds. Example: Azure Service fabric

Ultimate scalability: cloud resources are available on demand from the public clouds’ vast pools of resource so that the applications that run on them can respond seamlessly to fluctuations in activity. You can acquire a vast pool of resources on to your domain via self-service portal without engaging the service provider. Example: Azure Big Data

Delivery through internet: The availability of public cloud services through an internet connection ensures that the services are available wherever the client is located. This provides invaluable opportunities to enterprise such as remote access to IT infrastructure or online document collaboration from multiple locations. Examples: Microsoft Office 365.

Hybrid Deployments: If a dedicated server is required to run a high speed and high IO database application that on-premises resources can be integrated from a private cloud to public cloud, in effect, hybridising the solution between virtual servers and dedicated servers. The service provider also provides you an option to hybridise your environment you own.

To answer the question, here is my explanation why Managed vCenter Provider cannot be called Cloud Provider?

A single virtual center server is a management point of this type of service provider mostly managed by the small technology team. This type of provider is acting as a trustee of your data instead of a cloud provider. There are possible security and compliance flaws of the systems you may not aware of. There might be potential many single point of failure you may not aware. The bills you received from this type of service provider you never been verified that you truly used those services and application because there is no self-service mechanism with this unscrupulous service provider. There is potential downtime and service outage with this service provider which you have never been compensated. This type of unscrupulous service provider do not follow any service level agreement or respect the agreement they signed. You are sacrificing your productivity by relying on them to provide you a hosted service which you never received with reliably. You cannot simply call them cloud provider. A term should be introduced saying “Managed vCenter” and “Trustee of Data”.

I may be the blogger who is saying this. But here is the global researcher “Gartner Inc.” has to say on who can be called cloud service provider as on May 2015.

Garnter Magic Quadrent

Related Articles:

Understand “X as a Service” or get stuck in “Pizza box as a Service”

Gartner’s verdict on mid-range and enterprise class storage arrays

Understanding Software Defined Storage (SDS)

Manage Remote Workgroup Hyper-V Hosts with Hyper-V Manager

The following procedures are tested on Windows Server 2016 TP4 and Windows 10 Computer.

Step1: Basic Hyper-v Host Configuration

Once Hyper Server 2016 is installed. Log on to Hyper-v host as administrator. You will be presented with command prompt. On the command prompt type sconfig.cmd and hit enter

You will be presented with the following screen. For a workgroup hyper-v host perform the following tasks from the configuration window.

Task 2: Computer Name (Reboot required)

Task 4: Remote Management

Task 7: Remote Desktop

Task 8: Network Settings

Task 11: Windows Activation

Sconfig

  Step2: Configure Hyper-v Server 2016 Host

Open a PowerShell console as Administrator. Run the following command

Enable Remote Management

Configure-SMRemoting.exe -Enable

Open firewall for Remote Computer Management

Set-NetFirewallRule -DisplayGroup ‘Windows Management Instrumentation (WMI)’ -Enabled true –PassThru

Set-NetFirewallRule -DisplayGroup ‘Remote Event Log Management’ -Enabled true -PassThru

Open firewall for ping (ICMPv4)

Set-NetFirewallRule -DisplayName “File and Printer Sharing (Echo Request – ICMPv4-In)” -Enabled True -PassThru

Enable Remote Desktop and allow remote connections

cscript.exe c:\Windows\System32\SCregEdit.wsf /AR 0

Enable Remote disk management

Set-NetFirewallRule -DisplayGroup ‘Remote Volume Management’ -Enabled true -PassThru

Enable Anonymous logon from remote computer

Navigate to c:\windows\system32 folder Type dcomcnfg.exe , Expand Computers, Right Click on My Computer, Click Properties, Click Com Security, On the Access Permission Window, Click Edit Limits, Click ‘anonymous logon’ Select local access and remote access, Click Ok.

Step3: Configure Windows 10 Computer or Windows Server 2016 with Desktop Experience

If you choose to manage workgroup hyper-v host from Windows 10 computer, enable Hyper-v feature on Windows 10 computer. Open a PowerShell console as Administrator. Run the following command

Enable Hyper-v Feature on Windows 10

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All

Add Hyper-v Role in Windows Server 2016

Install-WindowsFeature –Name Hyper-V -IncludeManagementTools -Restart

Add a Host (A) Record

Open Windows Explorer or navigate to C:\Windows\System32\Drivers\etc

Type Notepad hosts

Add IP address and host name of the remote hyper-v host. Save and exist notepad

Enable Remote Management

winrm quickconfig

For Managing remote systems, add remote hyper-v host as a trusted computer

winrm set winrm/config/client @{TrustedHosts=”Name of the Remote Hyper-v Server”}

Enable remote disk Management (add this command on both systems) firewall rule

Set-NetFirewallRule -DisplayGroup ‘Remote Volume Management’ -Enabled true -PassThru

Enable Anonymous logon from remote computer

Navigate to c:\windows\system32 folder Type dcomcnfg.exe , Expand Computers, Right Click on My Computer, Click Properties, Click Com Security, On the Access Permission Window, Click Edit Limits, Click ‘anonymous logon’ Select local access and remote access, Click Ok.

Step3: Connect to Remote Hyper-v Host

Open Hyper-v Manager, Right Click on Hyper-v Manager, Click Connect to Server.., Type the host name you added in hosts file,

To specify credentials for the remote Hyper-V host, select Connect as another user: in the Select Computer dialogue box then select Set User….

Type HostName\Administrator and Password , Click Remember Password, Click Ok

Click Ok.

 

References

Using Windows 10 Client Hyper-V

Install Hyper-V on Windows 10

Enable and Use Remote Commands in Windows PowerShell

VMware Increases Price Again

VMware increases price again. As per VMware pricing FAQ, the following pricing model will be in effect on April 1, 2016.

vSphere with Operations Management Enterprise Plus from US$4,245/CPU to US$4,395/CPU

VMware vCenter Server™ Standard from US$4,995/Instance to US$5,995/Instance

vSphere with Operations Management Enterprise Plus now includes enhancements to Workload Placement, and vCenter Server™ Standard now includes 25 Operating System Instances of VMware vRealize® Log Insight™ for vCenter.

vSphere Enterprise and vSphere with Operations Management Enterprise customers also entitled for a 50% discount on optional upgrade to vSphere Enterprise Plus and vSphere with Operations Management Enterprise Plus. This offer is valid until June 25, 2016.

Relevant Information

VMware Licensing FAQ

Hyper-v Server 2016 licensing FAQ

Windows Server 2016 datasheet

Windows Server 2016

Understanding Software Defined Networking (SDN) and Network Virtualization

The evolution of virtualization lead to an evolution of wide range of virtualized technology including the key building block of a data center which is Network. A traditional network used be wired connection of physical switches and devices. A network administrator has nightmare making some configuration changes and possibility of breaking another configuration while doing same changes. Putting together a massive data center would have been expensive venture and lengthy project. Since the virtualization and cloud services on the horizon, anything can be offered as a service and almost anything can virtualised and software defined.

Since development of Microsoft SCVMM and VMware NSX, network function virtualization (NFV), network virtualization (NV) and software defined network (SDN) are making bold statement on-premises based customer and cloud based service provider. Out of all great benefits having a software defined network, two key benefits standout among all which are easy provisioning a network and easy change control of that network. You don’t have to fiddle around physical layer of network and you certainly don’t have to modify virtual host to provision a complete network with few mouse click. How does it work?

Software Defined Networking- Software defined networking (SDN) is a dynamic, manageable, cost-effective, and adaptable, high-bandwidth, agile open architecture. SDN architectures decouple network control and forwarding functions, enabling network control to become directly programmable and the underlying infrastructure to be abstracted from applications and network services. Examples of Cisco software defined networking is here.

The fundamental building block of SDN is:

  • Programmable: Network control is directly programmable because it is decoupled from forwarding functions.
  • Agile: Abstracting control from forwarding lets administrators dynamically adjust network-wide traffic flow to meet changing needs.
  • Centrally managed: Network intelligence is (logically) centralized in software-based SDN controllers that maintain a global view of the network, which appears to applications and policy engines as a single, logical switch.
  • Programmatically configured: SDN lets network managers configure, manage, secure, and optimize network resources very quickly via dynamic, automated SDN programs, which they can write themselves because the programs do not depend on proprietary software.
  • Open standards-based and vendor-neutral: When implemented through open standards, SDN simplifies network design and operation because instructions are provided by SDN controllers instead of multiple, vendor-specific devices and protocols.

Cisco SDN Capable Switches

Modular Switches

Cisco Nexus 9516
Cisco Nexus 9508
Cisco Nexus 9504

Fixed Switches

Cisco Nexus 9396PX
Cisco Nexus 9396TX
Cisco Nexus 93128TX
Cisco Nexus 9372PX
Cisco Nexus 9372TX
Cisco Nexus 9336PQ ACI Spine Switch
Cisco Nexus 9332PQ

Network Virtualization- A virtualized network is simply partitioning existing physical network and creating multiple logical network. Network virtualization literally tries to create logical segments in an existing network by dividing the network logically at the flow level. End goal is to allow multiple virtual machine in same logical segment or a private portion of network allocated by business. In a physical networking you cannot have same IP address range within same network and manage traffic for two different kind of services and application. But in a virtual world you can have same IP range segregated in logical network. Let’s say two different business/tenant have 10.124.3.x/24 IP address scheme in their internal network. But both business/tenant decided to migrate to Microsoft Azure platform and bring their own IP address scheme (10.124.3.x/24) with them. It is absolutely possible for them to retain their own IP address and migrate to Microsoft Azure. You will not see changes within Azure portal. You even don’t know that another organisation have same internal IP address scheme and possibly hosted in same Hyper-v host. It is programmatically and logically managed by Azure Stack and SCVMM network virtualization technology.

Network Functions Virtualization- Network function virtualization is virtualising layer 4 to layer 7 of OSI model in a software defined network. NFV runs on high-performance x86 platforms, and it enables users to turn up functions on selected tunnels in the network. The end goal is to allow administrator to create a service profile for a VM then create logical workflow within the network (the tunnel) and then build virtual services on that specific logical environment. NFV saves a lot of time on provisioning and managing application level of network. Functions like IDS, firewall and load balancer can be virtualised in Microsoft SCVMM and VMware NSX.

Here are some Cisco NFV products.

IOS-XRv Virtual Router: Scale your network when and where you need with this carrier-class router.

Network Service Virtualization- Network Service Virtualization (NSV) virtualizes a network service, for example, a firewall module or IPS software instance, by dividing the software image so that it may be accessed independently among different applications all from a common hardware base. NSV eliminates cost of acquiring a separate hardware for single purpose instead it uses same hardware to service different purpose every time a network is accessed or service is requested. It also open the door for service provider offer security as a service to various customer.

Network security appliances are now bundled as a set of security functions within one appliance. For example, firewalls were offered on special purpose hardware as were IPS (Intrusion Protection System), Web Filter, Content Filter, VPN (Virtual Private Network), NBAD (Network-Based Anomaly Detection) and other security products. This integration allows for greater software collaboration between security elements, lowers cost of acquisition and streamlines operations.

Cisco virtualized network services available on the Cisco Catalyst 6500 series platform.

Network security virtualization

  • Virtual firewall contexts also called security contexts
  • Up to 250 mixed-mode multiple virtual firewalls
  • Routed firewalls (Layer 3)
  • Transparent firewalls (Layer 2, or stealth)
  • Mixed-mode firewalls combination of both Layer 2 and Layer 3 firewalls coexisting on the same physical firewall. 

Virtual Route Forwarding (VRF) network services

  • NetFlow on VRF interfaces
  • VRF-aware syslog
  • VRF-aware TACACS
  • VRF-aware Telnet
  • Virtualized address management policies using VRF-aware DHCP
  • VRF-aware TACACS
  • Optimized traffic redirection using PBR-set VRF

Finally you can have all these in one basket without incurring cost for each component once you have System Center Virtual Machine Manager or Microsoft Azure Stack implemented in on-premises infrastructure or you choose to migrate to Microsoft Azure platform.

Relevant Articles

Comparing VMware vSwitch with SCVMM Network Virtualization

Understanding Network Virtualization in SCVMM 2012 R2

Cisco Nexus 1000V Switch for Microsoft Hyper-V

How to implement hardware load balancer in SCVMM

Understanding VLAN, Trunk, NIC Teaming, Virtual Switch Configuration in Hyper-v Server 2012 R2