Windows Server Patching Best Practices

This article provides actionable advice about how to manage patches to reduce downtime while still maintaining the security of software services through the proactive reduction of dependencies and the use of workaround solutions.

Patching Requirements

Windows Server patches, hotfixes and service pack is critical for compliance, service level agreement and security purposes. Keeping an operating systems and application up to date is the key to align your infrastructure with latest software. Patches and hotfixes also enable you to prevent any security breaches and malware infection.

Windows Patch Classification

The following are strongly recommended patches:

  1. Critical
  2. Security
  3. Definition Updates for malware
  4. Service packs

Windows Product Classification

It is highly recommended that you patch Windows Servers, Windows Clients, Office, Applications (Silverlight, .Net Framework, SQL, Exchange, SharePoint, FF TMG).

Patching Groups

Consultants should take time to test the patches in a non-production environment prior to being deployed to production. This will help to gauge the impact of such changes. Ideally you will have the following patching groups:

1. UAT (UAT1, UAT2, etc)

2. Test Environment (Test1, Test2, etc)

3. Development Environment (Dev1, Dev2 etc)

4. Production (Prod1, Prod2, etc)

If you have clustered environment like SQL, Exchange and SharePoint then create Prod1, prod2 group and place each node on each group.

Change Management

System administrators should maintain a log, written or electronic, of all changes to the operating environment, to include hardware, system security software, operating system, and applications. Prior to any changes being implemented on a system, the system administrator should receive approval of stakeholders.


Why am I discussing backup with patching best practice? In case of emergency you can rollback completely and restore a server to its original state if necessary. It is very important that servers be backed up on a regular basis. Depending on the use of the server, it may be adequate to backup the server once per week. A backup of a more critical environment may be needed daily, and possibly continuously. The backup program provided with Windows is capable of backing up to virtually any writable media, which can include network drives provided by a server in another physical location. This program is also capable of scheduling backups which can ensure backups occur on a regular interval.

Microsoft strongly recommends that you create the following backups before you install an update rollup, service pack and patch on Exchange and SQL:

  • A full backup of all databases on the server.
  • A full backup of transaction log and log backup
  • A system state backup of the server.
  • A snapshot of virtualized exchange server. Delete snapshot after successful patching and updating.

Application Compatibility

Read release notes of each hotfixes you are going to apply so that you are compliant with the application installed on the server. Consult with application vendor before applying service pack to any server if the server is hosting specific business application. Consult with application engineer about the importance of server patching. Inform and educate application engineer as much as possible to avoid conflict of interest.


Documentation released with the updates is usually in the form of web pages, attached Word documents and README.TXT files. These should be printed off and attached to change control procedures as supporting documentation.

Back out Plan

A back-out plan will allow the system and enterprise to return to their original state, prior to the failed implementation. It is important that these procedures are clear, and that contingency management has tested them, because in the worst case a faulty implementation can make it necessary to activate contingency options. Historically, service packs have allowed for uninstalling, so verify there is enough free hard disk space to create the uninstall folder. Create a back out plan electronically and attach with change management software.

User Notifications

You need to notify helpdesk staff and support agencies of the pending changes so they may be ready for arising issues or outages.

Consistency across Servers

Always install the same service packs or hotfixes to each SQL server node, Exchange DAG member and Domain Controller.

Routine Maintenance Window

A scheduled maintenance window must be agreed with business so that application outage and server reboot can maintain a respectable Service Level Agreement (SLA). If you have a large infrastructure with thousands of servers and many regions working round the clock then you must consider application dependencies. A patching schedule can be considered in between every Friday of every month at 6:00 P.M. Friday to 6:00 A.M Monday. Setup maintenance window in system center or deadline for WSUS to make sure patches are applied when you want instead of when patch is available. In this way you will have a complete control over change windows approved by change advisory board (CAB). Do not allow end users to update patches on their client machine according to their wishes and happiness! then user will never install any patch.

Patching Tools

I strongly recommend that you spend few $$$ to buy Microsoft System Center 2012 to manage and deploy Windows patches, service pack and hotfixes. However you can use Windows Server Update Services (WSUS) as poor man’s patching solutions.

Patching DMZ server can be accomplished using WSUS offline patching solutions available for free to download from

Automate, Automate and Automate!

Automated patch management using System Center could enable a single IT administrator to access a pre-populated patch policy. He then could execute the command and with the press of a single button, download the patches from Microsoft’s website, install them on a test machine and test for compatibility issues. Meanwhile, an automatic inventory check could search for systems with the affected software, wake them up, check their readiness and push the verified patches out to waiting machines. The patches would then be automatically installed on each system, and they’d reboot as necessary. The final step is an automated report on the status of the remediated devices.

Standardize Patch Management Processes

Standardized patch management processes could allow for daily assessment and remediation of client devices and weekly assessment and remediation for servers. Reports can then be generated to validate system status on a weekly or bi-weekly schedule. A systems monitoring task that used to take days now takes minutes, and patches are deployed more completely and consistently across the entire IT environment. A single IT administrator can proactively manage thousands of systems tasks in the same amount of time it took an entire team to do the tasks manually.

Reboot Windows Computer

Some application may require reboot of server before patching such as RSA Secure Console. However most of the server must be rebooted after patching. Do not suppress reboot after patching in any circumstances or you will have a messy environment and broken clusters.

X86 and X64 Windows Systems

The most prominent 32-bit application you’re likely to see on a 64-bit Windows system is Office. In this sort of situation System Center benefits most because you can adjust and make decision based on architecture and compliance as well. You can approve patches based on “Needed and Not Installed”. If a server or client need update it will install if not then it will not installed. It’s safe to do so.

Antivirus and Antispyware

Servers are vulnerable to many forms of attack. Implementation and standardization of security methods should be developed to allow early and rapid deployment on servers. It’s important that a Windows server be equipped with a latest centrally managed Antivirus program. Antivirus update must be scheduled with the same maintenance window to update antivirus with latest definition.

Audit Practices

Servers have a powerful auditing feature built in. Typically, server managers would want the auditing system to capture logins, attempted logins, logouts, administrative activities, and perhaps attempts to access or delete critical system files. Auditing should be limited to gathering just the information that is needed, as it does require CPU and disk time for auditing to gather information. Log Management software should be used, if possible, for ease of managing and analysing information. Report can be generated from Systems Center and WSUS as proof of patching cycle.

Log Retention

Servers keep multiple logs and, by default, may not be set to reuse log file entries. It is a good practice to expand the size of the allowed log file and to set it to reuse space as needed. This allows logging to continue uninterrupted. How far back your log entries go will depend on the size of the log file and how quickly you are accumulating log data. If your server environment is critical, you may wish to ensure that the log file size is sufficient to store about 30 days of logging information, and then rotate log files once per month.

Installing Updates on a single Exchange Server

Download Exchange Update from Microsoft Download Center. Record Current Exchange Version information

Check for publisher’s certificate revocation

1. Start Internet Explorer.

2. On the Tools menu, click Internet Options.

3. Click the Advanced tab, and then locate the Security section.

4. Clear the Check for publisher’s certificate revocation check box, and then click OK.

5. After the update rollup installation is complete, select the Check for publisher’s certificate revocation option.

Pre-check before installing

1. Determine which update rollup packages are installed on your Exchange server roles

2. Determine whether any interim updates are installed

3. Review interim updates

4. Obtain the latest update rollup package

5. Apply on a Test Exchange Server

Install Exchange Update

1. Ensure that you have downloaded the appropriate rollup to a local drive on your Exchange servers, or on a remote network share.

2. Run the Windows Installer *.msp Setup file that you downloaded in step 1.

Install Exchange Update on DAG Member

To update all DAG members, perform the following procedures on each DAG member, one at a time. Set the member server in maintenance mode using this PowerShell Command.

.StartDagServerMaintenance.ps1 <ServerName>

Install the update rollup

1. Close all Exchange management tools.

2. Right-click the Exchange update rollup file (.msp file) you downloaded, and then select Apply.

3. On the Welcome page, click Next.

4. On the License Terms page, review the license terms, select I accept the License Terms, and then click Next.

5. On the Completion page, click Finish.

Once installed exit from maintenance mode run the StopDagServerMaintenance.ps1 script. Run the following command to re-balance the DAG, as needed

.RedistributeActiveDatabases.ps1 -DagName <DAGName> -BalanceDbsByActivationPreference -ShowFinalDatabaseDistribution

When the installation is finished, complete the following tasks:

  • Start the Services MMC snap-in, and then verify that all the Exchange-related services are started successfully.
  • Log on to Outlook Web App to verify that it’s running correctly.
  • Restore Outlook Web App customizations, and then check Outlook Web App for correct functionality.
  • After the update rollup installation is complete, select the Check for publisher’s certificate revocation option in Internet Explorer. See “Certificate Revocation List” earlier in this topic.
  • Check Exchange 2010 version information
  • View Update rollup in Control Panel>Programs and Features

Patching Microsoft Failover Cluster

You can install Windows service packs on Windows Server Failover Cluster nodes using the following procedure. Administrative privilege is required to perform the following tasks.

Procedure to install Windows service pack or hotfixes in Windows Server 2003:

  1. Check the System event log for errors and ensure proper system operation.
  2. Make sure you have a current backup and updated emergency repair disk for each system. In the event of corrupt files, power outage, or incompatibility, it may be necessary to revert back to the state of the system prior to attempting to install the service pack/hotfixes.
  3. Expand Node A, and then click Active Groups. In the left pane, right-click the groups, and then click Move Group to move all groups to Node B.
  4. Open Cluster Administrator, right-click Node A, and then click Pause Node.
  5. Install the service pack on Node A, and then restart the computer.
  6. Check the System event log for errors. If you find any errors, troubleshoot them before continuing this process.
  7. In Cluster Administrator, right-click Node A, and then click Resume Node.
  8. Right-click Node B, and then click Move Group for all groups owned by Node B to move all groups to Node A.
  9. In Cluster Administrator, right-click Node B, and then click Pause Node.
  10. Install the service pack on Node B, and then restart the computer.
  11. Check the system event log for errors. If you find any errors, troubleshoot them before continuing this process.
  12. In Cluster Administrator, right-click Node B, and then click Resume Node.
  13. Right-click each group, click Move Group, and then move the groups back to their preferred owner.

Procedure to install Windows service pack or hotfixes in Windows Server 2008 and Windows Server 2012:

  1. Check the event log for errors and ensure proper system operation.
  2. Make sure you have a current backup and updated emergency repair disk for each system. In the event of corrupt files, power outage, or incompatibility, it may be necessary to revert back to the state of the system prior to attempting to install the service pack/hotfixes.
  3. On Node A, Expand Services and Applications, and then click the service or application
  4. Under Actions (on the right), click Move this service or application to another node, then choose the node or select Best possible.
  5. In the Failover Cluster Manager snap-in, right-click Node A, and then click Pause.
  6. Install the service pack/hotfixes on Node A, and then restart the computer.
  7. Check the event log for errors. If you find any errors, troubleshoot them before continuing this process.
  8. In Failover Cluster Manager snap-in, right-click Node A, and then click Resume.
  9. Under Actions (on the right), click Move this service or application to another node, then choose the node.
    Note: As the service or application moves, the status is displayed in the results pane (in the center pane). Follow the Step 9 and 10 for each service and application configured on the cluster.
  10. Install the service pack/hotfixes on Node B, and then restart the computer.
  11. Check the event log for errors. If you find any errors, troubleshoot them before continuing this process.
  12. From the Failover Cluster Manager snap-in, right-click Node B, and then click Pause.
  13. In Failover Cluster Manager, right-click Node B, and then click Resume.
  14. Right-click each group, click Move Group, and then move the groups back to their preferred owner.

You can use the following PowerShell Cmdlet to accomplish the same.

1. Load the module with the command: Import-Module FailoverClusters

2. Suspend (Pause) activity on a failover cluster nodeA: Suspend-ClusterNode nodeA

3. Move a clustered service or application (a resource group) from one node to another: Get-ClusterNode NodeA | Get-ClusterGroup | Move-Cluster Group

4. Resume activity on nodeA that was suspended in step 5: Resume-ClusterNode nodeA

5. Move a clustered service or application (a resource group) from one node to another: Get-ClusterNode NodeB | Get-ClusterGroup | Move-Cluster Group

6. Suspend (Pause) activity on other failover cluster node: Suspend-ClusterNode nodeB

7. Resume activity on nodeB that was suspended in step 10 above: Resume-ClusterNode nodeB


It is critical that when service packs, hotfixes, and security patches are required to be installed, that these best practices be followed.

Bottom line

1. Read all related documents.

2. Use a change control process.

3. Apply updates that are needed.

4. Test patches and hotfixes on test environment.

5. Don’t get more than 2 service packs behind.

6. Target non-critical servers first.

7. Service Pack (SP) level consistency.

8. Latest SP instead of multiple hotfixes.

9. Apply only on exact match.

10. Subscribe to Microsoft email notification.

11. Always have a back-out plan.

12. Have a working Backup and schedule production downtime.

13. Consistency across Domain Controllers and application servers.

Additional Readings:

SQL Server failover cluster rolling patch and service pack process

Patch Management on Business-Critical Servers

First Cumulative Update for Exchange 2013

Cumulative update 1 for Exchange Server 2013 (KB2816900)

Update Rollup 10 for Exchange Server 2007 Service Pack 3 (KB2788321)

Details can be found here

FF TMG 2010—Can future be altered?

I read the following articles about Microsoft Forefront TMG 2010. I was shocked by the news. TMG 2010 is one of the beautiful product Wintel Engineers and Security Administer can be proud off. I believe I am one of the biggest admirer of Forefront Product lines.

                                                                    Death of TMG? by Deb Shinder 

What will happen with TMG?

The demise of Threat Management Gateway: Is Microsoft backing away from the edge?

I would like to voice my own opinion on this matter. I am sure I will find lots of similar minded techie out there who would love to share same opinion as me. I would like to send an open request to Microsoft Corp and MVPs to pursue for an advanced version of TMG that incorporate cloud security and address modern day security challenges.

I decided to write on a different perspective of TMG 2010 what I would like to see next service pack of Forefront Threat Management Gateway or in a future version if there is one. This is not an official account of Microsoft Corp. This is just my wish list. I hope and cross my finger that Microsoft will listen to those who are on the field working for a better and even bigger Microsoft community.   

FF TMG 2010: Here is details of evolution of today’s TMG 


TMG 2010 can be more advanced in terms Firewall Policy, Publishing Rules and Cloud Security. TMG 2010 may be available in Downloadable virtual Appliance build on Windows Server “Code name 8” and physical appliance through the Microsoft partners program. Microsoft declared TMG 2010 is in sustainable mode and will not invest on TMG for further development so my dream to administer TMG administration console via internet explorer and Silverlight will be just a dream. I would like to see TMG service pack as separate installed and TMG 2010+SP3 integrated together in a installer for those who wants to refresh TMG and adopt as a new customer.

Topology and Installation Changes: I would like to see a Hyper-V network incorporated into TMG. As you all know when installing TMG, TMG installer prompt you for subnets of Local area network. The new version will prompt you to add your cloud networks in an installation window. The installer will secure the local area network and private cloud network using default configuration which you will be able to modify and align later on with your desired topology and network layout.  


Incorporating Cloud Security:

clients and partners have serious concern over the years about Service provides who sells cloud solutions. For example, service provider selling Exchange cloud, SharePoint cloud, Anti-Spam  and Security Cloud Solution. There are questions to be asked when you buying public cloud solutions. This is not just having a hypervisor and virtual center. what about application security, identity and governance. How would to address your client’s concern of internal threat and external threat. How client will trust a provider when they place their data in somewhere service provider’s cloud.

Microsoft can/should/must address these issues by providing Security as a service. Forefront TMG can play a key role if Microsoft is willing take a step ahead to the bottom line.

  • Application security
  • Privacy
  • Legal issues
  • Availability
  • Identity management
  • Compliance
  • Business Continuity and data recovery
  • Data Security

Firewall Rules: New Publishing Tools in Tasks pan should include

  • Publish FTP Servers
  • Publish Lync Server
  • Publish Streaming Media Server
  • Secure Cloud Network


Configure IM and Social media policy: Web Access Policy Tasks Pan should include

  • Configure IM Access (Allow/Deny Skype/Lync/MSN/Yahoo Messenger)
  • Configure Social Media Access (Allow/Deny Social Media such as Twitter/FaceBook/Google+/Youtube)


Networks: Network rules incorporate a build-in cloud network and network rules establishing communication from LAN to Cloud network and External to Cloud network. During installation of TMG; allow rules to be configured automatically when selecting Hyper-V Server in DMZ.


Multicast NLB Configuration: NLB Properties should be added another check box to create firewall rule for Multicast NLB in a virtualized environment. That means Multicast NLB mac address can communicate within array members in a virtualized environment if there is strict security policy deployed through out the infrastructure.


List of New Protocol available: New Protocols includes following protocols and many more:

  • Cloud Protocols
  • Lync Protocol
  • Hyper-v Protocols


Generate offline Certificate request: There should be an option to generate offline certificate request in Systems>Tasks pan.


Integrating Bing Search with TMG 2014 Cache: Search result cached in TMG from Bing Search Engine and presented to client.

Bandwidth Management: TMG should be able to manage bandwidth by single user, multiple users, AD Security groups, IP address, Computer Name, Department, Site, Branch.

Configure Branch or Site TMG Server: Option can be selected during installation of TMG 2010+SP3 (integrated installer) whether TMG is a primary site or branch site. Selecting Branch Site will auto configure site server with site to site VPN (if selected) and even replicate with primary sites firewall rules and policies (depending on topology). when installing a branch TMG branch TMG will automatically create branch cache depending on selection of topology .

Reporting: Following are the examples of the reports will be available in TMG 2010 SP3. there will be many more.


  • User based report
  • AD Security Group Based report
  • Web Site Visited
  • IP Address visited
  • Web/Content Uses report
  • Download reports by users/Group/Department
  • Bandwidth Uses report
  • Caching report
  • Search Engine Visitor by Search Engine report
  • Real Time/Custom Traffic report
  • Traffic Trending report
  • Top 20 Net users
  • Top 20 Site Visited
  • Default Monthly report
  • Default Yearly report
  • TMG Health report

Audit and Change Management: TMG will include complete change manage and recording of Tasks/Events generated by role based user and systems itself.

Role based TMG management: TMG Workgroup Deployment and Domain Member deployment should include RBAC management.

  • Administrator
  • Organization Administrator (member of this group manages cluster of Arrays )
  • Backup operator (Commvault/Symantec Client/SCDPM client integrated)
  • Auditor/User (view permission)
  • Firewall Rules and Web Access Policy Operator
  • Single or Multiple array administrator

Tool Box: Pre-installed BPA, Troubleshooting, Monitoring & Capturing  Real Time Traffic.

Learn more about TMG here .

Error message when you try to install Exchange Server 2010 SP2: “AuthorizationManager check failed”

Error: Message :



1. Exchange Servers placed in a OU which has GPO applied to them.

2. PowerShell Execution Policy set to unrestricted or remote signed.



Step1: Create a separate Organizational Unit in Active Directory and place Exchange Servers in that  Organizational Unit . Do not apply any GPO on newly created Organizational Unit.

Step2: Log on to Exchange Server. Start Menu>Run>gpedit.msc


Right click on Local Computer Policy>Property>Disable computer Configuration settings and User Configuration Settings


Step3: Open PowerShell> issue the following command

Set-ExecutionPolicy –Scope LocalMachine –ExecutionPolicy Undefined –Confirm –Force

Step4: Reboot Server. Once rebooted log back on to the Exchange Server. check execution policy by issuing the command

Get-ExecutionPolicy –List


Step5:  Start Menu>Run>services.msc . Stop any backup software and Antispam software services on the server.

Step6: Upgrade HT/CAS Server: Download Exchange 2010 SP2 and install Exchange SP2 by the issuing the following command in PowerShell /M:Upgrade /InstallWindowsComponents

Apply Service Pack 2 to HT and MBX server first. If you have multiple servers in HT/CAS Array than you can apply service to one exchange array member. your exchange infrastructure still be functional and service mail systems.






Step7: download Update Rollup1 for Exchange 2010 SP2 and apply rollup1.






Step7: Upgrade Mailbox Server: Log on to MBX server. Open Exchange Management Console>Click Server Configuration>Select Mailbox>Select Server>Click Switchover Server>Browse and Select a server>click ok. Wait few minutes to finish the operation. Check the mailbox node again. It should show Is Active: False.


Now follow the previous steps to upgrade to SP2 and Rollup1.

Caution: Take a snapshot if Exchange is a virtual server. If exchange 2010 SP2 installation fails for another reason revert the snapshot back to original. Exchange will still be functional even active directory schema is upgraded by exchange SP2 installer.

If server is physical than the following URL might be handy for you.

Recovery Databases

Understanding Backup, Restore and Disaster Recovery

Recover an Exchange Server

Reference Microsoft KB2668686

Exchange 2010 SP2 is available for download

Microsoft Exchange Server 2010 SP2 is available to download from Microsoft download center. Download link and benefits of SP2 is here. Read systems requirement and release notes before you proceed installation. You may need to backup/snapshot(if virtualized) exchange servers before final installation.

FF TMG 2010 Service Pack 2 is Now Available

Before you start installing TMG 2010 SP2, make sure you have the following infrastructure ready.

  1. TMG 2010 installed on Win2k8 or Win2k8 R2 Server.
  2. TMG 2010 SP1  and TMG 2010 Service Pack 1 Update 1 installed on top of TMG 2010.
  3. Download FF TMG 2010 SP2 and save on your server.

Pre-cautions: Take following steps before you run service pack installer

Verify/Note Current version


Check any alerts/issue in TMG 2010 server


Check event logs for any existing underlying issues

Back up an enterprise configuration: In the Forefront TMG Management console, in the tree, click the Enterprise node. On the Tasks tab>click Export Enterprise Configuration.


To export confidential information, such as user passwords and certificates, select Export confidential information and provide a password. Confidential information is encrypted during the export process. The password you enter here will be required to import the configuration.
To export user permissions, select Export user permission settings.
In Save this data in this file, specify the folder in which the export file will be saved, and the file name. In File name, enter a name for the exported file.

Important! To restore an enterprise configuration

In the Forefront TMG Management console, in the tree, click the Enterprise node>On the Tasks tab>click Import Enterprise Configuration.

Select the file that you saved when you exported the configuration.

Select Overwrite (restore) to restore configuration settings. If you exported user permissions, select Import user permission settings. If you exported confidential information, enter the password that you specified when you exported the file.

Install TMG 2010 SP2 on a TMG standalone server:

Installing SP2 in TMG 2010 standalone server is pretty straight forward.

Open elevated Command prompt, locate directory where you saved TMG 2010 SP2


run TMG-KB2555840-amd64-ENU or TMG-KB2555840-x86-ENU based on your architecture.







Install TMG 2010 Sp2 on Enterprise Array Members:

  • In-place upgrade
  1. Install the service pack on the EMS master with same credentials that were used to install the EMS during the initial Forefront TMG setup otherwise setup will fail.
  2. upgrade first the reporting server and then the array members.
  3. Install Service Pack 2 to all EMS array members.
  • Clone array upgrade
  1. Install Forefront TMG Enterprise Management on a different computer.
  2. Create a new array and import the previously exported enterprise configuration.
  3. Install the service pack on cloned EMS
  4. disjoin array members from the reporting server from the array, installing the service pack, and then joining it to the new array that is running the service pack. Continue the process with the other array members.

Installation steps for servers that use load balancing If the server is load-balanced by using network load balancing (NLB) or any other load-balancing mechanism, do the following:

  • Remove the server from the load-balancing configuration.
  • Drain existing connections that are served by the server.
  • Set NLB to suspended to prevent auto-rejoin when you restart.
  • Install the update.
  • Restart the server if it is required.
  • Start NLB on the updated server.
Post installation notes:
  1. Forefront TMG services may not start or may not sync with EMS after you install or remove a service pack. In this case, use the Monitoring node of the Forefront TMG Management console to manually restart the services.
  2. If you are logging to a remote SQL database, you are required to migrate the log database to the new schema. For instructions, see Upgrading a remote SQL database for Forefront TMG SP1
  3. Run BPA in TMG 2010 and check event logs as best practice.

Known issues: The following issues relate to the configuration and operation of Forefront TMG SP2:

  • Reload failure with local user

    Issue: After configuring the Firewall service user as a local user, reloading the configuration fails.

    Workaround: Configure a domain user for the Firewall service. See Kerberos authentication on an NLB array.

  • Uninstall failure

    Issue: After configuring the Firewall service user as a domain user, you cannot uninstall Forefront TMG SP2.

    Workaround: Reconfigure the Firewall service user to be the network service, then you can uninstall Forefront TMG SP2.