Friday, September 15, 2017

SQL Server – Uninstall /Remove SQL Server 2012 Failover Cluster Instance

Procedure to uninstall a SQL Server failover clustered instance

To update or remove a SQL Server failover cluster, you must be a local administrator with permission to login as a service on all nodes of the failover cluster.

To remove a node from an existing SQL Server failover cluster

  1. Insert the SQL Server installation media. From the root folder, double-click setup.exe. To install from a network share, navigate to the root folder on the share, and then double-click Setup.exe.
  2. The Installation Wizard launches the SQL Server Installation Center. To remove a node to an existing failover cluster instance, click Maintenance in the left-hand pane, and then select Remove node from a SQL Server failover cluster.
  3. The System Configuration Checker will run a discovery operation on your computer. To continue, Click OK. .
  4. After you click install on the Setup Support Files page, the System Configuration Checker verifies the system state of your computer before Setup continues. After the check is complete, click Next to continue.
  5. On the Cluster Node Configuration page, use the drop-down box to specify the name of the SQL Server failover cluster instance to be modified during this Setup operation. The node to be removed is listed in the Name of this node field.
  6. The Ready to Remove Node page displays a tree view of options that were specified during Setup. To continue, click Remove.
  7. During the remove operation, the Remove Node Progress page provides status.
  8. The Complete page provides a link to the summary log file for the remove node operation and other important notes. To complete the SQL Server remove node, click Close.

Screenshots: 
1 2 3 4 5 6 7


Repeat above steps on all nodes used for the SQL Server clustered instance. Make sure you uninstall the active node last.

Note:    Remove Passive Nodes First and Active node Last.

Wednesday, September 13, 2017

What is Interactive Services Detection and Why is it Blinking at Me?

Have seen this button flashing on the taskbar?
image
When you click on the button, you get this dialog.
image
If you click “View the message”, your screen blinks and you are taken to a blank desktop with a couple of dialog boxes.

Why is this Happening?

Services and system processes run in session 0. Prior to Vista, the console (first logged on user’s desktop) ran in session 0 as well. Vista introduced session 0 isolation to protect services from elevation of privilege exploits from the console desktop. Now, the first user’s desktop runs in session 1.
Interactive Services Detection (the blinking button on the taskbar) is a mitigation for legacy applications that detects if a service is trying to interact with the desktop. This is handled by the Interactive Services Detection (UI0Detect) service.
When you choose “View the message”, you are taken to session 0’s desktop and you can only interact with the dialog or message that services have tried to display on the desktop.

Behavior Depends on Your Bits

The Interactive Services Detection service is set to start “manually”. This means that it won’t start automatically when the system boots.
image
On a 32-bit version of Windows, the OS will detect desktop interaction and start the UI0Detect service and you will see the flashing taskbar button.
On a 64-bit version of Windows, if the service is a native 64-bit application, the OS will not start the UI0Detect service. Therefore, the service that is trying to interact with the desktop will appear to hang.  If you need the mitigation for a 64-bit service, you will need to be sure the service is running in order to get the mitigation.
Here’s the really weird part… If you have a 32-bit service on a 64-bit Windows, WOW64 will start the UI0Detect service and you will see the mitigation. The reason for this is that the WOW64 environment will behave as close to native 32-bit as possible – including mitigations.
A way to play around with Interactive Services Detection is to use PsExec tool.
For example, if we try this:
> psexec \\localhost -i 0 calc.exe


This will launch the calculator application in session 0. This is a quick way to simulate a service interacting with the desktop. I find this example interesting.  PsExec is a 32 bit application. Therefore, UI0Detect will always get started even though calc.exe is 64-bit. Here’s a snip from Process Explorer.
image
We can see that UI0Detect service is started in session 0 which creates a new process in session 1.  The UI0Detect process in session 1 is the Interactive Service Detection dialog. PSEXESVC.exe is the PsExec command service and note that it is a 32-bit app in session 0. Calc.exe is started in session 0 because we specified session 0 as an argument to PsExec.

Friday, September 1, 2017

Fixing Disk Signature Collisions

Disk cloning has become common as IT professionals virtualize physical servers using tools like Sysinternals Disk2vhd and use a master virtual hard disk image as the base for copies created for virtual machine clones. In most cases, you can operate with cloned disk images unaware that they have duplicate disk signatures. However, on the off chance you attach a cloned disk to a Windows system that has a disk with the same signature, you will suffer the consequences of disk signature collision, which renders unbootable any of the disk’s installations of Windows Vista and newer. Reasons for attaching a disk include offline injection of files, offline malware scanning , and - somewhat ironically - repairing a system that won’t boot. This risk of corruption is the reason that I warn in Disk2vhd’s documentation not to attach a VHD produced by Disk2vhd to the system that generated the VHD using the native VHD support added in Windows 7 and Windows Server 2008 R2.
The disk signature collision problem and see from a Web search that there’s little clear help for fixing it. These are easy repair steps you can follow if you’ve got a system that won’t boot because of a disk signature collision. I’ll also explain where disk signatures are stored, how Windows uses them, and why a collision makes a Windows installation unbootable.

Disk Signatures

A disk signature is four-byte identifier offset 0x1B8 in a disk’s Master Boot Record, which is written to the first sector of a disk. This screenshot of a disk editor shows that the signature of my development system’s disk is 0xE9EB3AA5 (the value is stored in little-endian format, so the bytes are stored in reverse order):
image
Windows uses disk signatures internally to map objects like volumes to their underlying disks and starting with Windows Vista, Windows uses disk signatures in its Boot Configuration Database (BCD), which is where it stores the information the boot process uses to find boot files and settings. When you look at a BCD’s contents using the built-in Bcdedit utility, you can see the three places that reference the disk signature:
image
The BCD actually has additional references to the disk signature in alternate boot configurations, like the Windows Recovery Environment, resume from hibernate, and the Windows Memory Diagnostic boot, that don’t show up in the basic Bcdedit output. Fixing a collision requires knowing a little about the BCD structure, which is actually a registry hive file that Windows loads under HKEY_LOCAL_MACHINE\BCD00000:
image
Disk signatures show up at offset 0x38 in registry values called Element under keys named 0x11000001 (Windows boot device) and 0x2100001 (OS load device):
image
Here’s the element corresponding to one of the entries seen in the Bcdedit output, where you can see the same disk signature that’s stored in my disk’s MBR:
image

Disk Signature Collisions

Windows requires the signatures to be unique, so when you attach a disk that has a signature equal to one already attached, Windows keeps the disk in “offline” mode and doesn’t read its partition table or mount its volumes. This screenshot shows how the Windows Disk Management administrative utility presents an offline disk that I caused when I attached the VHD Disk2vhd created for my development system to that system:
image
If you right-click on the disk, the utility offers an “Online” command that will cause Windows to analyze the disk’s partition table and mount its volumes:
image
When you chose the Online menu option, Windows will without warning generate a new random disk signature and assign it to the disk by writing it to the MBR. It will then be able to process the MBR and mount the volumes present, but when Windows updates the disk signature, the BCD entries become orphaned, linked with the previous disk signature, not the new one. The boot loader will fail to locate the specified disk and boot files when booting from the disk and give up, reporting the following error:
image

Restoring a  Disk Signature

One way to repair a disk signature corruption is to determine the new disk signature Windows assigned to the disk, load the disk’s BCD hive, and manually edit all the registry values that store the old disk signature. That’s laborious and error-prone, however. In some cases, you can use Bcdedit commands to point the device elements at the new disk signature, but that method doesn’t work on attached VHDs and so is unreliable. Fortunately, there’s an easier way. Instead of updating the BCD, you can give the disk its original disk signature back.
First, you have to determine the original signature, which is where knowing a little about the BCD becomes useful. Attach the disk you want to fix to a running Windows system. It will be online and Windows will assign drive letters to the volumes on the disk, since there’s no disk signature collision. Load the BCD off the disk by launching Regedit, selecting HKEY_LOCAL_MACHINE, and choosing Load Hive from the File menu:
image
Navigate to the disk’s hidden \Boot directory in the file dialog, which resides in the root directory of one of the disk’s volumes, and select the file named “BCD”. If the disk has multiple volumes, find the Boot directory by just entering x:\boot\bcd, replacing the “x:” with each of the volume’s drive letters in turn. When you’ve found the BCD, pick a name for the key into which it loads, select that key, and search for “Windows Boot Manager”. You’ll find a match under a key named 12000004, like this:
image
Select the key named 11000001 under the same Elements parent key and note the four-byte disk signature located at offset 0x38 (remember to reverse the order of the bytes).
With the disk signature in hand, open an administrative command prompt window and run Diskpart, the command-line disk management utility. Enter “select disk 2”, replacing “2” with the disk ID that the Disk Management utility shows for the disk. Now you’re ready for the final step, setting the disk signature to its original value with the command “uniqueid disk id=e9eb3aa5”, substituting the ID with the one you saw in the BCD:
image
When you execute this command, Windows will immediately force the disk and its corresponding volumes offline to avoid a signature collision. Avoid bringing the disk online again or you’ll undo your work. You can now detach the disk and because the disk signature matches the BCD again, Windows installations on the disk will boot successfully. You might find yourself in a situation where you have no choice but to cause a collision and have Windows update a disk signature, but at least now you know how to repair it when you do. 

Thursday, August 31, 2017

Hyper-V File Storage and Permissions

Introduction


Hyper-V takes a different approach to virtual machine file storage than Virtual Server 2005 R2. The default locations are different, the storage approach is different, and the configuration files are different. What does this mean to the seasoned Virtual Server 2005 R2 administrator? It means that you will need to start with a fresh slate and learn how it all works. So let us dive in.

Service SIDS


To allow all of the file storage and access changes, Hyper-V leverages a new type of SID added to Windows Server 2008 called a Service SID to control access to virtual machine files. A virtual machine gets a GUID when created and a corresponding unique Service SID is created for the virtual machine using a combination of the Service SID ""NT VIRTUAL MACHINE" and the VMGUID.

Example: "NT VIRTUAL MACHINE\ C64FB013-6D92-4B9B-B106-690182B00FFA "
If you look at the files of a virtual machine you will see the permissions show an entry for the VMGUID with security permissions assigned, this is the Service SID that is assigned to the virtual machine. Let’s refer to this service SID as the VMSID.

You can use iCACLS to set and view the permissions on files:

icacls "C:\Programdata\Microsoft\Windows\Hyper-V\Virtual Machines\ C64FB013-6D92-4B9B-B106-690182B00FFA.xml" /grant "NT VIRTUAL MACHINE\ C64FB013-6D92-4B9B-B106-690182B00FFA":(F) /L

The figure below shows an example of a VMSID assigned permissions to a virtual hard disk.




Figure 1

If the VMSID is ever removed from the security permissions of a file that makes up the virtual machine, taking the virtual hard disk as an example, the virtual machine cannot power on.


A special group called Virtual Machines is also created to contain all the virtual machines Service SIDS registered on the Hyper-V server.

Default File Locations


Hyper-V consists of the following default file configuration locations for storing virtual machine files:


  • Default virtual hard disk storage location - C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks
  • Default virtual machine storage location - C:\ProgramData\Microsoft\Windows\Hyper-V

These can be modified using the Hyper-V Settings option in the Hyper-V Manager MMC shown below:




Figure 2

When Hyper-V is installed, these folders are created. The Virtual Hard Disks folder is initially empty but the default virtual machine storage location folder contains two subfolders:


  1. Virtual Machines – Stores the virtual machine configuration files
  2. Snapshots – Stores the snapshot files taken of a virtual machine

It is your choice to use the default locations or to specify a different path to store the virtual machine files when creating a virtual machine using the wizard. Your selection results in different actions being taken from a file security perspective.

Let us discuss the two scenarios of the default locations and specifying a different storage location.

Scenario 1: Using the default locations


If you use the default locations during the creation of a virtual machine using the New Virtual Machine wizard, the following happens.



  1. A GUID is generated and assigned to the virtual machine and a VMSID is generated for permissions use. The term VMGUID will be used to refer to the virtual machines GUID in this article.

  2. The virtual hard disk (VHD) for the virtual machine is placed in the C:\Users\Public\Documents\Hyper-V\Virtual Hard Disks directory using the filename you specified.

  3. The VHD security permissions are modified to add the VMSID with Read and Write access to the VHD.

  4. A XML configuration file a filename consisting of the VMGUID with an XML extension is created in the C:\ProgramData\Microsoft\Windows\Hyper-V folder.

  5. A subfolder is created in the C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines directory consisting of the VMGUID as the folder name and the following permissions are assigned. This folder is used to store the saved state files (VSV and BIN).
    a.       Virtual Machines group is assigned the following special permissions that inherit to the folder and sub folders
    i.      List folder / read data
    ii.     Read attributes
    iii.    Read extended attributes
    iv.    Create files /  write data
    v.     Create folders / append data
    vi.    Read
    b.      The VMSID is assigned and given Full control special permissions for the folder only.



Figure 3


If you save the state of the virtual machine, the save state files are stored in the C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\[VMGUID] folder.

This approach has the following advantages:



  1. You do not have to think about where you store files – the defaults are used automatically.

  2. Everything is stored in two directories.

This approach also has the following disadvantages:



  1. The virtual machine VHD, configuration files, save state files and snapshots are all stored on the system drive. This will cause the drive to run out of disk space quickly.

  2. Different virtual machines files are stored together making it harder to track the associations and troubleshoot permissions.

Scenario 2: Using a different storage location


When you use the New Virtual Machine wizard, you have the option of specifying a different location to store the virtual machine files. You enable this option by enabling the checkbox called Store the virtual machine in a different location and then provide the alternate path in the Location text box.




Figure 4

When you use an alternate storage location for your virtual machine files, permissions and files are created differently.



  1. The VMGUID is still generated and assigned to the virtual machine and a VMSID is generated for permissions use.

  2. Instead of folders being created in the default locations, a subfolder is created in the alternate location specified using the name of the virtual machine as the folder name.
    a.       So if you had specified a virtual machine name of VMTEST and an alternate location of D:\VMs, a new folder called D:\VMs\VMTEST would be created.

  3. The VMTEST folder is assigned the following special permissions
    a.       Virtual Machines group is assigned the following special permissions that inherit to the folder and sub folders.
    i.      List folder / read data
    ii.     Read attributes
    iii.    Read extended attributes
    iv.     Create files /  write data
    v.      Create folders / append data
    vi.     Read

  4. Inside the VMTEST folder, the two subfolders Virtual Machines and Snapshots are created and assigned the following permissions: The VMSID is assigned and given Full control special permissions for the folder only

  5. The VHD created by the New Virtual machine Wizard is placed in the D:\VMs\VMTEST folder and assigned the following permissions: The VMSID is given Read and Write permissions

  6. The virtual machine XML configuration file consisting of the VMGUID with an XML extension is created in the D:\VMs\VMTEST\Virtual Machines folder and assigned the following permissions: The VMSID is assigned Full Control permissions

  7. A subfolder with the name VMGUID is created in the D:\VMs\VMTEST\Virtual Machines folder and assigned the following permissions:
    a.       The subfolder inherits the Virtual Machines group permissions
    i.      List folder / read data
    ii.     Read attributes
    iii.    Read extended attributes
    iv.    Create files /  write data
    v.     Create folders / append data
    vi.    Read
    b.      The VMSID is assigned Full control special permissions for the folder only

Hyper-V looks for the configuration file in the default folder C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines, but since the configuration file is stored in the D:\VMs\VMTest\Virtual Machines folder Hyper-V needs a way to reference the configuration file from the default location. This is where symbolic links are employed. If the virtual machine is not stored in the default location, Hyper-V will create a symbolic link to the actual location of the xml configuration file.

Symbolic links are created with the MKLINK command:

mklink "C:\Programdata\Microsoft\Windows\Hyper-V\Virtual Machines\C64FB013-6D92-4B9B-B106-690182B00FFA.xml" "D:\VMTEST\Virtual Machines\C64FB013-6D92-4B9B-B106-690182B00FFA.xml"

For example, if the configuration is stored in D:\VMs\VMTest\Virtual Machines, Hyper-V will create a symbolic link in the C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines folder using the VMGUID as the name of the link. The link will have a pointer to the actual storage location.

If you create a snapshot, then Hyper-V will also create a symbolic link in the C:\ProgramData\Microsoft\Windows\Hyper-V\Snapshots folder to the snapshot version of the configuration xml file in the D:\VMs\VMTest\Snapshots folder.

This approach has the following advantages:



  1. All files for a virtual machine are stored in one folder hierarchy that is rooted with the name of the virtual machine

  2. You can place the virtual machine files on any available drive

  3. Automatic organization of the folders.

This approach has the following disadvantages:



  1. The additional symbolic links that need to be created and managed

  2. You must remember to use the different storage path when you create a new virtual machine

Conclusion


Hyper-V virtual machine file storage and permissions provides two approaches to managing the virtual machine files. Using the default folders makes it easy to create virtual machines, but places the Hyper-V server at constant risk of running out of disk space on the system volume. Using the alternate storage location eliminates the system volume disk issue and provides better organization of the virtual machines storage, but increases the complexity of the storage approach. I personally would always recommend the use of the different storage path

How can I change the Disk ID of a drive?

In some cases if a VM has the same Disk ID as the host, Microsoft VSS will fail and thus backups will not be successful. In such cases you will need to change the Disk ID as below.


To view the Disk ID:
  1. Open Command prompt
  2. Enter the command  DISKPART and hit enter
  3. Enter the command  LIST DISK  and hit enter to list all available disks
  4. Enter  SELECT DISK X (Substitute  "X" for the number of the disk you wish to select) and hit enter
  5. Enter  UNIQUEID DISK  and hit enter
  6. A four byte disk ID will be returned, for example: "e9eb3aa5"

To Change the Disk ID:
    1. Open Command prompt 
    2. Enter the command  DISKPART and hit enter
    3. Enter the command  LIST DISK  and hit enter to list all available disks
    4. Enter  SELECT DISK X (Substitute  "X" for the number of the disk you wish to select)
    5. Enter UNIQUEID DISK ID=a4e19dc0 and hit enter
    6. This will change the Disk ID to "a4e19dc0"
    7. Enter  UNIQUEID DISK  and hit enter to view the new Disk ID


    Note

    Please be advised that changing the Disk ID of a volume hosting the OS is not recommended as it will cause the machine not to boot.

    Hyper-V backups using Backup Exec 2012 fails to create snapshot indicating a disk signature collision.

    Problem

    When performing a backup of Hyper V virtual machines the backup jobs may fail while creating snapshots and indicating that disk signatures for vhd's are colliding.

    Error Message

    \\\Hyper-V?Virtual?Machine\machine-name>". Snapshot technology used: Microsoft Volume Shadow Copy Service (VSS).
    A snapshot could not be created.  The most probable cause is a disk signature collision between two or more VHDs that are assigned to the virtual machine.
     

    0xa0009650 - A snapshot could not be created.  The most probable cause is a disk signature collision between two or more VHDs that are assigned to the virtual machine.
    Also seen error:
    0xa0009651 - A snapshot could not be created.  The most probable cause is a disk signature collision between a VHD that is assigned to the virtual machine and physical disk mounted on the Hyper-v host. 

    Cause

    1. The disk signatures for the virtual machine's (vhd's) are colliding.
    2. The Hyper-V writers on the host machine are failing during the backup process.
    3. If the disk space of the VM doesn't have enough disk space.
    4. Using multi path i/o configuration which creates two identical disks of which one is active and one is passive (seen by IBM but other vendor might have the same configuration).

    Solution

    1. Check and Fix the Disk Signature Conflict. 
           To check Disk Signature or ID use the Windows DISKPART utility commands "detail disk" and "uniqueid".
           Refer to the following image. The "Disk Id" should be unique.
            Contact Microsoft Support for further assistance changing the Disk Signature.

    1. Check the writer status on the Hyper V host machine.
             Reboot the server to check if the writers are stable, re run the backup job.
             If writers continue to fail, Contact Microsoft Technical Support to resolve the issue with the writers.

    1. Clear up additional space on the VM if it's out of space. 

        4.      Install an additional IBM driver specific for multi path i/o configuration (for more detailed information where to find the driver please contact your vendor)

    Tuesday, August 15, 2017

    Warning "Snapshot provider error 0xe00008526. Backup Exec could not locate a VSS software or hardware snapshot provider for this job."

    Problem

    Backup Exec unable to locate snapshot providers installed on the computer.

    Error Message

    V-79-57344-34086 - AOFO: Initalization failure on: "C:". AOFO used: Microsoft VSS. Snapshot provider error 0xe0008526: Backup Exec could not locate a VSS software or hardware snapshot provider for this job. Select a valid VSS snapshot provider for the target computer.


    Application Log :
    Log Name: Application
    Source: VSS
    Event ID: 7001
    Computer:
     
    Description:
    VssAdmin: Unable to create a shadow copy: The volume shadow copy provider is not registered in the system.
    Command-line: 'C:\Windows\system32\vssadmin.exe Create Shadow /AutoRetry=15
    C: Drive properties window shows error about VSS Provider:
    1. Go to Windows Explorer
    2. Right click on C: drive
    3. Click " Shadow Copies" tab.
    4. Check for this error:
    Initialization Failed:
    Error: 0x80042304 : The volume shadow copy provider is not registered in the system.

    Cause

    The snapshot provider selected for the backup job is not installed on the server being backed up.

    Solution

    1.  Check that snapshot providers are installed on the system being backed up.
    • Open a command prompt.
    • Type "vssadmin list providers"
    • If nothing is returned, then there is most likely an issue with VSS that will most likely need assistance from Microsoft to resolve
      • You can also verify that VSS is properly registered by checking the following registry key:
        •  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\VSS\Providers\{b5946137-7b9f-4925-af80-51abd60b20d5}
    • If writers not listed or show they are in an error state, reboot the server and run the command again
     


    2.  Change the backup job to use a specific snapshot provider.
    •  Open the backup job properties.
    •  Navigate to Advance Open File.
    •  Confirm the "Use advanced open file option" is checked.
    •  Select the appropriate snapshot provider - "Microsoft Volume Shadow Copy Service" with "System" selected in the pull down menu is typically best: