Identity Manager (FIM/MIM): Planning Disaster recovery

  • Article

Introduction

It’s a cliché, better safe than sorry…

But getting FIM up and running again shouldn’t be that hard, at least if you have prepared for it. There are a few components which you should take into account for backing up your FIM configuration.

 

 

↑ Return to Top

Short URL

Bookmark this page as: http://aka.ms/FIMDRP

Download a template document of this procedure at : http://aka.ms/fimdrpdownload or http://aka.ms/mimdrpdownload

General

DRP

  • A DRP plan must provide methods to recover different situations and scenarios
    • its required to include multiple backup methods, as each method is to be used in different situations
    • Do not rely on a single method, eg. 'system wide' backups only, as these have important disadvantages
  • Whatever (combination of) method(s) you choose to backup your configuration, the value of your backup is only guaranteed if you are able to restore it.
  • DRP (Disaster Recovery Planning) is only valid if you have tested it thoroughly, this includes
    • testing backup
    • restoring backups
    • executing bare metal restoration
  • In case of disaster, you don't want to spend time on considering which option is the best
    • Make sure the DRP planning and execution is documented in detail
    • Describe which methods to use in which case
  • DRP should not depend on the skills of the FIM admins only
    • Make sure you can execute the recovery blindly
    • Make sure that server admins with limited FIM skills can execute the DR
  • DRP planning must be reviewed on a regular basis (yearly)

 

SQL

  • Carefully verify the SQL backups, as the the backups in SQL have double functionality
    • Backup data
    • Optimize logs, truncate logs to minimize use of space
  • Make sure to have the proper maintenance plans for your FIM databases
    • Keep fragmentation under control (eg the FIM Sync database has the tendency to fragment...)

 

Note

The core reference for database maintenance is https://ola.hallengren.com/

 

↑ Return to Top

Backup Server System

This applies to any server on which FIM components are installed.

VM System Snapshot/Backup

Lots of environments run their servers on a virtualized machines, most of these VM systems allow to snapshot the guest system.

Advantage

  • Entire machine backup
  • Fast method to take a snapshot of a working machine
  • Quick method to safe the system state just before or just after configuration changes

Disadvantage

  • Snapshots cannot be used for long term storage (as you will run into issues with AD thumbstone when you restore an old machine account)
  • If the snapshot contains corruption of configuration issues, it's very hard to recover the system
  • It's mostly an all-in-one recovery (no single component recovery)

Known issues with Snapshots

 

Windows Backup

Advantage

  • Entire machine backup
  • Allows for bare metal restore
  • Quick method to safe the system state just before or just after configuration changes

Disadvantage

  • This method usually takes some time to restore
  • Snapshots cannot be used for long term storage (as you will run into issues with AD thumbstone when you restore an old machine account)
  • If the snapshot contains corruption of configuration issues, it's very hard to recover the system
  • It's mostly an all-in-one recovery (no single component recovery)

 

↑ Return to Top

Backup FIM Synchronization server

More details at: MIIS/ILM/FIM: How to Backup the Synchronization Engine

 

FIM Sync Database

The FIMSynchronisationService database is hosted on SQL.

Make sure you have a regular backup of your database using the SQL Management tools.

It seems obvious, but also perform a restore once in a while. (Doing is the best practice!)

 

Advantage

  • Contains all identity data and configuration items you configure in the GUI
  • Contains most configuration data (eg files in Extensions directory), but not all files (like service config files, 3rd party client setup & config, source code ...)

Disadvantage

  • Does NOT contain all configuration items
  • Not able to restore partial configuration or single components
  • If database is corrupt, this method is not valid
  • Requires Encryption key to reinstall FIM on same DB, move DB, failover to standby...

Recommendation

  • Do NOT rely on DB backup only, you must combine it with other backup methods (like configuration export)

 

↑ Return to Top

FIM Sync encryption key

When you install FIM, a tool MIISkmu.exe is installed with it, and available in the Programs > Microsoft Identity Integration Server menu.

Recommendation

  • Take a backup (export) of the key regularly (eg before/after an hotfix, upgrade,...)

 

↑ Return to Top

FIM Sync registry

The following registry keys under the path [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\FIMSynchronizationService]:

  • Logging
  • ManagementAgents
  • Parameters
  • Performance

 

↑ Return to Top

Source Code

Safeguard the source files of the extensions you created and compiled.

This allows you to recompile the extensions if needed.

Before you start editing and recompiling the extensions, make a backup of the source.

 

↑ Return to Top

Log files

In some opinions maybe less critical, but certainly useful.

 

↑ Return to Top

Configuration files

Some extensions (like logging, GAL,…) use config files to read data.

Also the task scheduler or MASequencer configuration files, if applicable…

FYI: The FIM Sync Extensions directory is backed up into a Microsoft SQL Server table in the FIM Sync database, and the XML files are restored during the FIM Syncactivate process…

 

↑ Return to Top

Scripts

The scripts, commands and batch file you use to automate FIM Sync operations.

 

↑ Return to Top

MAData (Filebased MAs)

(Cfr. FIM Sync Design and Planning collection)

File-based management agent import and export files that are located in FIM Sync installation path\MAData

 

↑ Return to Top

Software setup files

And it’s always handy to make sure you have the software available you used to prepare your FIM Sync server.

Not only the FIM install files, but also the FIM Sync Resource Took Kit files, Visual Studio, SQL server and other tools you used for configuration, troubleshooting and other operations…

In general, make sure your backups are stored safely (and better at safe distance).

 

↑ Return to Top

FIM Sync server configuration export

Using the FIM Sync GUI (Identity Manager) you can export

  • The entire FIM Sync server configuration (Menu File > Export Server Configuration)
  • The MA configuration (export management agent, one by one)
  •  the MV configuration (Metaverse Designer > Export metaverse)

 

Important notice:

When exporting the FIM Sync configuration,the export does NOT export the content of the connector spaces, nor the metaverse. Also the exensions associated with the MA and MV are not exported.

 

↑ Return to Top

Backup Task scheduler configuration

Independent the tool used to schedule you FIM Sync run profiles, make sure you have a backup of

  • the Run scheduler scripts, configuration files, ...
  • the schedule used (via backup or via proper documentation)

 

↑ Return to Top

Backup 3rd party clients & tools configuration

Theconfiguration of the "non-FIM Sync"tools and clients

For some MA’s you need to configure a client or specific files, or specific network settings. You better make sure you’ve documented these settings or make a backup of this configuration part.

 

Let me explain: for example the configuration of the hosts file, the services file, system files that are customized to support clients like the Oracle or SAP client.

Other client specific configuration (names.orafile, ID files for Notes, …)

 

PCNS

Although it might be difficult to "backup" the PCNS config. Documenting the setup and the procedures used is of important value.

 

↑ Return to Top

Non-FIM components (OS, ...)

Another part of non-FIM Sync backup is the OS, but I’ll suppose it’s sufficiently known how to backup Windows Server…

One way of backing up the system is a full system backup of your server.

On server system level, virtualisation also offers a way of taking a snapshot of the server image.

But still this doesn’t cover the list if the FIM Sync data is distributed over more than one system (eg FIM Sync server with remote SQL).

Remark: be aware of the Microsoft support policy for systems in a virtualized environment (here and here)

 

↑ Return to Top

Backup FIM Service

Partially taken from: FIM Service Backup and Restore

FIM Service export

Sample script

cls If(@(get-pssnapin where-object {$_.Name  "FIMAutomation"} ).count 0) {add-pssnapin FIMAutomation} $targetdir = "C:\Backup\FIM Service Config\" $URI = "http://localhost:5725/ResourceManagementService"   #export schema configuration $export = Export-FIMConfig -uri $URI -schemaConfig $targetfile = $targetdir + "schema.xml" $export | ConvertFrom-FIMResource -file $targetfile   #export policy configuration $export = Export-FIMConfig -uri $URI -policyConfig $targetfile = $targetdir + "policy.xml" $export | ConvertFrom-FIMResource -file $targetfile   #export portal configuration $export = Export-FIMConfig -uri $URI -portalConfig $targetfile = $targetdir + "portal.xml" $export | ConvertFrom-FIMResource -file $targetfile

Database

  1. By default this database is called FIMService.
  2. You do not have to stop the FIM Service when you create this backup.
  3. If you added configuration data that was not included in the Setup process, you can also back up those items.
  4. The FIM Service database is deployed in full recovery model by default. Using the full recovery model makes it possible to back up the log to provide incremental recovery points. See Backup Under the Full Recovery Model (for more information.

See also: How to back up and restore the relevant SQL databases. For more information, see Backup Overview (SQL Server) (http://go.microsoft.com/fwlink/?LinkId=184023).

 

↑ Return to Top

Backup FIM Service Config file

The FIM Service Configuration File, which is located at %programfiles%\Forefront Identity Manager\2010\Service\Microsoft.ResourceManagement.Service.exe.config. 

 

↑ Return to Top

Registry

The following registry keys under the path HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\FIMService:

  • DatabaseServer
  • DatabaseName
  • CertificateThumbpoint
  • DefaultKeySize
  • DefaultTokenLifetimeInMinutes
  • ServiceAccountSid
  • PollExchangeEnabled

 

↑ Return to Top

SQL Agent Jobs

SQL Server Agent Jobs:

  • FIM_DeleteExpiredSystemObjectsJob
  • FIM_MaintainGroupsjob
  • FIM_MaintainSetsJob
  • FIM_TemporalEventsJob

 

For detailed guidance, check TechNet FIM Service Backup and Restore

 

Tools

 

↑ Return to Top

Backup FIM Portal

See: FIM Portal Backup and Restore

 

Tools

 

↑ Return to Top

Backup FIM Reporting

Databases

See: FIM 2010 R2 Reporting Disaster Recovery: http://technet.microsoft.com/en-us/library/jj133846(v=ws.10).aspx

 

SCSM

See: http://technet.microsoft.com/en-us/library/jj133846(v=ws.10).aspx

Best Practices for backing up the SCSM Management Server and Data Warehouse Server Databases

It is highly recommended that you follow the procedures outlined in the System Center Service Manager 2010 SP1 Disaster Recovery Guide. These procedures describe backing up the databases and the encryption key that is created when System Center is deployed. It also provides useful SQL scripts for capturing and preserving SQL Server logon and object level permissions information.

 

↑ Return to Top

Backup BHOLD

<in progress>

 

↑ Return to Top

Backup FIM CM

See: http://technet.microsoft.com/en-us/library/fim_cm_backup_and_restore(v=ws.10).aspx

For a downloadable version of this guide, see the Microsoft Download Center FIM CM 2010 Backup and Restore (http://go.microsoft.com/fwlink/?LinkId=200498).

 

↑ Return to Top

Backup Documentation

Document your setup in detail and keep it up to date.

Documentation should include:

  • Project documentation (business requirement, business analysis & design)
  • Technical architecture/design
  • Technical implementation
  • DTAP migration procedures
  • FIM hotfix & upgrade procedures
  • Disaster recovery planning
  • Operational management procedures of FIM (maintenance tasks, task scheduling, ...)
  • ...

 

↑ Return to Top

Backup External Source & Target data

An important piece of data that FIM Sync is managing, is the data residing in the external data sources.

If for some reason FIM Sync exported wrong of faulty data, it can be quite hard to roll-back the export.

Better make sure the data source administrators got their data backed up.

 

Now you should have secured the most important pieces of the configuration.

Let’s hope we haven’t forgotten anything (otherwise send feedback).

 

Having a backup is one thing, restoring is another.

Depending on how bad your server got damaged, you have some options for recovery.

It depends on the type of disaster which options to choose or to combine.

 

↑ Return to Top

Schedule Backup

Configuration components

Configuration components don't change frequently (or should not). Configuration items only need to backupped before and after the change to allow recovery.

Suggested backup schedule (Manual)

It's a best practice for the administrators to take a backup of the configuraton before and after they change configuration.

 

Data components

Component that change base on the FIM processes require frequent backup to allow recovery of the entire FIM content.

Suggested backup schedule (automated)

It's higly suggested to take regular (automated) backup of the data components. But it's not required to take full backups on a high frequency

Suggestion

  •  regular low frequency (weekly, biweekly, monthly) full database backup
  •  regular high frequency (daily) incremental or differential database backups.

 

↑ Return to Top

Failover FIM Synchronization

Failover - Warm-standby

A warm standby server is a 2nd FIM Sync up and running along your production system.

But an active server means you need a FIM Sync license for that system…

"FIM Sync Help : Activating a standby server Running ILM 2007 FP1" explains this procedure in detail (using FIM Syncactivate.exe to fail over).

When you run the FIM Syncactivate utility, you’ll need the current FIM Sync encryption key.

 

↑ Return to Top

Failover - Cold standby

This is the same procedure, but with the secondary server NOT up and running simultaneously.

For this reason there is no need for an additional FIM Sync license.

The procedure for recovery is the same, except for a bit more delay to boot up the standby server.

 

↑ Return to Top

Server recovery

Full system recovery

Snapshot restore, bare metal restore, system restore… Get the existing system up and running in one step from scratch.

Easy! ;)

But usually a pretty lengthy job, that requires additional tasks...

 

FIM Server reinstallation

The most time consuming way to recover is completely reinstall your FIM Sync server from scratch.

A well documented server setup you created before, will be your guide in that case…

Reinstall OS, restore FIM Sync DB, reinstall FIM Sync with existing encryption key, restore config and log files, restore MA data, … the full path.

 

↑ Return to Top

FIM Server Refresh (repair installation)

Recovering the FIM Sync application and service with a fresh FIM Sync install (database intact, encryption key OK, file backup OK…)

One of the interesting features of FIM Sync is the fact that you can rerun the FIM Sync setup on top of an existing installation without loosing the configuration.

In the case that FIM Sync gets corrupted as application, just rerun the setup and reconnect to the existing database when the wizard asks for the database location.

A fresh install of FIM/ ILM also serves other purposes:

  • upgrading IIFP to ILM, ILM to FIM, FIM 2010 to FIM 2010 R2,
  •  FIM Sync update/upgrade
  • eval version to licensed version
  •  relocating the FIM Sync database (documented here)
  •  …

 

Luckily, in some situations you don’t need a complete recovery or a complete reinstallation.

FIM Sync allows for exporting and importing the server configuration , the management agents and the metaverse, via the MIS GUI.

 

↑ Return to Top

Partial / component recovery

FIM Sync Server

FIM Sync server configuration

This FIM Sync server configuration export only contains configuration data only, not the live data of objects in the CS or metaverse. The actual compiled extensions are not included in the FIM Sync config export.

You can recover the FIM Syncconfiguration quickly for example if you wish to recover with a clean FIM Sync database (and then run the full import and sync run cycles..)

This method also allows for initially uploading an existing (test)configuration in to a blank FIM Sync production server.

But also an existing setup can be restored, for example in case of a misconfiguration.

If you need more detailed info on this method, check the ILM 2007 FP1 Help: Importing and Exporting a Server.

 

MV recovery

The metaverse can be exported and imported separately from the server config.

This is a useful for a emergency recovery of the MV.

More info: FIM 2010 Help: Export Metaverse Schema to File

 

MA Recovery

The MA export can be used for multiple purposes

  •  fresh creation of a new MA by importing the MA (without existing MA available)
  •  duplication of an existing MA (import MA function)
  •  updating the schema of an existing MA (update MA function)

Keep in mind that adding or creating a new MA, also impacts the MV attribute flow precedence…

 

Database recovery

In some situations it is faster to to restore the complete SQL DB with the actual data (configuration AND object data).

Supposing the encryption key is OK, stop the FIM Sync service, restore the database and restart the FIM Sync service.

BBTW, as the re-installation of FIM Sync only takes a short time, it still might be a good idea to reinstall on top of the existing FIM Sync.

Advantage

  • Database backup contains major part of configuration and data
  • Depending the data volume database restore is quicker than reloading the FIM Sync data via run profiles

Disadvantage

  • Aggressive procedure (high impact)
  • When the database is corrupt, configuration must be recovered from the FIM Sync Server configuration. Data must be reloaded via the run profiles.
  • Provisiong must be disabled during initial load and initial sync. A second cycle with provisioning must be executed to get back in operational state.

 

Source code - Extensions recovery

If the source files of the compiled FIM Sync .NET extension have been deleted of corrupted, restore (or copy) a backup of the project files to original location. Eventually you might need to recompile (rebuild) the .NET projects.

 

MA Data

Some MA’s need external configuration (like the ERPMA config tool for SAP).

Other MA’s (file based MA’s) need their data in the MA subfolder…

Restore the configuration and/or data files from backup to their initial location.

 

↑ Return to Top

External data source

Restore the data on the external data source and rerun your FIM Sync operations using preview, logging… before performing the export again.

 

Note

When recovering your FIM Sync server, it might be wise to temporarily turn off the provisioning in the first phase.

Then run the imports and synchronization cycles.

If that runs fine, re-enable provisioning and run the cycles again.

 

↑ Return to Top

FIM Service Recovery

Database restore

  1. Install the FIM Service on a computer in the same Active Directory domain as the FIM Service was previously installed. When installing the FIM Service to a new database the SQL Server Agent Jobs are automatically restored.

  2. Ensure that the FIM Service and SQL Server Agent are not running. If it is not possible to stop the SQL Server Agent, ensure that the FIM stored procedures are not running and disabled. Use the appropriate method for restoring the database, depending on your backup type and the data that you want to recover. See Restore and Recovery Overview (SQL Server) (http://go.microsoft.com/fwlink/?LinkId=184026) for more information.

  3. Restore the SQL Server database. It may be necessary to close open connections on the SQL Server database. For more information, see sp_who (Transact-SQL) (http://go.microsoft.com/fwlink/?linkid=181177).

  4. Restore and validate the .NET Application Configuration file.

  5. Restore the registry key values.

  6. Restore the SQL Server Agent jobs.

  7. Start the FIM Service and SQL Server Agent.

 

↑ Return to Top

Repair install with existing database

  1. Install the FIM Service on a computer in the same Active Directory domain as the FIM Service was previously installed. When installing the FIM Service, connect to the existing database.

  2. Restore and validate the .NET Application Configuration file.

  3. Validate the registry key values. Restore if required.

  4. Validate the SQL Server Agent jobs. Restore if required.

  5. Restart the FIM Service and SQL Server Agent.

 

↑ Return to Top

Planning

Review

Review your disaster recovery planning from time to time.

Keep in mind that a good preparation and the necessary technical precautions are not the only part of disaster recovery.

 

Crisis communication

Plan for crisis communication: who is impacted if the FIM Sync server fails, who should you warn or inform if disaster happens, what information do they need,…

 

When disaster strikes, keep cool and think!

  • Immediately shut down/block/disable automated operations and scripts to avoid more trouble 
  • Check how bad it got
  • Choose a recovery plan
  • If possible, test the recovery.
  • Perform the recovery
  • Communicate !

 

↑ Return to Top

See also

 

↑ Return to Top

References

FIM

 

SQL

 

↑ Return to Top

Downloads

 

↑ Return to Top

Additional Resources

 

Well, never say again you hadn’t a disaster recovery plan!