Read-Only Replication in R2

  • [アーティクル]

Ned here again. I’ve been involved with DFSR since its late beta days in 2005. Through good luck and a previous interest in FRS, I decided to take on DFSR as a focus area; when the support cases started rolling in I dealt with most of the issues. My expertise in DFSR is pretty much happenstance. :-)

Since that first week, I have had hundreds of customers ask me about configuring DFSR “one-way” replication. I’ve also had many customers try to invent their own read-only configurations for DFSR, with results ranging from pretty decent – using share permissions - to “OMG cleanse it with fire!!!” topology hacking.

Thankfully, Windows Server 2008 R2 takes care of this now in a more sophisticated way. I am going to discuss the architecture of read-only replication in Win 2008 R2, cover some best practices, and describe a few troubleshooting techniques. If you are looking for introductory information about why you would use read-only DFSR or how to configure it step by step, I highly recommend the DFSR development teams blog posts and help file:

On with the show.

Architecture

Read-only (RO) replication refers to the concept of “one-way” replication, where a downstream RO replicated folder is incapable of sending files outbound. It also prevents the creation, modification, or deletion of files in the RO content set locally. RO replicated folders sets can exist on a server that also contains other Read-Write (RW) replicated folders - the boundary of RW or RO is the replicated folder itself on a given server. This new system is ideal for static data that end users are never supposed to modify, such as application installation points, corporate procedural files, centralized backup locations etc.

RO replication relies on the new mini filter file system driver DFSRRO.SYS. This IO-blocking driver is at an altitude below most common filters, meaning that other drivers will have to abide by DFSRRO’s decisions – including anti-virus software. DFSRRO.SYS is implemented at the “FSFilter Content Screener” level:

image

Because the filter driver blocks write IOs, any applications or even many kernel-mode drivers that attempt to save data in the RO folder will receive "Access is denied" (0x5 ERROR_ACCESS_DENIED). This also means that if an application opens files with a WRITE disposition unnecessarily - even if the application only intends to read data – DFSR will return access denied. You can read more about CreateFile dispositions here and file system drivers here.

A Replication Group containing an RO replicated folder must still contain both inbound and outbound connections. Since a replication group could potentially maintain a mix of RO/RW, and because an RO can change to an RW dynamically, both connections must exist at all times. RO replicated folders can only directly replicate from RW replicated folders. An RO replicated folder cannot source from another RO replicated folder. Also, an RW cannot transitively replicate through an RO.

So these configurations are good:

image                     image                 image

And these configurations are bad:

 

image                                 image

Note: my arrows indicate direction of replication, not a single DFSR connection object! Don’t make me come over there.

An RW replicated folder can be converted to an RO replicated folder (and back) “on the fly”. Converting will cause a non-authoritative sync to occur on the replicated folder for the server being altered. This is done to ensure that the contents are up to date with the partner before data can flow inbound/outbound. This gives you DFSR event log messages 4620 or 4622:

Log Name:      DFS Replication
Source:        DFSR
Date:          2/22/2010 4:27:29 PM
Event ID:      4620
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      2008r2-f-01.contoso.com
Description:
The DFS Replication service has detected that ken, which used to be a read-write replicated folder has now been configured to be a read-only replicated folder. The DFS Replication service will not allow any changes to be made to the contents of this replicated folder. Any updates occurring on other read-write replicated folders will be replicated in and applied to the contents of this read-only replicated folder.
Additional Information:
Replicated Folder Name: ken
Replicated Folder Root: c:\ken
Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
Replication Group Name: clu2
Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

==========================

Log Name:      DFS Replication
Source:        DFSR
Date:          2/22/2010 4:31:31 PM
Event ID:      4622
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      2008r2-f-01.contoso.com
Description:
The DFS Replication service has detected that ken, which used to be a read-only replicated folder has now been configured to be a read-write replicated folder. The DFS Replication service will now allow any changes to be made to the contents of this replicated folder. Any updates occurring on this read-write replicated folder will be replicated out and applied to the contents of other replicated folders on other members.
Additional Information:
Replicated Folder Name: ken
Replicated Folder Root: c:\ken
Replicated Folder ID: 171E6A7E-6182-496D-8277-1797FF18C05C
Replication Group Name: clu2
Replication Group ID: 8A645529-FE74-4430-9ECD-D1BDC0BA800A
Member ID: 0EAD46B4-A442-48EA-97F6-109714968E40

These events would be followed by 4102 and 4104. There is also a new event that will write when someone tries to restore over a read-only RF:

Log Name: DFS Replication
Source: DFSR
Date: 11/21/2008 11:29:14 PM
Event ID: 1112
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: WIN7-6946-02.southridgevideo.com
Description:
The DFS Replication service failed a restore request. This could happen if an
attempt was made to restore the contents of a read-only replicated folder
authoritatively. Read-only replicated folders should only be restored
non-authoritatively. Authoritative restores should be performed only on read-write
replicated folders.

Additional Information:
Replicated Folder Name: foo
Replicated Folder ID: 02436A8B-D620-4D83-980A-657E2603FBA0

Finally, there are also new debug entries for RO, under the module ID’s BFLT, FREP, RFLT ad well as updated in VLMG and CCTX. If that makes no sense, review this insomnia cure.

Best Practices and Answers to Common Questions

Note: some of these will be repeats from this and other blog posts. I do that because no matter how many times I talk about certain best practices, people continue to ignore me. :-)

1. Before deploying read-only folders you must test! Your antivirus software, your backup software, and especially your line of business applications that open files from the replicated folder may all react badly to RO. There are no known MS applications that have any problems.

2. Converting a replicated folder from RW to RO (or the reverse) will cause a non-authoritative sync to occur on the replicated folder. Plan accordingly.

3. DFSR read-only is not a panacea! If users can access files on the read-write folders, RO will not prevent anything. If you are trying to stop administrators from changing data, get better administrators.

4. Tell your users! Unless you want to have lots of help desk tickets about “I get access denied errors”, make sure your users understand that you are marking previously writable data as read-only. Not that they will read the email where you told them this, but at least you tried.

5. You cannot authoritatively restore data to an RO folder. Duh. Getting backups from there is fine and highly recommended.

6. Don’t bother trying to circumvent RO. Even when running as SYSTEM, you will not be able to write files into an RO-protected folder. If you don’t want RO protection, don’t run RO. Make your changes on RW folders.

7. RO can replicate inbound from Windows Server 2003 R2 and Windows 2008. You will need to have extended the AD schema to at least Windows Server 2008 though.

8. An RO replicated folder can have more than one RW partner replicating inbound, as long as those RW servers have more than just the transitive RO connection.

Troubleshooting

There aren’t many new steps to troubleshooting read-only replication compared to classic read-write – make sure you have two connections between every server, in both directions, just like RW. Make sure you have the latest hotfixes. Review everything I described above about topology – most issues in RO have been admin-generated so far. :-)

The one thing you can add to your test environment troubleshooting is FLTMC.EXE. This tool allows you to dynamically load and unload the DFSRRO.SYS filter driver for testing purposes, using the LOAD and UNLOAD commands. This way if you are testing an application that mysteriously fails when talking to a read-only folder, you have a way to temporarily allow writing to the folder and confirm the application is trying to write data without the need to completely remove RO. There is a lazy sync worker thread that removes changes and replicates down changes from the RW server after the driver is reloaded, so that the file system does not get irrevocably out of sync. Remember to load the driver back up when done testing!

Wrap Up

It’s one more good reason to ditch FRS and deploy DFSR.

And a funny story about that previous interest in FRS I mentioned: most people in MS support really dislike FRS for all of its inadequacies, but I owe it a debt of gratitude. When I was in my hiring interview years ago, one of the questions I got asked was “Can you describe in as much detail as possible how FRS processes files?” The panel thought that was a real stumper. So I drew this on the whiteboard from memory and explained it:

image

And I got the job. So thanks FRS, I owe you this gig.

Until next time.

Ned “r” Pyle

Comments

  • Anonymous
    March 10, 2010
    The comment has been removed

  • Anonymous
    March 10, 2010
    My first IT job I got because I was a smoker and my prospective boss was too. The first big project I got because I was constantly hammering the architect with problems in his design and he said "fine, come join my team and do this, I'm sick of getting beat up". It's a weird little world.

  • Anonymous
    March 16, 2010
    thanks for this post ;-) Robert