cfapi CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS calls back repeatedly

Chris Wheeler 1

Bringing this here from Stack...

I'm working with the cfapi and are attempting to extend the sample project available here https://github.com/Microsoft/Windows-classic-samples/tree/master/Samples/CloudMirror

Specifically we are looking to implement the CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS to provide directory content on-demand. Our callbacks are registered like so:-

CF_CALLBACK_REGISTRATION FakeCloudProvider::s_MirrorCallbackTable[] =
{
    { CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS, FakeCloudProvider::OnFetchPlaceholders },
    { CF_CALLBACK_TYPE_FETCH_DATA, FakeCloudProvider::OnFetchData },
    { CF_CALLBACK_TYPE_CANCEL_FETCH_DATA, FakeCloudProvider::OnCancelFetchData },
    CF_CALLBACK_REGISTRATION_END
};

The OnFetchPlaceholders method calls CfExecute() to respond to the callback with an array of placeholders for the associated files in the server directory.

CF_OPERATION_INFO opInfo = { 0 };
CF_OPERATION_PARAMETERS opParams = { 0 };

opInfo.StructSize = sizeof(opInfo);
opInfo.Type = CF_OPERATION_TYPE_TRANSFER_PLACEHOLDERS;
opInfo.ConnectionKey = callbackInfo->ConnectionKey;
opInfo.TransferKey = callbackInfo->TransferKey;

opParams.ParamSize = sizeof(opParams);
opParams.TransferPlaceholders.CompletionStatus = status;
opParams.TransferPlaceholders.Flags = CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_NONE;
opParams.TransferPlaceholders.PlaceholderArray = placeholders;
opParams.TransferPlaceholders.PlaceholderCount = placeholdersCount;
opParams.TransferPlaceholders.PlaceholderTotalCount.QuadPart = placeholdersCount;

winrt::check_hresult(CfExecute(&opInfo, &opParams));

The issue that I am seeing is that the callback is invoked over and over again despite the fact that we have provided all of the placeholders that are currently available. The invocation does eventually stop but the number of times that it's called seems random.

I've tried setting the flag CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION in the callback and when I do this we are only called once but once set we are never asked again for the placeholders even if we have new ones to provide.

I've tried doing the following when we have no new placeholders to provide:-

opParams.TransferPlaceholders.PlaceholderCount = 0;
opParams.TransferPlaceholders.PlaceholderTotalCount.QuadPart = 0;
opParams.TransferPlaceholders.PlaceholderArray = NULL;

...and this works to a point but we're still called back again and again. I suspect that there must be some other piece of the puzzle here but unfortunately the documentation isn't very detailed on this and the sample project doesn't implement this callback.

Rita Han - MSFT 2,161 Reputation points

2020-08-17T01:41:01.86+00:00

Thank you for bringing this to our attention. I can reproduce this issue. However, there seems no document about detailed information of how to implement CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS callback. I'll consult the related engineer for helping on this issue and keep you update here.
Fei Xue - MSFT 1,111 Reputation points

2020-08-28T01:29:29.12+00:00

@Chris Wheeler , Thanks for posting on Microsoft Q&A. Since the issue is complex, I suggest you contacting Microsoft support to raise an incident so that our engineer could work closely with him to identify the root cause and resolve this issue as soon as possible.

If the support engineer determines that the issue is the result of a bug the service request will be a no-charge case and you won't be charged. Please visit the below link to create an ticket via 'Contact us':
https://developer.microsoft.com/en-us/windows/support/

1 answer

WebDAV 21 Reputation points

2020-08-21T04:35:54.873+00:00

I experience the same behavior. My opinion is that the behavior, when the callback is called only 1 time with flags = CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION is correct.

My guess is that you are expected to populate the folder only during first access. After that, you will add, delete, and update placeholders via other Cloud Files API functions, not inside a callback. My dev team was able to build a working prototype using this approach in .NET+Win32 API.
Please sign in to rate this answer.
Chris Wheeler 1 Reputation point

2020-08-24T12:21:54.7+00:00

Thanks for the input, that was what I was assuming but the docs don't make that clear.

I guess in that case the question becomes whether there a way to revoke the CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION status to be able to query the server again in case things get out of sync?

WebDAV 21 Reputation points

2020-08-25T03:31:49.873+00:00

I think you do not need to revoke the CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION. The folder will be initially populated only one time in the callback.

Then, you do one of the following (or in many cases both):

Send notifications from the server to clients (using web sockets or any other technology) about files/folders are being updated, created and deleted. You update and create placeholders directly in your file system, using CF API.

Create service on the client to pull the server periodically and synchronize the client and server. This will keep the client in sync, in case the client has missed changes via notifications (machine restart, client failure, etc.)

As you can see, in this case, re-enabling the on-demand population is not needed. This will keep the client in sync with the server after the first population via CF API callback.

WebDAV 21 Reputation points

2020-08-25T03:32:35.473+00:00

Moreover, we found that if you do not disable the on-demand population, the CF API becomes very slow, calling too many callbacks, the file system becoming unusable. Collapsing/expanding of folders in Windows File Manager becomes broken.

We are still working on this, maybe we will discover some more ideas later. But for now, we see that initial on-demand population + later notifications/pooling provides the best reliability and performance.

Chris Wheeler 1 Reputation point

2020-08-25T08:13:40.787+00:00

Great and thanks again for your input.

My concern around point 2 that you made above is that we'd potentially need to keep quite a substantial list of folders to poll for updates. That would be fine if users only visit a few dozen folders - but a several hundred folder remote file system would quickly become unwieldy.

In theory the CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS would allow us to only update a folder when the user actually browsed to that folder - which would seem like a much more efficient way of doing things. Unfortunately, unless I'm missing something obvious it does appear that this isn't really the way that the API has been designed. (Unless there is a way to revoke that CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION flag of course.)

Fu Wang 26 Reputation points

2020-08-26T11:42:39.65+00:00

I agree with WebDAV-8207, the folder only need to be populated once during the first access, then you will need use other methods to make it's contents in sync(e.g. pull the server periodically).

In my experience with the API, if you do not disable the on-demand population, every access to this folder by the explorer.exe or SearchProtocolHost.exe(Windows Index Service) will trigger a new FetchPlaceholders call back, which will cause the folder got populated over and over, this not only impact the system performance, will also increase the loads of your server.

Fu Wang 26 Reputation points

2020-08-26T11:49:45.763+00:00

In your scenario, there actually is an API to revoke the CF_OPERATION_TRANSFER_PLACEHOLDERS_FLAG_DISABLE_ON_DEMAND_POPULATION, just call the CfUpdatePlacehoder API and pass in the CF_UPDATE_FLAG_ENABLE_ON_DEMAND_POPULATION flag, this will re-enable the on-demand population, and any future access to the folder will trigger the FETCH_PLACEHOLDERS callback again.

WebDAV 106 Reputation points

2020-09-09T03:03:42.99+00:00

Regarding server pooling. To avoid pooling every folder you can do one of the following (or typically a combination of both):

Use web sockets (or similar technology) to push updates from server to clients. So clients will immediately receive updates when a file/folder is added, modified, deleted or moved.

Implement synchronization algorithm based on Sync-ID. Used in CalDAV, WebDAV and other systems. In short: when you create, update, delete or move a file or folder you need to associate a number with it (Sync-ID), incrementing during each operation. When needed (after client machine restart, app failure, etc), send a request to the server providing the largest Sync-ID that the client has. The server returns all files/folders modified, added, deleted or moved since provided Sync-ID.

WebDAV 106 Reputation points

2020-09-15T05:31:56.617+00:00

We were able to build a sample in .NET with bi-directional synchronization for file system storage: https://github.com/ITHit/UserFileSystemSamples/tree/master/VirtualFileSystem. It is using an initial on-demand population + later notifications + pooling. From what we see it is possible to create a Sync-ID-based synchronization algorithm that will avoid pulling every folder. It still has several hiccups and no conflict resolution mechanism, but most of the documents editing works and syncs bidirectional, including it works with MS Office documents editing.
Sign in to comment

Share via

cfapi CF_CALLBACK_TYPE_FETCH_PLACEHOLDERS calls back repeatedly

1 answer