NETCF: Memory leak... now what??

Subtitles:

- OutOfMemoryException (OOM)

- SqlCeException: Not enough memory to complete this operation

- ...

I'm a Support Engineer at Microsoft CSS ("Customer Service and Support"), and I pretty often handle requests related to NETCF applications crashing with OOM messages, or under a heavy memory pressure and therefore performing bad. I collected some hints and techniques ISV Application Developers may use to prevent or troubleshoot it, so that I can point next developers to them by using a link and not copying\pasting every time the same initial mail... :-) [which frankly btw addresses half of the requests related to leaks...] Happy reading!

GENERIC SUGGESTIONS

It’s quite difficult to get a comprehensive set of actions in order to debug a memory leak, as it’s dependent on the specific scenario you’re running. So, some of the following suggestion could be useful in your case, some others couldn’t. Accordingly to Three Common Causes of Memory Leaks in Managed Applications the 3 most common causes are:

1. unmanaged resources not correctly cleaned up
2. drawing resources not correctly .Dispose-d
3. still referenced managed objects

Regarding (1), if you're developing custom controls then remember to implement Finalize & Dispose, as recommended by Implementing Finalize and Dispose to Clean Up Unmanaged Resources.

Regarding (2), consider the following subtle things you should know: drawing NETCF classes such as Font, Image, Bitmap, Pen, Brush, etc are tiny wrappers around the native resources, which are handled in Windows CE by the GWES (Graphics, Windowing and Event Subsystem). This simply means that in NETCF applications, when you instantiate such classes you must invoke .Dispose on the object, otherwise a leak will be produced. This is the same for MODAL forms (myForm.ShowDialog) as I pointed above: call .Dispose() from the method showing the dialog.

Note that, because of a bug addressed in v3.5, calling Dispose() on an ImageList object causes a memory leak. Therefore, if you’re using an ImageList, do not call Dispose() on it, unless you're application is targeting 3.5. In any case you must clear the list if you’ve finished with it:

 frm.ImageList1.Images.Clear();

// NOT NEEDED frm.ImageList1.Images[0].Dispose();
// NOT NEEDED frm.ImageList1.Images[].Dispose();
// DO NOT frm.ImageList1.Dispose();
// because it leaks in NETCF v2 – addressed in v3.5

The same applies to Toolbars, because they have a .ImageList property. When you close the form and free up the images, don’t call .Dispose on the .ImageList property of the toolbar, but simply

 frm.ToolBar1.ImageList.Images.Clear()

Regarding (3), you can leverage on some monitoring-tools (below) and you should be aware of some caveats, some of which are described in How to: Improve Performance (.NET Framework Developer's Guide). So for example:

• Since String objects are immutable in .NET, every time you modify the value of a String internally a new String object is created. Use StringBuilder instead of concatenating (“+”) String objects, because otherwise too much garbage will be created and the GC will kick too often, thus affecting the whole performances of the application.
• Avoid "NEW" statements in loops
• Limit the number of open SqlCeCommand objects and dispose of them when finished
• ...

TAKE SOME TIME to read through that article...

Still about (3), remember that if you add an event handler to a control that has been dynamically created, before disposing the control you need to explicitly remove that eventhandler - otherwise a reference will be hold for that control and it won’t be effectively garbage-collected.

SPECIFICALLY ABOUT SQLCE\SQL Mobile\SSCE:

Again, some generic suggestions from the experience I had with some cases.

Do not use a shared\static SqlCeConnection object (or a class’ field) and Open\Close it as long as strictly required. While the SqlCeConnection is open, "hidden" native resources are allocated to access the database (query plans, temp data, etc). If opening the connection takes unacceptably long, then you need to find an appropriate balance with caching the connection.

What is the benefit of closing a SqlCeConnection immediately after finished with it, in terms of process' Virtual Memory? The SSCE engine is not a service: when you open the first connection, you initialize the engine. At that time the engine consumes minimal resources. Once you start running queries, it acquires memory from the OS to process those queries, e.g. space for query plans, result set buffers, etc. The SSCE engine does not dispose of the buffers immediately: it would keep them aside for later use so that we don’t need to request the OS again and again. This is due to the internal heap management architecture. Closing the connection results on the engine disposing everything it used so far.

Note that the size of NATIVE resources (in terms of virtual memory) associated to a SqlCeConnection depends mostly on the size of the database, not with the complexness of the query. Native resources associated to the connections the the database may take up to 2MB in size (rarely with more than this, even for really big databases). Finally, being NATIVE resources, they are not shown up in NETCF RPM counters.

Reduce the “Max Buffer Size” (in the connection string, from its default value 640KB to for example 256KB): modifications are flushed to the memory more often, but native resources holding process’ virtual memory is less (virtual memory hold by "hidden" native resources). The balance is:

• in order to reduce NATIVE MEMORY consumption, you want to decrease the "Max Buffer Size"
• in order to increase application's perceived PERFORMANCES, you want to increase the "Max Buffer Size"

Max Buffer Size sets the largest amount of memory, in KB, that SQLCE storage engine can use before it starts flushing changes to disk. Optimal values for Max Buffer Size depend on the data schema and the load. It is important to understand the impact of this parameters for optimal performance. An accurate approach would be to profile the performance and reach the optimal value. A suggested profiling approach would be:

1. Choose the query which is costliest in amount of data that needs to be processed, query execution time spent and joining your bulkiest tables amongst your query set.

2. Time query executions with Connection Strings exploring a range of Max Buffer Size values.

Specifically if you work with multiple databases concurrently, reducing the the Max Buffer Size in all the connections can give a lot of virtual memory back to the process...

Point the temp db to a storage card (from BOL: "If a temporary database location is not specified, then the current database location is used as the location of the temporary database"), to save storage and physical RAM. This is done in the connection string.

If you set Mode=Read Only in the connection string for those databases that you know the code won’t modify, then you must specify a location for the temp db (otherwise, error 25120 - SSCE_M_RODATABASEREQUIRESTEMPPATH)

Close or dispose all SSCE Objects when done (I think this is written everywhere on the web :-)

Reduce usage of memory intensive data structures: datasets in primis... I know that programming with datasets is easy, especially together with DataAdapters, however they were not introduced for a scarce-memory scenario such as Windows Mobile (at least so far). And datasets are a local duplication of the same data you have on the database: thus, in general they can be seen as a data cache that enhances perceived performances. They were introduced to work in disconnected scenarios, such as web pages for example: but when an application accesses to a local SSCE database, then datasets may not be the right ADO.NET approach for the data layer. You may consider SqlCeResultSet instead, or leverage on the performance provided by DataReaders. More on this on a later post I presume...

In any case, if you think that caching much data improves perceived performances of the application, remember that when many hundreds of KB of the Managed Heap are hold by data caches, then the GC will kick more often. And a GC is very expensive, as it needs to "freeze" all application's threads to run. And, as you'll see below, the Managed Heap is usually not larger than 1~3 MB in total for real-world applications.

KB\Doc:

• SqlCeCommand objects are not automatically disposed if you use a SqlCeDataAdapter object 
• Methods to dispose of SQL Server CE or SQL Server 2005 Compact Edition managed objects from memory
• Query Performance Tuning (SQL Server Compact Edition)
• INFO: SQL Server CE Performance Tips and Efficient Memory Handling

NETCF AND VIRTUAL MEMORY:

If you've never looked at how the memory looks like on Windows CE-based platforms, you may start from this 2-part blog post:

Slaying the Virtual Memory Monster [DumpMem explained] - 1st part

Slaying the Virtual Memory Monster [DumpMem explained] - 2nd part

I won't enter on the details described in that blog, so let me write down some probably useful notes:

1- The DLLs related to the NETCF loaded in the application's address space (aka "process slot") are only the ones for the CLR (netcfagl* and mscoree*, in case they’re not XIP DLLs already in ROM, thus meaning they're loaded at slot 1): all the other managed assemblies are loaded into the Large Memory Area (1GB\2GB in the virtual address space). Therefore, the only impact managed modules have in the application’s address space is the JITed code. This is why "unloading assembly", even if it was possible, doesn't give any benefit. This is documented by Device Memory Management (.NET Framework Developer's Guide) and by Mike Zintel’s blog (he was the Program Manager for NETCF Product Group) .Net Compact Framework Advanced Memory Management

2- The code+stack+heap can be allocated ABOVE the “DLL Load Point”: this is NOT a problem. They can oversee that line and can even fill blank spaces between separate loaded DLLs. An OOM may happen if:

* a NEW library is loaded (i.e. a library that wasn’t loaded by any other process so far, thus meaning that the DLL won't find any available space downwards)

* there’s no more room for loading code+stack+heap even in the blank spaces among the loaded DLLs (virtual memory is completely full, considering also fragmentation)

3- COSTs:

only mscoree*_0.dll and netcfagl*_0.dll are loaded into a managed application’s virtual memory address space (in case they’re not XIP DLLs already in ROM), and this is the only fixed cost of a managed application (roughly 750KB). Dynamic costs might be:

* CLR Data structures: ~250KB

* JIT Heap: ~500KB (remember about “Code Pitching”, i.e. the ability to throw away the JIT-ed code in case of scarce memory)

* for each thread: a minimum of 64KB-stack (and a NON-multithreaded managed application has 2 or 3 threads)

* GC Heap: ~2MB, even if a Garbage Collection takes place every 1MB of managed objects allocated (moreover, note that when allocating an object, a VirtualAlloc is invoked for at least a 64KB-segment

* there are other heaps (AppDomain, Process, Short-Term), in any case what RPM calls “Total size of objects” is the total of all of them.

Apart from the cost associated to “pure” managed memory, don’t forget NATIVE resources, for example the ones associated to:

- SSCE engine

- Graphical resources

- P/Invoke or COM Interop

- Fragmentation when loading a DLL:

WM5: 64KB-segments

WM6: 4KB-segments

This means even if you loaded two DLLs that were 10K in size, they would eat up 128K of VM on WM5. In WM6, we use a 4K boundary which makes more efficient use of memory and those two DLLs would eat up 24KB in total.

- Usual values:

* Mscoree2_0.dll + netcfagl2_0.dll (NETCF native code)

            Virtual: 650KB

            Physical: 475KB (assume 75% hot)           

            VM [32MB] or Large Files Area [1GB]? VM

* NETCF Memory Mapped Files (system.dll, mscorlib.dll, etc)        

            Virtual: 3.8MB (worst case)        

            Physical: 1MB (assume 25% hot)  

            VM [32MB] or Large Files Area [1GB]? 1GB space (worst case)

* Other Assemblies, including the managed application’s EXE       

            Virtual: File size x 150% (uncompressed)

            Physical: Assume file size x 25%  

            VM [32MB] or Large Files Area [1GB]? 1GB space (worst case)

* CLR Data Structures    

            Virtual: 250KB  

            Physical: 250KB  

            VM [32MB] or Large Files Area [1GB]? VM

* JIT Heap [this can be “pitched” under low-memory]       

            Virtual: 500KB  

            Physical: 500KB  

            VM [32MB] or Large Files Area [1GB]? VM

* GC Heap Max (1MB, reachable objects’ set (= GC.GetTotalMemory))

[it can reach 2 / 3 MB]

[for each allocated object, at least 64KB --> “segment” = 64KB+x]           

            Virtual: Max(1MB, reachable objects’ set (= GC.GetTotalMemory))           

            Physical: Max(1MB, reachable objects’ set (= GC.GetTotalMemory))           

            VM [32MB] or Large Files Area [1GB]? VM

* STACKs, for each thread (even if app is not multithreaded, at least 2/3 threads)  

            Virtual: At least 64KB    

            Physical: At least 64KB    

            VM [32MB] or Large Files Area [1GB]? VM

* NATIVE DLLs

(such as sqlceme30.dll, which is P/Invoked by v3.1’s System.Data.SqlServerCe. dll) 

            Virtual: File size x 150% (uncompressed)

            Physical: Assume file size x 75%  

            VM [32MB] or Large Files Area [1GB]? VM

TOOLS:

0- DumpMem: https://support.microsoft.com/kb/326164

1- NETCF v2 SP2’s RPM will give you the new opportunity to take multiple pictures of the managed heap (“View GC Heap” button) during application lifetime and prepare some statistics for you to compare different moments.

Finding Managed Memory leaks using the .Net CF Remote Performance Monitor

2- FxCop might be a good tool if RPM don’t help (https://www.gotdotnet.com/team/fxcop): to use with NETCF see https://blog.opennetcf.org/ncowburn/PermaLink,guid,aa3bec5c-94d7-48df-95f4-89c2c87211eb.aspx.

When using FxCop, search for messages whose TypeName="TypesThatOwnDisposableFieldsShouldBeDisposable"

--> Implementing Finalize and Dispose to Clean Up Unmanaged Resources

This is something you need to take care of when developing UserControls that include for example a graphical object.

3- Finally with 3.5 you can use CLR Profiler to identify code that causes memory problems, such as memory leaks and excessive or inefficient garbage collection. However, this is available only for NETCF v3.5 applications. Surely more on this on a later post.

4- Since SSCE can also run on desktops, if the NETCF application doesn’t use special components (barcode scanner, for example) or doesn't P/Invoke Windows CE-specific APIs, it might be interesting to run it against the DESKTOP runtime, purely because there are many profiling tools – starting for example with the ones available within VS2005 Team System:

Analyzing Application Performance (Visual Studio Team System)

I couldn't terminate a post about NETCF + Memory without mentioning the 2 following readings:

- Mike Zintel's .Net Compact Framework Advanced Memory Management  

- Steven Pratschner's An Overview of the .Net Compact Framework Garbage Collector

CONCLUSIONS: If you have ever troubleshoot-ed a memory leak in a NETCF application, you'll probably know how to design its next version... ;-)

-----------------------------------

Appendix:

When you don’t know yet if the problem is with the Virtual memory or with the Physical Memory (RAM), you can monitor both by P/Invoke-ing the GlobalMemoryStatus() API within the lifetime of the application. The code you should insert might be similar to the following:

 namespace MemoryStatus
{
    public class MemoryStatus
   {
        [DllImport("coredll.dll")]
        public static extern void GlobalMemoryStatus(ref MEMORYSTATUS lpBuffer);

        public struct MEMORYSTATUS
        {
            public int dwLength;
            public int dwMemoryLoad;
            public int dwTotalPhys;
            public int dwAvailPhys;
            public int dwTotalPageFile;
            public int dwAvailPageFile;
            public int dwTotalVirtual;
            public int dwAvailVirtual;
        };

        const string CRLF = "\r\n";
        public static string GetStatus()
        {
            MEMORYSTATUS ms = new MEMORYSTATUS();
            ms.dwLength = Marshal.SizeOf(ms);
            GlobalMemoryStatus(ref ms);
            string strAppName = "Memory Status";
            StringBuilder sbMessage = new StringBuilder();
            sbMessage.Append("Memory Load = ");
            sbMessage.Append(ms.dwMemoryLoad.ToString() + "%");
            sbMessage.Append(CRLF);
            sbMessage.Append("Total RAM = ");
            sbMessage.Append(ms.dwTotalPhys.ToString("#,##0"));
            sbMessage.Append(CRLF);
            sbMessage.Append("Avail RAM = ");
            sbMessage.Append(ms.dwAvailPhys.ToString("#,##0"));
            sbMessage.Append(CRLF);
            sbMessage.Append("Total Page = ");
            sbMessage.Append(ms.dwTotalPageFile.ToString("#,##0"));
            sbMessage.Append(CRLF);
            sbMessage.Append("Avail Page = ");
            sbMessage.Append(ms.dwAvailPageFile.ToString("#,##0"));
            sbMessage.Append(CRLF);
            sbMessage.Append("Total Virt = ");
            sbMessage.Append(ms.dwTotalVirtual.ToString("#,##0"));
            sbMessage.Append(CRLF);
            sbMessage.Append("Avail Virt = ");
            sbMessage.Append(ms.dwAvailVirtual.ToString("#,##0"));
            return sbMessage.ToString();
        }
    }
}

Comments

• Anonymous
  April 17, 2008

• GPS Signal Quality - Error Reduction - Avoid problems with localizations - Data Layer: do NOT use XML

• Anonymous
  November 23, 2008
  Really many mobile developers have often various issues related to memory. This is obviously mainly related

• Anonymous
  November 24, 2008
  This is way out of the scope of what I normally troubleshoot, but I have gotten some questions around

• Anonymous
  December 02, 2008
  Nothing really unique in the blogosphere here, however worth mentioning anyway as it gives me the opportunity

• Anonymous
  January 06, 2009
  If you know my interest on troubleshooting memory issues on Windows Mobile applications (this was the

• Anonymous
  March 24, 2009
  Abhinaba’s blog What’s new in Windows Mobile 6.1 and 6.5 around memory Dynamic assembly unloading Again

• Anonymous
  May 19, 2009
  http://blogs.msdn.com/sqlservercompact/archive/2009/05/05/sql-server-compact-garbage-collection-whys-and-hows.aspx is really interesting as it explains some internals of SSCE implementation about memory. check it out!

• Anonymous
  May 19, 2009
  I’m losing my expertise on this!! After blogging about it to spread out the usual suggestions I was used

• Anonymous
  November 28, 2010
  Nice post i added it to my bookmarks <a href="http://www.party-dresses.biz">party dresses</a>

• Anonymous
  February 04, 2011
  Thank you Raffaele, thank you very much!!!!!!!!!!!!!!! I had a big problem with memory leak, and it was a miracle when I googled this page and your "Avoid "NEW" statements in loops" fixed very bad situation!!!!!!!! Barcode reader scanner wrapper had declared new buffer into the method scope, so each barcode reading decreased the memory by 1 kilobyte :( When I moved the buffer into the class scope (to avoid new buffer creation into this method - THE PROBLEM HAD BEEN SOLVED!!!!!!!!!!!!!) THANK YOU AGAIN!!!!!!!!!!!!!!!!!!