“Cloud Numerics” Example: Distributed Numerics on Azure with F#

This post walks through the steps required to use the “Cloud Numerics” distributed numerical and data analytics library from F#. While the “Cloud Numerics” lab focuses on C#, a few additional setup steps enable you to:

• Write F# applications that use the “Cloud Numerics” lab libraries.
• Use the deployment utility to scale-out your F# application to the Windows Azure cloud.

Because most numerical functions in “Cloud Numerics” are static, calling these methods from F# methods feels natural. For example, consider the following F# source code example:

 open Microsoft.Numerics

// Compute the standard deviation for each time series
let stdev = 
    let sum = ArrayMath.Sum(timeSeriesNormalized * timeSeriesNormalized, 0L)
    let sqrt = BasicMath.Sqrt(sum / float(timeSeriesNormalized.Shape.[0]))
    Operations.MatrixMultiply(sqrt.Transpose(), sqrt)

Before You Begin

Before you begin to write “Cloud Numerics” F# code, complete the instructions in the “Cloud Numerics” Getting Started wiki post to:

• Create an Azure account (if do not have one already)
• Install “Cloud Numerics” on your local computer where you develop your F# applications
• Configure and deploy a cluster in Azure
• Submit the sample C# “Cloud Numerics” program to Azure.

Once you are able to build and run the sample C# application provided with the “Cloud Numerics” project template, you are ready to follow the steps in this post to create an F# “Cloud Numerics” application.

If you are interested in testing your “Cloud Numerics” F# project locally, on your local development machine, follow the steps in the Getting Started wiki post to run the sample application locally.

Of course, before you begin, you also need to have F# installed: go to https://fsharp.net and install the Visual F# CTP (April 2011).

Distributed Numerics with F# on Windows Azure

Step 1: Start Visual Studio and create a new C# Microsoft Cloud Numerics Application project

Step 2: In Visual Studio, in the same solution that you created in the previous Step containing the C# “Cloud Numerics” sample application, create a new F# project.

a.   In Solution Explorer, right-click the solution, and choose Add/New Project.

b.   In the Add New Project window, choose Visual F# on the left, then choose F# Application in the main panel, ensure “.NET Framework 4” is selected at the top of the window, type a name for the F# project or use the default name, and click OK.

Step 3: Change the target framework to .NET Framework 4. In Solution Explorer, right-click on the new project and choose Properties. On the Application tab, change the Target Framework to .NET Framework 4, and click OK to the warning that appears.

Step 4: Change the project platform to x64. You need to do this because “Cloud Numerics” is an x64 bit only application and F# is x86 by default.

a.   In Solution Explorer, right click the solution and choose Properties. Select Configuration Properties in the left-hand panel and click Configuration Manager. Then select x64 from the Active solution platform dropdown.

b.   Click the Platform column in the row for your new F# project, and select <New> . Select x64 from the “New Platform” dropdown and click OK. Ensure that the “Create new solution platforms” checkbox is unchecked.

c.   Click the Platform column again for your new F# project, and select Edit. Select x86 from the platform list and click the Remove button, then Yes, and then Close.

d.   In the Solution Properties window, select the Build checkbox in the row for your new project, and click OK.

Step 5: Now you’ll need to add references to the Cloud Numerics assemblies to your F# project.

a.   Right-click your F# project, select References choose Add Reference.

The Add Reference window displays.

b.   Click the Browse tab.

In the “Look In” field, choose the bin directory where Cloud Numerics was installed. By default this is C:\Program Files\Microsoft Numerics\v0.1\Bin. If you aren’t sure where it’s installed, you can right-click one of the Microsoft.Numerics assembly references under the sample C# project, choose Properties, and look at the path in the Properties window. Select all of the DLLs that start with “Microsoft.Numerics”, as well as Microsoft.Solver.Foundation.dll, and then click OK.

Step 6: Right-click the FSharp.Core assembly under References in your F# project and choose Properties. In the properties window choose True in the Copy Local field.

Step 7: That’s it. You can now write F# code that uses the “Cloud Numerics” numerical libraries.

a.   Copy the F# sample program below and copy it into the Program.fs file in your “Cloud Numerics” project.

 // Learn more about F# at https://fsharp.net
// Learn more about Cloud Numerics at https://www.microsoft.com/en-us/sqlazurelabs/labs/numerics.aspx
open System

// Microsoft Numerics runtime namespaces. 
// These contain the local and distributed array types, constructors and array manipulation methods.
open Microsoft.Numerics

// Microsoft Numerics library namespaces.
open Microsoft.Numerics.Statistics
open Microsoft.Numerics.LinearAlgebra
open Microsoft.Numerics.Mathematics

// Workaround for an F# 2.0 bug which doesn't allow linking with x64 assemblies. This wrapper
// allows calling some static methods on the NumericsRuntime class.
type NumericsRuntimeWrapper() =
    static let runtimeType = 
        let myExePath = System.Reflection.Assembly.GetEntryAssembly().Location
        let myExeDir = System.IO.Path.GetDirectoryName(myExePath) 
        let runtimeAssemblyPath = System.IO.Path.Combine(myExeDir, "Microsoft.Numerics.Runtime.dll")
        let runtimeAssembly = System.Reflection.Assembly.LoadFrom(runtimeAssemblyPath)
        runtimeAssembly.GetType("Microsoft.Numerics.NumericsRuntime")

    // Call the Microsoft.Numerics.NumericsRuntime.Initialize() method
    static member Initialize() =
        ignore (runtimeType.GetMethod("Initialize", [||]).Invoke(null, [||]))

    // Call the Microsoft.Numerics.NumericsRuntime.Shutdown() method
    static member Shutdown() =
        ignore (runtimeType.GetMethod("Shutdown", [||]).Invoke(null, [||]))

// Helper function to create a distributed array containing a bunch
// of time series. Each column in the array is a time series. 
// The time series in this example are different only in normalization, which
// means that we'd expect a correlation matrix of all 1's
let CreateTimeSeriesDate nTime nStock =
    Operations.MatrixMultiply(
        new Distributed.NumericDenseArray<float>(ProbabilityDistributions.Uniform(0.1, 0.9, [| nTime; 1L |])),
        new Distributed.NumericDenseArray<float>(ProbabilityDistributions.Uniform(0.1, 0.9, [| 1L; nStock |])))

// Helper function to tile/replication a row vector n times along the row dimension.
// In inputArray has a shape of 1 x m, then the output is of shape n x m.
let Tile (inputArray : Distributed.NumericDenseArray<float>) n =
    let s = inputArray.Shape
    if s.Count = 2 && s.[0] = 1L then
        let onesArray = new Distributed.NumericDenseArray<float>(n, 1L)
        onesArray.Fill(1.0)
        Operations.MatrixMultiply(onesArray, inputArray)
    else
        raise (System.ArgumentException("Tile expects an row vector"))

// Helper function that prints the contents of a distributed array
let Show (inputArray : Distributed.NumericDenseArray<float>) =
    for i in seq { 0L .. (inputArray.Shape.[0] - 1L) } do
        for j in seq { 0L .. (inputArray.Shape.[1] - 1L) } do
            printf "%A " inputArray.[i, j]
        printfn ""


///////////////////////////////////////////////////////////////////////////
// Main code 
//
// Initialize the Microsoft.Numerics runtime; the initialization is required 
// to create and operate on the Microsoft Numerics distribute arrays.
NumericsRuntimeWrapper.Initialize()

let nTime = 5000L
let nStock = 100L

let timeSeries = CreateTimeSeriesDate nTime nStock

let shape = timeSeries.Shape
printfn "Shape of timeSeries = %A by %A" shape.[0] shape.[1]
// Show timeSeries

let timeSeriesMean = Descriptive.Mean(timeSeries, 0u)
let mTiled = Tile timeSeriesMean timeSeries.Shape.[0]
printfn "Shape of mTiled = %A by %A" mTiled.Shape.[0] mTiled.Shape.[1]
// Show mTiled

let timeSeriesNormalized = timeSeries - mTiled
// Show timeSeriesNormalized

let covariance = Operations.MatrixMultiply(timeSeriesNormalized.Transpose(), timeSeriesNormalized) / float(shape.[0])
// Show covariance

// Compute the standard deviation for each time series
let stdev = 
    let sum = ArrayMath.Sum(timeSeriesNormalized * timeSeriesNormalized, 0L)
    let sqrt = BasicMath.Sqrt(sum / float(timeSeriesNormalized.Shape.[0]))
    Operations.MatrixMultiply(sqrt.Transpose(), sqrt)

// Show stdev

let correlation = covariance / stdev
Show correlation

// Shut down the Microsoft Numerics runtime.
// DO NOT REMOVE THIS LINE
NumericsRuntimeWrapper.Shutdown()
printfn "Done ..."

b. Build the project

You can also download the sample program from the “Microsoft Numerics” Connect Site (note that you need to sign up to our Connect program first). See links for Sign-up / Download.

Step 8: Try building and running the F# program locally to verify that it works. Ensure the F# project is set to be the startup project (right-click on the F# project and select Set as Startup Project) and press CRTL+F5 to build and run.

Step 9: To run the F# program on Windows Azure, build and run (CTRL+F5) the AppConfigure project to start the “Cloud Numerics Deployment Utility” you used previously to deploy your C# sample application.

a.   In the Application Code tab, click Browse and select the executable for your F# program.

b.   Click Submit Job to run it on a Windows Azure cluster. Use the procedures described in the Getting Started wiki post to view the status and standard output of your program on Azure.

You can now run distributed F# applications on Azure!

Comments

• Anonymous
  January 21, 2012
  Operations.MatrixMultiply? How about operator overloads? Come on!
• Anonymous
  January 08, 2014
  many links on this article are broken. For example, the like to the sample program from the “Microsoft Numerics” Connect Site is broken. The link to sign up is also broken.