Learn F# for the masses


Anyone who reads my blog will have observed that I’ve not posted anything for several months now. In addition to my moving country and trying to build a company in 2016, I’ve also been writing a book.

I’m delighted to now share that Learn F# is now available on Manning’s MEAP program – hopefully the book will be content complete within the next couple of months.

The book is designed specifically for C# and VB .NET developers who are already familiar with Visual Studio and the .NET ecosystem to get up to speed with F#. Half the book focuses on core language features, whilst the second half looks at practical use cases of F# in areas such as web programming, data access and interoperability.

The book doesn’t focus on theoretical aspects of functional programming – so no discussion of monads or category theory here – but rather attempts to explain to the reader the core fundamentals of functional programming (at least, in my opinion) and apply them in a practical sense. In this sense I think that the book doesn’t overlap too much with many of the F# books out there – it doesn’t give you a hardcore understanding of the mathematical fundamentals of FP, and relates many concepts to those that the reader will already be familiar with in C# etc.. – but it will give you confidence to use, explore and learn more about F# alongside what you already know.

I’d like to think it will appeal to those developers that are already on the .NET platform and want to see how they can utilise and benefit from F# within their “day-to-day” without having to throw away everything they’ve learned so far. So you’ll see how to perform data access more easily without resorting to Entity Framework, how to perform error handling in F# in a more sane manner, parsing data files, and creating web APIs, whilst using FP & F#-specific language features directly in solving those problems.

I’ll blog about my experiences of writing a book when it’s done – for now, I hope that this book is well received and a useful addition to the excellent learning materials already available in the F# world.

Hosting Suave in the Azure App Service

logo

In my previous post, I spoke about the deployment aspects of the Azure App Service, and how in conjunction with Kudu, F# and FAKE, we can utilise a SCM-based solution for deployment that can essentially follow the exact same build process as is performed locally.

In this post I want to discuss the process behind hosting Suave (or indeed any application that listens to HTTP traffic) in the Azure App Service.

What is Azure App Service?

The Azure App Service is a broad service that contains multiple “sub services”: –

  • Web apps
  • Logic apps
  • Mobile apps
  • API apps

We’re interested in Web Apps in the post. If you’ve used Azure before, and had an ASP .NET web application, it was an easy decision to pick the Azure App Service as the service to host your app. What’s not so well known – and I admit that until I spent some time looking into it I just assumed that it wasn’t possible – is that you can use the service to host any executable application within the IIS process, and have the app service simply act as a passthru, routing HTTP requests through to your application and back again.

Why would you want to do this though? Why not just use a Cloud Service or raw VM? I would direct you to my previous post on Azure services but in a nutshell, the app service provides a higher-level service than either of the others – think of it as IIS-as-a-service – with support for: –

  • SCM-based deployment e.g. GitHub, BitBucket etc.
  • Metrics and alerting services
  • Scale up application size on demand
  • Automatic load balancing with scale out on demand or based on metrics such as CPU
  • Turn-key authentication features
  • Slots – deploy different versions of code to test before flipping to live
  • A/B testing support
  • Web jobs

So, basically a lot of things that you’d need to manage yourself in a production web application all come out of the box. And the good thing is that you can get all of these features with e.g. a Suave web application as well – it’s not just for ASP .NET.

Creating an Azure Web App

To create a web app, simply log into the Azure portal, select New from the left hand side menu, choose Web and Mobile and finally Web App. Fill in the details, confirm, and you’ll end up with an empty website that you can browse to and receive a stock Azure Website page. So now we have an empty application, how do we put our code into the app?

Binding Suave to an Azure Web App

The first thing you’ll need to do is get your code into the Azure web app that you’ve just created. There are a number of ways that you can achieve this.

Firstly, you can use SCM-based deployment, which I detailed in my previous post. But a quicker way to go for a “one-off” deployment is simply to FTP in and copy the files across. To do this, in the Portal, navigate to your empty web application and hit the Get Publish Settings option from the menu bar of the web app pane. This will give you an xml file, inside of which are the FTP address and credentials. You can then FTP in and simply copy up your application into the wwwroot folder.

Note that you can also use HTTP as well as MS Web Deploy (either through the command line or Visual Studio), although I suspect that that would require making your Suave application appear as a web app through custom project GUIDs.

Configuring the Azure App Service

A standard Suave application (at least, all the examples I’ve seen) run as either .fsx scripts or executables. Indeed there are already a few examples of running Suave within an Azure website – and I should give credit to Scott Hanselman and Steffen Forkmann for getting the basic Suave example up and running here. The majority of what I’ve done from here is based on that work – the difference is that rather than hosting FAKE itself which runs a simple Suave application within an .fsx file, I’m not using FAKE as a host at all (although I do use FAKE for the build stage as per my previous blog post). Instead, all I’m doing is hosting an .NET executable that launches Suave.

So how do we do it? Bear in mind that the Azure App Service is essentially just IIS as a managed service. It’s actually rather straightforward once you know what’s required, which is simply to instruct IIS to redirect all traffic to our Suave application. How do we do that?

Adding a custom web.config

In addition to your standard executable app.config which contains all the config and binding redirects etc., you need a slimline web.config which is used by IIS to startup and then redirect traffic to your Suave application. It doesn’t contain much: –

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <remove name="httpplatformhandler" />
      <add name="httpplatformhandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
    </handlers>
    <httpPlatform stdoutLogEnabled="true" stdoutLogFile="suave.log" startupTimeLimit="20" processPath="%HOME%\site\wwwroot\SuaveHost.exe" arguments="%HTTP_PLATFORM_PORT%"/>
  </system.webServer>
</configuration>

The key parts to take away from this are: –

  1. Remove the standard HTTP Platform Handler and replace with another one.
  2. Specify to use SuaveHost.exe (my application name) as the application in the processPath attribute.
  3. Pass in an argument to the application – the internal port that traffic will come in on using the %HTTP_PLATFORM_PORT% variable. You can pass in multiple arguments here, just ensure they are space separated.

Handling arguments in Suave

Now that we have hooked our application into IIS, in Suave we simply run our application and take in the port number as an argument: –

[<EntryPoint>]
let main [| port |] =
    let config =
        { defaultConfig with
              bindings = [ HttpBinding.mk HTTP IPAddress.Loopback (uint16 port) ]
              listenTimeout = TimeSpan.FromMilliseconds 3000. }
    // rest of application

That’s it!

Managing Suave through Azure App Service

Now that you have your application up and running in Azure, what can you do? Well, you can log into the Azure portal and get some metrics of your website immediately as a configurable dashboard: –

Untitled.png

The charts are configurable, so you can select which metrics you’d like to show e.g. which HTTP codes etc. over what time period. We can also look at the process explorer – and sure enough, there’s our SuaveHost.exe application: –

Suave2

And we can even drill into the process: –

suave3

Conclusion

Of course, what I’ve shown you above is just scratching the surface of what you can do with Azure. It’s possible to do all the other things I mentioned at the start of this post, such as scale up the size of the web server, scale out to multiple instances, create multiple deployment slots etc., all from within the portal. Or perhaps you’d like to set up custom alerts based on any of the dashboard metrics over a certain period of time e.g. “> 50 HTTP 404s in the last 5 minutes” and send an email / hit an HTTP endpoint etc.? No problem – that’s supported out of the box.

It’s actually all incredibly easy and really allows you to simply focus on the work of developing an application and let Azure manage the infrastructural challenges. In fact, I can’t imagine self-hosting (or self-managing) any web-facing application when you have a service like this available. Hopefully I’ve shown though, that’s it’s not just the stock ASP .NET website that can be run through Azure web apps – we can host Suave as well without much effort at all.

The source code that was used for this post is available here.

 

 

Deploying Azure web applications with FAKE


The Azure App Service is a great service that makes hosting web-facing applications extremely easy, with support for many value adds out of the box e.g. scale out, A/B testing and authentication are all included. I’ve recently been looking at how you can use this service within the context of some F# frameworks and libraries e.g. Suave. I’ll blog about the Suave side of things in another post – there’s a lot to it – but one of the other parts I wanted to mention was that FAKE now has support for Kudu, the Azure App Service SCM deployment engine.

What is Kudu?

One of the features that App Service offers is a multitude of deployment options, including FTP, HTTP and also source control web hooks. The latter supports a number of providers, including GitHub, BitBucket, VSTS and even a locally hosted git repository. The App Service listens to push events on a specific branch, downloads the source code onto the web server (into a sandboxed location) and then copies it into the website proper. The latter stage – the copy – is the one of most interest. Essentially, the app service runs a batch file which can do whatever is needed to do a build and deploy. For a .NET application, this typically includes: –

  1. Perform an MSBuild of the application.
  2. Copy the outputs to the “staging” directory on the web site.
  3. Run KuduSync to deploy to the actually web application folder.

KuduSync itself essentially does a few things: –

  • Does a diff of the current files to deploy from the previous deployment.
  • Removes any obsolete files from the app.
  • Copies over any new / updated files from the staging directory to the app.
  • Makes a list of the deployed files for comparison the next time it runs.

The Azure CLI extensions come with some commands to “pre-generate” a batch file for specific, common use cases e.g. ASP .NET application etc.., but you’ll often need to do something more than just that – and this means getting your hands dirty with a kudu script.

A Sample Kudu script

So here’s a standard Kudu build script (which I’ve actually minimised as much as possible) which deploys some raw web assets (HTML, JS etc.), builds a .NET application and deploys a web job : –

:: Restore NuGet packages
.paket\paket.bootstrapper.exe
.paket\paket.exe restore

:: Copy static site content over - note the "excludes.txt" which contains file types to ignore....
xcopy src\webhost "%DEPLOYMENT_TEMP%\" /Y /E /Q /EXCLUDE:excludes.txt
IF !ERRORLEVEL! NEQ 0 goto error

:: Deploy an F# script as a continuously running Web Job
xcopy src\Sample.fsx "%DEPLOYMENT_TEMP%\app_data\jobs\continuous\Sample\" /Y
IF !ERRORLEVEL! NEQ 0 goto error

:: Build to the temporary path
cd "%DEPLOYMENT_SOURCE%"
call :ExecuteCmd "%MSBUILD_PATH%" /m /t:Build /p:Configuration=Release;OutputPath="%DEPLOYMENT_TEMP%";UseSharedCompilation=false %SCM_BUILD_ARGS% /v:m
IF !ERRORLEVEL! NEQ 0 goto error
cd ..

:: KuduSync
call :ExecuteCmd "%KUDU_SYNC_CMD%" -v 50 -f "%DEPLOYMENT_TEMP%" -t "%DEPLOYMENT_TARGET%" -n "%NEXT_MANIFEST_PATH%" -p "%PREVIOUS_MANIFEST_PATH%" -i ".git;.hg;.deployment;deploy.cmd"
IF !ERRORLEVEL! NEQ 0 goto error

There’s actually a whole host of things here: –

  1. First I’m pulling down my Nuget dependencies with Paket (this could just as easily be Nuget.exe) before moving onto our main build.
  2. xcopy across the website assets. There are some files I don’t want, so I also have to add an “excludes.txt” file which contains file types I don’t want as an argument to xcopy. This was a real pain to figure out and to use the correct arguments to xcopy.
  3. Copy across an .fsx file as a web job. I needed to figure out how web jobs are stored on the app service in order to know the path to build up, of course.
  4. Do some jumping around of folders before doing an MSBuild of my application.
  5. Call Kudu Sync to do the final deploy, passing in the set of folder locations needed for the tool.

Using batch files for a built pipeline probably isn’t the best way to go. The problem with this is that managing a set of build sets quickly becomes a pain with a batch file – you have GOTOs everywhere, labels etc., you can’t do complex control flow etc. etc. Imagine you wanted to now run unit tests, and if that failed, do some set of tasks, but if it passed, do something else etc. etc. – it quickly becomes a nightmare.

Enter FAKE

On the other hand, FAKE is an excellent library and DSL designed to act manage a build pipeline. Not only does it have loads of helpers for e.g. file system access, config file rewriting, environment variables and MSBuild etc. but it allows us to define build pipelines with dependencies – even conditional stages. Finally, because FAKE is just F# and runs on the full .NET framework, you can always break out and just run any .NET code you want directly from within a FAKE script. With FAKE, you can have a single build script for e.g. local builds, CI builds and also now supports Kudu deployment builds through the newly-added Kudu module in FAKE. Let’s see what the above build script looks like in FAKE: –

You can see here that there are several distinct build steps, which are composed together as dependencies on one another at the very end using the ==> operator. Note that the code above is actually just F# although we’re using a specific DSL with custom operators to set up a “build chain”. So if any of the stages fail, the whole build will fail and we’ll be presented with a summary log (which we can see directly in the Azure portal) of the results of the build. Notice also the lack of environment variables etc. – the Kudu helper module takes care of all of that for us – whilst we don’t need gotos anymore because FAKE handles the build pipeline.

Now our Kudu script is much simpler, because we’re delegating control of the main build orchestration to a language better able to reason about and define program flow: –

:: Restore NuGet packages
.paket\paket.bootstrapper.exe
.paket\paket.exe restore

:: Start main build script
packages\FAKE\tools\FAKE.exe build.fsx

Conclusion

Kudu and Azure App Service are great tools. By plugging FAKE into the mix, we get both a succinct and easy to use scripting experience with the power of the .NET framework and a fantastic language like F# as well.

Working with running totals in F#


This post is an expanded version of February’s F# Gazette.

A common issue that developers that come from an OO background struggle with is how to effectively create accumulations over sequences of data without resorting to mutable state in an effective and succinct style.

Let’s start with a fictional set of results for a sports team, and a function to convert from the outcome to a points earned: –

let results = [ "W"; "L"; "L"; "D"; "W"; "W"; "L"; "D" ]
let convertToPoints = function | "W" -> 3 | "D" -> 1 | _ -> 0

We would like to graphically show the form of the team by accumulating the points earned over the series: –

newplot.png

So the question is really, how can we easily go from a series of singular results to one that maintains a running total of all the results previously as well? Let’s look at some alternatives that you might come up with.

Managing accumulation with mutation

The implementation that you might initially consider doing is one which uses a mutable variable to track the running total.

This code is not terrible – it’s not that hard to infer its meaning, particuarly if you come from an imperative background. And although this code doesn’t use the mutable keyword anywhere, it’s still of course mutating the output collection through the call to .Add(). So because we’re functional programmers and we’ve been told that mutation is the spawn of satan, we have to find another way around this. Let’s try recursion – that’s usually a good trick.

Managing accumulation with recursion

Ehhhhh. I’m not fond of this solution. I find it hard to reason about recursion sometimes, and this is a good example – I need to think about the “end case” of the recursive function; the code doesn’t look particularly nice. We have to manually manage the output list, and remember to reverse it at the end (because in F#, it’s more efficient to add to the head than tail of a list) etc. etc. But at least we got rid of that mutable variable, which is all important, right?

Improving readability through higher order functions

This sort of dogmatic approach is guaranteed to turn people off to FP. Yes, mutation is best avoided (although in a case like this I would argue it’s not that dangerous as the mutable state is short lived – just a few lines of code) but I would argue that in this case, the cost of the recursive version in terms of readability is too high. So we can improve this by using the fold() function that exists in the collection libraries (I’ve put type annotations in for illustrative purposes, but they’re not necessary).

fold() is nice! It generalises the problem of managing state for us across iterations of a collection, so we don’t have to pattern match or manually pass items into a recursive function. This code is certainly less than the recursive version, and once you know how fold works, it’s much easier to reason about. So we start with a list of [ 0 ], and then manually calculate the next item in the list by manually adding the head of the list with the next result we’re passed by fold(), and then pushing that answer onto the head of the list: –

[0], W - add 3 points.
[3; 0], L - add 0 points.
[3; 3; 0], L - add 0 points.
[3; 3; 3; 0], D - add 1 points.
[4; 3; 3; 3; 0], W - add 3 points.
[7; 4; 3; 3; 3; 0], W - add 3 points.
[10; 7; 4; 3; 3; 3; 0], L - add 0 points.
[10; 10; 7; 4; 3; 3; 3; 0], D - add 1 points.
val it : int list = [0; 3; 3; 3; 4; 7; 10; 10; 11]

However, we’re still having to manually maintain the output list ourselves and then reverse it with that ugly List.rev at the end.

Using higher-level folds

So it turns out that there’s a derivative, constrained version of fold() that is designed for exactly what we need – generating output lists based on accumulations – and it’s called scan(). Scan works exactly like fold, except it generates a list automatically based on the result of each iteration. Here’s the code: –

That’s it. In this version, we don’t need to manage state across calls, just like fold – but we also don’t need to generate a list either – that’s done by scan as well! In addition, we don’t need to reverse the list – again, this is taken care for us. The same sort of logging as above now yields the following: –

Current form is 0, result is W - add 3 points.
Current form is 3, result is L - add 0 points.
Current form is 3, result is L - add 0 points.
Current form is 3, result is D - add 1 points.
Current form is 4, result is W - add 3 points.
Current form is 7, result is W - add 3 points.
Current form is 10, result is L - add 0 points.
Current form is 10, result is D - add 1 points.
val it : int list = [0; 3; 3; 3; 4; 7; 10; 10; 11]

Notice that this time, we never actually see the output list within each iteration – instead we’re just given the current running total.

Taking it even further

If you want to be one of the cool kids you can reorder the transformations so that you can make the code even more succinct: –

And this can be simplified further still by taking advantage of curried functions as: –

Conclusion

I daresay that you might not necessarily want to go as far as this last version, depending on your experience with currying and composition in F#, but you can see that by delegating the “boilerplate” of iteration with state, and outputting a running total, we can reduce even the mutating version to just: –

  • our original function to go from W | D | L -> points
  • two higher order functions that we compose together
  • an addition operator
  • a starting state of 0

No mutation, for loops, recursion or list manipulation.

And just to come full circle – I believe that many of the accumulation functions within the collections library in FSharp.Core (include fold()) actually use mutable variables!

Running F# on Microsoft Azure

fsharp128

As we keep seeing the same questions on Twitter (or elsewhere) regarding how to run F# applications on Azure, someone suggested to me that I write a post that outlines what your options are, and when and where you should use one service or the other.

.NET on Azure

Azure has always had a good “developer-focused” story for deploying .NET code. Unlike, say, Amazon Web Services (which started with a VM-first story and only more recently has started to offer higher-level abstractions), Azure has always been about offering the .NET developer ways to run your code in a way that are abstracted from the VM. In the early days of Azure, there was really only one option for hosting code (and zero options for running VMs), but as the number of Azure services offered grows at an incredible rate, there are now a number of different options available out-of-the-box in Azure. So, here’s a brief outline of some of the compute options that you have in Azure currently.

Disclaimer: If you’re an Azure boffin and notice some simplifications in this posting – it’s by design🙂 But please let me know if you notice something that’s completely wrong!

Cloud Services

Cloud Services were as far as I’m aware the first platform service offering on Azure. You write your .NET code, you wrap it in a “cloud service”, and push it into Azure. Azure will handle provisioning of a VM for you, deploy your code onto it, and provide you with a load-balancer from which you can expose specific ports to your application as required. Alternatively, you can use e.g. some of the Azure messaging mechanisms such as Queues or Service Bus for sending commands to your worker role.

The process of wrapping your application in a Cloud Service basically involves bundling up your application into a zip file which contains the application files plus metadata about the service e.g. the VM size that you wish to run on, how many instances you want to run, the name of the service etc. etc.

Once your code is running, you can scale out the service to a number of “instances” of that service (aka “role”) by dragging a slider within the Azure portal, or use an auto-scaler which scales based on a number of metrics e.g. CPU utilisation.

Of course, Visual Studio has a number of templates included out of the box for creating such projects, including (which might surprise some people) an F# one. The only issue is that FSharp.Core isn’t included on Azure VMs by default, so you’ll need to ensure that it gets included in the output of your code (either by setting CopyLocal to True, or using the FSharp.Core nuget package).

The good thing about Cloud Services is that they’re very flexible – you can run any arbitrary .NET code (or non .NET) on them, as they essentially run in a similar manner to Windows Services – you get a number of events such as when the service is spawned, when it shuts down, if there’s a change to the configuration etc. You also get an API for configuration settings and even a distributed cache service across all instances of a role (although my understanding is that this service is being deprecated in the next 12 months).

However, Cloud Services are starting to show their age – there’s no support for e.g. direct integration with GitHub or other source control mechanisms. Instead, you need to package up code into the “.cspkg” format and deploy – which can take quite a while (minutes rather than seconds). Furthermore, there’s not much flexibility in terms of deployment – you write your code, you deploy it, and it maps to a VM. There’s no scope for multi-tenanting multiple services on a VM unless you manually do this yourself within the context of the Cloud Service at the code level, and configuration is a little bit of a pain compared to some of the other options out there.

F# and Cloud Services

Use Cloud Services for any F# applications you want to use – think of Cloud Services as distributed, replicated Windows Services. As an example, the MBrace Azure project runs on Cloud Services. Just note, that because they’re so flexible, you might have to do some plumbing to fit a Cloud Service to your specific needs (this isn’t just in F#) such as opening ports or the like, and there’s no specific Cloud Service APIs to connect to other Azure services.

Web Apps

Web Apps (formerly Azure Websites) were originally designed in order to simplify the use case for hosting ASP .NET applications in Azure as opposed to using Cloud Services, responding to HTTP traffic over port 80. As such, there’s out-of-the-box support for hosting any ASP .NET application in IIS directly from a source code repository e.g. GitHub; in addition, you can use the WebDeploy mechanism and tooling built into Visual Studio, plus there’s FTP support. Web Apps also come with the Kudu scripting language / tooling which is a kind of batch scripting set of commands for things like deployment-time file diffs. WebApps also support eight “deployment slots” that can host multiple versions of your website. So you can deploy to a number of “slots”, and hot-swap the “live” one in and out with basically instant results.

But, Web Apps are written at a higher level of abstraction than, say, Cloud Services – think of it as “IIS as a service” rather than just “code as a service”. This is what they’re designed for, and everything about them, from all the examples you see on the Azure website to the out-of-the-box reports and metrics in the portal, shows that they are geared up for HTTP-enabled applications. There are actually some good examples of using F# within ASP .NET – either directly using some of the third-party Visual Studio project templates etc., or using an IIS module that allows you to redirect all HTTP requests to any arbitrary executable – you can use such a mechanism to e.g. run Suave on Azure websites.

Of course, there’s nothing to stop you writing an application that ignores HTTP traffic and communicates through e.g. Azure Queues or whatnot, but you’d probably be better off using Cloud Services for something like that. As a “get out” for the HTTP route, you can also make use of “Web Jobs”, which are arbitrary executables that run within the context of a Web App, and can be set up to run or a schedule, continuously, or whenever e.g. a message gets dropped onto a storage queue. So imagine an e-commerce website where search results are displayed based on a batch-generated set of read-only database results or similar. You can use a web job that runs every n minutes or whenever some data changes to regenerate the table. This all runs within the context of the “website”, on the same VM. So if you want to scale the web job, you also have scale up website at the same time.

Web Apps are a great option for running applications that are geared up towards serving HTTP requests, particularly if you’re running an ASP .NET application over IIS. There are also some options for running other types of application e.g. batch processes within the context of the website. You can also run them for less money than Cloud Services, as the cheaper options allow for hosting multiple web apps within a single physical VM. So whilst Wep Apps lack the flexibility of Cloud Services e.g. you don’t get the same choice of VM sizes and run within the framework of IIS, they’re a much more modern option, with better support throughout Azure, and you should consider this where possible. There’s also much better support on the newer portal for Websites, and features like authentication with a number of providers e.g. Twitter, AD, Google etc. are included as part of the service for free.

F# and Web Apps

Use Web Apps for any application that is primarily HTTP-facing. You have several options for running F# within a Web App: –

  • Create a C# web project as a thin e.g. MVC veneer and delegate calls to your F# code.
  • Create an F# web project (either hand rolled or using one of the third-party templates available) and use something like Web API through the full ASP .NET stack or via OWIN / Katana. I believe that there’s also a full MVC F# template out there as well.
  • Deploy an alternative application that can respond to HTTP traffic e.g. Suave, and use a custom module in IIS to route traffic through that. There’s a working solution for this within my GitHub repo.

Service Fabric

As Cloud Services have aged, and containerised solutions have become more and more popular e.g. Docker, Microsoft have looked to create a more modern Azure service for hosting your code. Service Fabric allows you to host any executable (.NET or not) – so in this respect it’s somewhat like a Cloud Service. However, unlike a Cloud Service, a Service Fabric cluster is made of a number of nodes, on which you can host a number of services. In Microsoft’s mind, this allows you to create a whole host of micro-services hosted on your SF cluster. Each service within an SF cluster will be replicated across the nodes of the cluster – you can shape this behaviour with a reasonable level of control e.g. singleton services / n instances / maximum i.e. = cluster size. If a node goes down another one will spin up automatically and rehydrate its state, and you can also dynamically resize a cluster to introduce new nodes as required.

In addition to the usual Azure services that you may want to connect to, you can also use a number of SF-specific services that are hosted and run within the context of your cluster. These are services to allow sharing data in a consistent manner, and messaging across services in the cluster. These include creating .NET collections that work in a consistent, transactional manner across the cluster (sort of like the Concurrent collections in .NET but distributed across nodes rather than threads). So if you can imagine a micro-service which needs to maintain some state, rather than resorting to a database you could simply use an e.g. out-of-the-box distributed list.

Whilst Service Fabric is a relatively new public service offering on Azure, it’s actually been used internally by Microsoft for a number of services for a while one – so it’s a tried and tested technology.

It’s worth pointing out that in addition to the “host any raw executable” model, Service Fabric comes with an Actor model loosely based on the Orleans model. It’s not exactly the same, but a fairly close fit. But don’t think that Service Fabric is an actor service – it’s a much more flexible container model that happens to come with an actor model.

F# and Service Fabric

It’s important to know that out of the box, there is zero tooling support for Service Fabric with F#. There are no project templates, and no automated tooling for updating the myriad of XML configuration files or Powershell deployment tools that are required for Service Fabric. I’ve documented on this blog the steps that are (or were – that post is a little old now) required to get SF up and running with F# in terms of the project structure and tooling, and it is possible. I do wonder if there’s a more “lightweight” way to go about it though i.e. just deploy an executable with a lightweight project shell that hosts that exe, which would be decoupled from the SF project fluff. Currently the story that’s told by Visual Studio and Microsoft in terms of the configuration that’s required seems a bit of a backwards step to me.

Having said all that – once it’s up and running, Service Fabric offers a great story in terms of features for hosting services. It’s more powerful than Cloud Services, with out-of-the-box micro-service messaging and both stateless and stateful services that can replicate and scale automatically, as well as multi-tenanting services across single nodes.

If you’re think of hosting non-web facing code – whether micro services working together or arbitrary disconnected services – Service Fabric is a good choice. But it’s a more heavyweight option than e.g. Cloud Services in the sense that it contains a cluster of nodes (i.e. VMs) designed for hosting multiple services. It might be overkill to use it for a single service.

Conclusion

Azure offers a number of great developer-facing options for hosting F# applications, whether web-facing APIs or back-end services that receive requests via e.g. queues. There are also other compute options that I’ve not touched on here (such as Azure Batch), and more features coming all the time – so it’s definitely worth trying out to see how it can open up some new possibilities for your specific needs.

Visual Studio Team Services and FAKE

4

What is VSTS?

Visual Studio Team Services (VSTS) is Microsoft’s cloud-based source control / CI build / work item tracking system (with a nice visual task board). It’s a platform that is evolving relatively quickly, with lots of new features being added all the time. It also comes with a number of plans including a free plan which entitles you to an unlimited number of private repositories with up to 5 users (and MSDN users do not count), plus a fixed number of hours for centralized builds.

The catch is that there is (at least, I couldn’t find!) any way to host a completely public repository with unlimited users – obviously a big problem if you want to have an open source project with lots of contributors. But for a private team, it might be a good model for you. You can also use the CI build facilities of VSTS with GitHub repositories – so in this sense you can treat it as a competitor to something like AppVeyor perhaps.

Contrary to common opinion, VSTS is completely compatible with Git as a source control repository. Yes, you can opt to use TFS as a source control model, but (in my opinion) you’d have to be crazy to do this or have a team that are really used to the TFS way of working – I find Git to be a much, much more effective source control system.

Why FAKE?

I wanted to try and see whether it was possible to get VSTS working with FAKE.

One of the best things about FAKE is, in addition to the ease of use, flexibility and power that you get by creating build tasks directly within F# (and therefore with the full .NET behind it), is that because you are not dependent on hosting a bespoke build server with custom tasks etc., such as Team City, it’s extremely rare (hopefully never) that you have a situation where a build runs locally but fails to run on the build server.

Unlike relying on e.g. Team City to orchestrate your build, you delegate the entire CI build steps to FAKE – MSBuild, Unit Tests, configuration file rewriting etc. etc.. So if the build fails, you don’t have to e.g. log into the Team City box, check some log files etc. – your FAKE script does all the heavy lifting so you can run the exact same steps locally.

Putting it all together

So my goal was to write a simple FAKE build script which pulled down any dependencies, performed a build and ran unit tests – all integrated within VSTS. As it turns out, it wasn’t very difficult at all.

Firstly, we hook up the build to source control. In our case, it’s the Git repository of the Team Project, so works straight out of the box, but you can point to another Git repository e.g. GitHub as well. You can also select multiple branches. We then set a trigger to occur on each commit.

1.pngSecondly, we have to set up the actual build steps. As we’re delegating to FAKE to perform the whole build + tests, we want to use as few “custom” VSTS tasks as possible. In fact, we actually only need two steps.

  1. Some way to download Paket or Nuget, and then initiate the FAKE build.
  2. Some way of tieing in the results of XUnit that we’re going to run in FAKE into the VSTS test reports.

Unlike old-school TFS etc., VSTS now has an extensible and rich set of build tasks that you can chain together – no need for Workflow Foundation etc. at all here: –

3.png

Notice the “Batch Script” task above – perfect for our needs, as we can use it to perform our first build task to download Paket and then start FAKE.

We can now see what the FAKE script does – this is probably nothing more than what you would do normally with FAKE anyway to clean the file system, perform a build and then run unit tests: –

Notice that when we run unit tests, we also emit the results as an XML file. This is where the second build task comes into VSTS (Publish Test Results), which is used to parse the XML results and tie into VSTS’ build report.

2.png

So when we next perform a commit, we’ll see a build report that looks something like this: –

4.png

Notice that the chart on the right shows that I’ve run 2 unit tests that were successful – this is the second build task parsing the XUnit output. Of course we can also drill into the different stages if needed to see the output: –

5.png

Conclusion

This post isn’t as much about either VSTS or FAKE features per se, as it is about illustrating how both VSTS and FAKE are flexible enough that we can plug the two together. What’s great about this approach is that we’re not locked in to VSTS as a build system – we’re just using FAKE and running it centrally – but if we’re using VSTS we also can benefit from the integration that VSTS offers with e.g. Visual Studio and the build system e.g. creating work items, associating commits to work items and viewing from VS etc. etc. – whilst still using FAKE for our build.

F#, .NET and the Open Source situation

fsharp128

If you read my (generally sporadic) blog postings, you’ll know that in general I write about either F# and / or Azure, usually from a programmatic point-of-view. How easy is it to reason about a certain thing? How quickly can we make use of some Azure service from F#? And so on. In this post, as part of the annual F# Advent calendar, I want to switch focus and instead discuss an increasingly important areas of software development that I’ve been exposed to as part of my learning and adopting of F# – that of open source software development.

Disclaimer: There have been a number of debates on Twitter and GitHub regarding open source within the context of several Microsoft-led projects over the past few weeks. This post is not a response to those issues; this post was in fact authored in early November, well before any of these debates had been initiated.

The distinction between .NET communities

Coming from someone who has been a .NET developer almost exclusively throughout my career (aside from a brief stint with C++ and the obligatory various web and database languages that you get dragged into), one of the elements of working in F# that you can’t really ignore is the community involvement that permeates through just about everything that happens in and around F#, from the development of the language to key packages that are developed, through community meetups and user-groups, through to the tooling that is used on both “classic” .NET and Mono / CoreCLR stacks. Before I started working with F#, I had very little experience of open source development or tools etc. Thankfully, I’ve found the community to be extremely open, friendly, and always happy to help out others, as well as to continually improve the state of software development in F#. So if there’s a lack of tooling available for a specific purpose, or a tool that doesn’t quite do what is needed, the mentality in the F# community is to improve the existing tooling if possible, and where not, simply write a new tool from scratch.

This is a very different mentality in the rest of the .NET community, where we are by and large reliant on what Microsoft serves up both in terms of tooling, libraries and frameworks. This is not necessarily a failing of either the community or Microsoft – it’s just the way it is, and it’s something that has been fostered over the years by both parties. This model is slowly changing, and there are some popular OSS projects out there – JSON .NET being an example of something that is owned by the open source community and Microsoft themselves now use. Other examples of open source software within the .NET community include EventStore and Nancy. However, generally the average .NET developer is content to accept the pace of change, and direction, that Microsoft sets. This has historically not necessarily been a bad thing, but with the increasing rate of change of programming languages, frameworks and tools, it’s hard to see that remaining a successful model.

Misunderstanding the F# community

I think that these two different mindsets have (somewhat unfairly) earned the F# community a reputation of being somewhat disruptive and of not “going with the flow”. It’s true that some F# projects (notably type providers) are not compatible with C# and VB .NET. Yet it’s often forgotten (or not even known) that many open source projects that happen to have been written in F# work perfectly well in C# and VB (and often are explicitly designed with that in mind). Some examples include: –

  • Paket – a dependency manager designed to work well with NuGet packages and GitHub repositories
  • Fake – a build automation system using a DSL built in F#
  • FSCheck – a property-based testing framework based on Haskell’s QuickCheck
  • Project Scaffold – a build template with everything needed for successful organisation of code, tools and publishing.

It’s sometimes disappointing to hear well-known people throughout the .NET and Microsoft community often misconstrue projects like these as somehow to only be of use to F# users.

At the same time, there is an element of smugness and / or superiority that sometimes permeates from some corners of the F# community, which I think isn’t all that helpful. To an extent though, rather been being an essentially negative act, this is caused by the enthusiasm of the community and its desire to improve the state of software development within .NET, and perhaps also by the frustration caused by some of the misunderstandings and FUD about F# in the public domain e.g. it’s only for maths people, it’s too hard to learn, it’s not of use in line-of-business applications etc..

Defining open source software

I’d like to discuss a little about open source software now. Frankly, I would not consider myself an expert in defining what open source really is, but having now spent a few years now in the F# community contributing to a few different projects, I certainly have a better idea than I used to, and do feel qualified to at least put down some opinions. I’m taking a leap of faith here that my experiences in F# open source projects reflects positively on open source as a whole, because despite any  criticisms that can be leveled at the F# community and ecosystem, it really has a fantastic approach to open source, collaborative software development that can be learned from. Here are some points of interest – and misconceptions – that I’ve observed from looking at a number of projects across the .NET community, both in and outside of the F# community.

GitHub

GitHub is a great site, and has become the de-facto repository site for most F# open projects, as well as even Microsoft’s attempts to move into the open source world. However, creating a code repository on GitHub (or moving an existing code repository from CodePlex to GitHub) does not instantly make a project “open source”. I’ll say that again, because it’s worth making a point of this. Just the act of putting some code on GitHub does not mean it is an open source project. If it were, it’d be easy to make a project open-source. However, I’ve learned that working on truly open source software is much more than simply making the repository public for everyone to look at.

Cross Platform .NET

I also see this mention of “open source” as somehow being a catch-all term for software developed for Mac and Linux (particularly from some Microsoft quarters). This is a mistake. It suggests that software that is developed on (or for) Windows doesn’t need to be open source, or is somehow different in terms of how software is developed. It’s great that Microsoft are moving to a more open model, and adopting CoreCLR for running .NET code cross platform – but, again, just making software run on Mac and / or Linux does not immediately elevate a project to being successfully “open source”.

Sharing is Caring

But the biggest challenge I see in creating a successful open source project is to adopt a truly collaborative, pro-active approach to software development. This means being genuinely interested in external feedback on your project – even if it’s not what you want to hear. It means encouraging people to submit pull requests. It might also mean giving up complete ownership of the project where you might not agree with every feature proposed by the community on the project, or feel comfortable with strangers submitting PRs to your project written in a different coding style to what you are used to. It also means getting feedback in at an early a stage as possible – ideally at the planning stage of any feature – and accepting help from the community in shaping the design and implementation of said feature. This is not the same as working on a project for several months, releasing it to the public in GitHub, and then getting feedback on it retrospectively.

It’s only through adopting a really open, collaborative approach to software that you can have a cycle of < 48 hours from someone submitting a feature request, to several people giving feedback on the feature, to someone implementing it.

Microsoft, .NET and Open Source

Obviously, Microsoft have taken a big step in the large year or so to try to adopt “open source” development processes. This is to be applauded. But they’re not there yet, and there is some evidence which lead me to believe that it will take them a while to truly get open source – but they are trying. Some of the teams seem to have gotten the idea of getting feedback early e.g. the C# compiler team. Ironically, a mature programming language like C# is one type of project that you probably need more control and direction over than e.g. a library, where the barrier to entry is much lower. Conversely, there are also a number of teams at Microsoft developing frameworks and libraries that are ostensibly open source, yet I’m really not seeing that much in the way of a collaborative mentality except for the superficial steps of putting a repository onto GitHub and asking feedback after a period of time spent developing in isolation. Releases and features and still managed with a reactive rather than proactive mentality. PRs from non-Microsoft team members are few and far between. Direction in many of these projects is controlled almost entirely by Microsoft. In short, the general open source community aren’t yet empowered to contribute to these projects in a way that I would like to see.

Conclusion

An important element of the distinction between the F# community and the rest of .NET is likely due to the differences in terms of investment in both F# and C#/VB.NET stacks by Microsoft – the fact that F# has been forced to stand on its own feet has helped it mature extremely quickly. In addition, it should be mentioned that there have been a number of key individuals within the F# community who have helped both grow and nurture the community – I’m not sure just how F# would be looking  today without them. Ultimately, what we do have today with F# is a growing global community, and an ecosystem of tools and libraries which takes the best of existing .NET packages combined with others that harness the power of F# as well. It’s a community which includes the Microsoft Visual F# team as a valued member, rather than a all-controlling entity, and a community  which – whilst still growing and evolving – is open, confident and in control of its own destiny.

Thanks to the community for showing me all of this over the past few years, and allowing me contribute towards a fun, vibrant and growing ecosystem. If you want to learn about open source projects, and secondly about F# in general, you could do a lot worse than look at some of the up-for-grabs issues of the projects on http://fsprojects.github.io/.