Analysing website download sizes


When I go to watch football matches at Tottenham, I find it interesting to see the scores and summaries of other teams during e.g. the half time break. There are many sites and applications that can do this. One I use often is the BBC website, where they have a page that summarises what is happening “around the grounds” in one page, with a list of “comments”. A comment might look as follows: –

90+2 mins Man City 1-0 Everton

Throw to Everton deep inside their own half. They have to hurry up.

Some comments may be longer. Others will have images or thumbnails alongside them. But the basic idea is to let you know what is happening across all the games in the Premier League (there are usually 4-5 games being played at once, but sometimes this can rise up to 10)

If my mobile phone network provider EE are not playing up, and I can get reception, then I will use that site at half time, full time, and sometimes mid-way through the game if there’s a stoppage for e.g. injury. The problem is that the site has recently had a bit of an overhaul and I’ve noticed that it sometimes takes age to load, so I wanted to find out why this was – was it EE or just the new BBC site design?

Analysing website performance

The site’s changed because in the past it used to simply be a static list of “comments” that would auto refresh on a regular interval; it’s now more like a SPA, with a static HTML shell with JS, plus a dynamically updating set of data in the centre. It also now loads in two parts – first, a subset of data that contains the 10 most recent comments, and then a button to load all comments for the day (for a typical Saturday, this might be up to 500 comments).

To capture the data, I used the excellent debugging and profiling support built into Firefox that lets you monitor network activity and then divide it up based on the type of data e.g. HTML or CSS. I then put that into a simple F# script and then used FSPlot to push it into HighCharts in a variety of graphs.

Some of the results were quite surprising.

The first is that in the “initial” load of the page (just to see the ten most recent comments), you have to download an astonishing 2.2MB of data, across 85 files. Worst still is how this is spread: –

chart1To put it another way – around two-thirds of the data is taken up by JavaScript or images, some more for the HTML shell and CSS, and a measly 3.5% is used for those actual ten most recent comments that you want to read. It gets even worse if you then click the “load all comments” button: –

chart2

That’s right. Over 9mb of data needs to be downloaded on your phone just to see what’s happening with the eight or ten teams playing football today. Whilst the JS, CSS and HTML shell are unchanged, over 7mb is taken up by 192 images (some of which weigh in at over 700k each). The extra content file (containing the remaining 400+ text comments) comes in at about 700k in total, which accounts for around 8% of the total amount download (although whilst this in itself is in JSON, there’s embedded HTML within the payloads).

Conclusion

BBC web team – this is not a good experience. Even on my desktop PC, when I click the “load all comments” button, it locks Firefox for a couple of seconds when it downloads the remaining comments and then associated images etc. And the main use case is very simple: –

As a football fan, I want to quickly read what is happening around the grounds so that I can laugh/cry at the results.

Nowhere in the above sentence does it say anything about wanting to wait 5 minutes whilst it loads a million images. Nowhere does it say anything about wanting to use up a percentage of my data plan for tons of irrelevant pictures. I just want to read some football commentary!

Lightweight websites with F#


There are several common approaches I’ve seen people take on the .NET platform when writing web-based applications that I want to review in terms of language and framework choice: –

  • Adopt a conventional MVC application approach. Write static HTML that is emitted from the server using e.g. Razor markup + C# / VB .NET, write your controllers and any back-end logic in C#.
  • As above, but replace your back-end logic with F#. This is a reasonable first step to take, because essentially all your data access “back-end” processing are performed in a language that it’s best suited for, whilst your C# is relegated to essentially thin controllers and some simple markup logic.
  • Adopt a “SPA”-style approach. But this I mean split your web application into two distinct applications – a client-side application that is self-managing, typically using Javascript and some framework like Knockout or AngularJS; meanwhile your back-end is a hosted WebAPI written in F#.
  • Write the entire application in F#. Surely you can’t write websites in F# can you? Well, actually, there are some (pretty sophisticated) frameworks like WebSharper out there that can do that, rewriting your F# into e.g. Typescript and the like.

I haven’t used WebSharper in depth so can’t comment on the effectiveness of writing your client-side code in F# and therefore not going to talk about the latter option today. but I have written WebAPIs in F# and want to talk about where I do think your separation of concerns should lie with respect to client and server side code.

As far as I’m concerned, if you’re a .NET developer today, writing websites, then you should be writing as much as of the CLR-side code as possible in F#. I am really pleased with the brevity that you can get from the combination of OWIN, Katana (Microsoft’s web-hosting OWIN framework), Web API and F#. This combination will allow you to create Web APIs that can be created simply and easily, and when combined with a SPA client-side website is a compelling architectural offering.

Sudoku in F#

Some months ago, I wrote a Sudoku solver in F# (I think that there’s a gist somewhere with the implementation). I wanted to try to write a website on top of it with a visual board that allowed you to quickly enter a puzzle and get the solution back. So, having borrowed some HTML and CSS from an existing website, I set about doing it. You can see the finished site here and the source code is here.

Untitled2

Client

  • HTML
  • AngularJS
  • Typescript (no native Javascript please!)

 Server

  • F#
  • F#
  • F#

Standard JSON is used to pass data between website and server. On the server side, we use OWIN, Katana and Web API to handle the web “stuff”. This then ties into the real processing with the minimum of effort. This was all done in a single solution and a single F# project.

OWIN with F#

I’m no Angular or Typescript expert so I’m not going to focus on it – suffice it to say that Typescript is a massive leap over standard Javascript whilst retaining backwards compatibility, and AngularJS is a decent MVC framework that runs in Javascript. What I’m more interested in talking about is how to host and run the entire site through a single F# project. Mark Seeman‘s excellent blog has already discussed creating ASP .NET websites through F#, and there are indeed some templates that you can download for Visual Studio that enable this. However, they still use ASP .NET and the full code-bloat that it comes with. Conversely, using OWIN and Katana, this all goes away. What I like about OWIN is that there’s no code-generation, no uber folder hierarchies or anything like that, you have full control over the request / response pipeline, plus you get the flexibility to change hosting mechanisms extremely easily. To startup, all we need is to download a (fair) few NuGet packages, and then create a Startup class with a Configuration method: –

Now you have that you can simply create Web API controllers: –

So two F# files, a web.config and you’re good to go from a server-side point of view. Talking of web.config – how do you create an F# web project? Mark Seeman’s blog gives full details on creating Visual Studio web projects that are F# compliant, but essentially just adding the “Project Type” GUID in the .fsproj file (I think it’s 349C5851-65DF-11DA-9384-00065B846F21) will do the job.

Combining client and server side assets

UntitledBecause this is a full .NET web project you can do all the things that you would normally do in C# web projects, such as serve up static files (perfect for a SPA) like HTML, Javascript and CSS, as well as generating from Typescript files (just add an project import for the Typescript msbuild target). If you appreciate the extra security you get from F# over other statically typed .NET languages, you’ll almost certainly want to use Typescript over raw Javascript as well, so this should be a given.

A single project that can serve up your web assets and the server side logic to go with it looks pretty simple – in this screenshot, in the api folder is my back-end logic – message contracts between client and server, the actual puzzle solver and the Web API controller.

Client side assets are few and far between – just a SudokuController.ts to hold the controller logic and Index.HTML + stylesheet for the presentation layer. It’s important to note that with a SPA framework like AngularJS, you serve static HTML and Javascript; the Javascript then essentially bootstraps, modifying the HTML dynamically, requesting JSON from the WebAPI and occasionally getting more static HTML. You never modify HTML on the server as you would do with something like Razor.

In addition, as it’s a “normal” website, with Visual F# 3.1.2, you can use Azure websites to deploy this easily – either through VS to manually publish out to Azure, or through Azure’s excellent source control integration to e.g. GitHub or BitBucket webhooks. It’s never been easier to get a CI deploy of a website out.

More flexibility with Web API

Another important thing about Owin is that it separates out the hosting element of the website from the actual project structure. So, after talking about all this nice website project integration, there’s actually nothing to stop you creating a standard F# library project, and then use either the Owin WebHost console application (available over NuGet), or create an empty website or Azure worker and then host it through that via the Owin Host. All this can be done without making any changes to your actual configuration class or actual controllers.

Conclusion

A common misconception around F# is that it’s great for use as a “computation engine” where you give it a number and it gives you back another number. Or perhaps a “data processing engine” where it can read some data from a flat file or a web service and do something to it. These are both true – however, there is very little reason why you can’t use it for full featured Web APIs using Owin (as there’s no code generation to miss out on from e.g. VS/C# projects), and with a minimum amount of effort, even as a full website host for a SPA that will consume that same Web API.

In my next post I want to replace the use of Typescript with F# using Funscript to illustrate how you can have a fully-managed end to end solution for both client and server in F#.