Migrating Complex NuGet solutions to Paket

In the previous post of this series, we looked at migrating a simple, single-project solution over to Paket. In this post, we’ll look at a more complex solution that has multiple projects with shared dependencies, and see some of the issues that Paket picks up that might not have been previously identified. Most of this post will actually be an analysis of highlighting existing problems in a solution, and how Paket can help to highlight them in a clear way.

The solution under test

The solution we’ll use is a recent build of the official Azure Webjob SDK. This project is much larger than the one we looked at in a previous post, with twelve C# projects that are a mixture of application and test projects (some of which reference each other). The latest builds also target .NET Standard 2.0 (the latest preview), and there are a  mixture of packages.config and NuGet references. In other words, this is a decent test of Paket’s ability to cope with a reasonably complex project structure as well as to identify some of the issues that can crop up with PackageReference.

I should stress, there was no specific decision to pick this particular solution – I simply pulled the first large-looking one down from GitHub on the Azure repos on GitHub, tried a convert to Paket and waited to see what would happen. So before anyone jumps to the wrong conclusion, this is not an assassination attempt on this project / the Azure Functions team / Microsoft / NuGet / Azure etc. etc. The Functions team are doing a great job and (incidentally) happen to be one of the more open-source looking teams at MS today in my opinion. In fact, I suspect that one would find these same sorts of issues across many .NET projects today – think of this project as a sample case study.

The Conversion Process

The conversion process in principle should be as per the previous blog post – download the repo, run convert-from-nuget, and happy days. But in reality, this repository highlights a number of interesting issues with the current set of NuGet dependencies.

Unifying package versions

Running the standard convert-from-nuget command shows us immediately one of the “hidden” issues that can develop without Paket – running the conversion process will present the following output:

Moq is referenced multiple times in different versions ["4.7.99"; "4.5.30"].
Paket will choose the latest one.

This is also observed for the following other packages:

Newtonsoft.Json [10.0.3; 9.0.1]
WindowsAzure.Storage [8.3.0; 8.1.3; 7.2.1]
XUnit [ 2.3.0-beta3-build3705; 2.1.0]
XUnit.Runner.VisualStudio [2.3.0-beta3-build3705; 2.1.0]

This is Paket’s way of highlighting the fact that the package references for projects across the solution are currently inconsistent. In other words, the solution is using different versions of the same project across the solution. By default, Paket doesn’t allow this, so it unifies them into a single version. Paket does this by choosing the highest across all versions in the repository, so in the example above Paket will remove the older Moq version and replace it with the latest one (4.7.99). This is illustrated in the figure below, where the project in red has an inconsistent version number compared to the other two projects.

This is one of the most common issues you’ll get when converting from NuGet to Paket. A fundamental difference between Paket and NuGet is that Paket, by default, enforces a single version for a given package across the entire solution. Conversely, NuGet allows you to specify different versions per project (and won’t stop you doing this). Now, there are definitely times when this is a valid thing do to (and Paket does allow you to opt-in to this through its groups feature), but I would suggest that 9/10 times, you’ll want a common, consistent version for your packages across your entire solution.

In this example, the problem occurred because the ServiceBus test project has a packages.config file that points to the older 4.5.30 version, whilst the four other test projects have been updated and are now pointing to 4.7.99.

The dangers of inconsistent packages versions

It’s worth pointing out that this issue in NuGet can appear to be completely harmless. At runtime, .NET will choose one of the two versions (I believe it picks the newer one), and you might continue merrily on your way, never knowing about this problem (assuming that your binding redirects are set up correctly). However, if there’s a breaking change between the two versions, things may start to go wrong:

1. If a method has been removed from one version to the next, you won’t get a compile error – your application will simply stop working at runtime with a missing method expection if and when your code hits that point. Boom. This could be minutes, days or even years later depending on how often that code path is run.
2. If the behaviour of the method has changed across the two inconsistent versions of the dependency, you may find that your application doesn’t crash, but simply acts differently than you expect. This is in some ways worse than a missing method exception – it’ll simply silently do the wrong thing. You won’t even know at runtime until perhaps you get a bug report and start digging into it (I can only imagine how long it would take to figure this out).

This sort of thing is particularly dangerous if you have multiple references to shared DLLs, as in the following diagram:

Here, three assemblies all have a dependency on Package A. But unfortunately, the last time a developer went to update the package, instead of clicking on the Solution node in VS, he accidentally selected the Web project. He then updated the package to that project only and committed the change. At the same time, V2.0 introduced a new bug which breaks the application.

Now, at runtime, which version of Package A gets deployed? With the Web project, it’s v2.0 – which could mean serious problems with the behaviour of any code in the Common project, since at runtime it’ll be using a different version than at compile time. Remember, you might not know this until a long time post-deployment.  Even worse, because the Test project is still at the working version of v1.2, your unit tests will still continue to behave correctly, giving the developers a false sense of security. If they’ve invested in a large automated integration test suite that runs as part of a CI/CD process, they might catch this issue before deploying.

This sort of worse-case-scenario isn’t something I’ve spent hours dreaming up. These situations can (and do) easily happen.

Hidden Inconsistencies

The above example I showed you from the Azure solution was a fairly simple one that is probably relatively harmless – Moq is a test library, and the upgraded version was a minor step up, so the potential for damage was most likely minor. Here’s another inconsistency taken from the same solution with a package used in many, many places: Newtonsoft.Json.

Recall from the earlier Paket message that we had two different versions of Newtonsoft.Json used across the solution: 9.0.1 and 10.0.2. Where do these come from? Why are there two versions? A look through the various config files shows that it comes from several places (I identified these by manually looking at all the projects in the solution. It took a while.):

• Host: 10.0.3 (an explicit PackageReference)
• Logging.Application: 9.0.1 (transitive of the Azure.Storage.8.3.0 PackageReference)
• Logging: 9.0.1 (transitive of the Azure.Storage.8.3.0 PackageReference)
• Protocols: 10.0.3 (an explicit PackageReference)
• ServiceBus:
  • 9.0.1 (explicit in packages.config)
  • 9.0.1 (transitive of the Azure.Storage.8.1.3 PackageReference)

Disclaimer: It’s possible that the implicit resolutions based on the transitives resolve to 10.0.3. I couldn’t confirm this though.

This is pretty complicated! There are two different package versions of Newtonsoft.Json here, coming from four different routes:

1. An explicit PackageReference of 10.0.3.
2. An implicit reference of 9.0.1 that comes from projects that reference Azure Storage 8.3.0;  NuGet pulls down the minimum version by default for transitives, which in this case is Newtonsoft.Json 9.0.1.
3. An explicit reference to 9.0.1 in a packages.config file.
4. An implicit reference as per point 2, except this time it’s off of Azure Storage 8.1.3.

The implicit references are somewhat dangerous, since there’s no easy way to know that this has happened. Even more confusing is that the Service Bus project has both a packages.config file and an implicit PackageReference. In this case they both end up resolving to 9.0.1, but what would happen if they didn’t? Which version would end up winning? Is the packages.config even used?

Also, note that we’ve silently gone from 9.0.1 to 10.0.3 – a major package upgrade which theoretically could include a breaking change. I have no idea what changes have been made here, but it’s evidently something that you should at least be aware of.

As an exercise to the reader, do the same with WindowsAzure.Storage which has three different versions. Where do they come from? Why are they there? What’s the impact on unifying them?

Conflicts

It’s important to note that the above inconsistencies were issues with the existing solution – they’re there, right now, in this repository. After we run convert-from-nuget, these inconsistencies will be unified by Paket as part of the conversion process. Most of the time this will work without a problem, but occasionally it’ll identify conflicts which will prevent Paket from creating a valid lock file. In such situations, you’ll have to resolve such errors one-by-one until the dependencies file is valid and a lock file can be created. Once Paket has completed the conversion process of this repository, you’ll see that it fails:

So, Paket has created a dependencies file based on a unified set dependencies taken from all the NuGet references, but one that is invalid, and this prevents it creating a lock file.

Fixing conflicts

The incomptability is caused because we’ve unified Moq to 4.7.99, yet Castle.Core was explicitly imported as 3.3.3, and the two are incompatible: Moq 4.7.99 needs at least Castle.Core 4.1.1. We have two choices:

1. Downgrade Moq back down to 4.5.30, which is compatible to Castle.Core 3.3.3
2. Upgrade Castle.Core to 4.1.1, which is compatible with Moq 4.7.99

In this case, let’s try upgrading Castle.Core. Open the paket.dependencies file, and modify the Castle.Core line so that it points to Castle.Core 4.1.1 instead of 3.3.3, and run paket install. This will resolve that error but introduce another similar problem, this time with XUnit. This time, you can fix it by downgrading XUnit from 2.3.0-beta to 2.1.0.

Tip: When fixing these issues, it’s worth looking at the report from the initial conversion process – usually the different versions that have been unified will lead to the possible solutions.

Keep in mind that the objective at this stage shouldn’t be to get all packages at the latest and greatest versions. Instead, concentrate on getting a valid set of dependencies. Once you’ve done that, you can then move forwards in a controlled fashion from a stable base, one dependency at a time.

Restoration

Once you fix the issues, Paket will complete resolution of the entire graph. As this is a combined “full” .NET Framework and .NET Core solution, it’ll take a while to complete – mostly due to the sheer number of packages that are involved in any .NET Core (note – as an example of the graph complexity, the lock file that gets generated will be almost 1500 lines long!). Once Paket has completed resolution of the lock file, it’ll then download all required packages.

Cleaning up

Once you get to this point, ensure that the build still works, and then commit. Then, run simplify to get to a more managable dependencies file and remove any pins. Here’s a sample of the simplified dependencies files which represents all dependencies across the entire solution. Compare this with what we started with – instead of inconsistent versions of packages scattered across multiple packages, all our dependencies are now in a single place – and known to be consistent. We can update this one file and automatically update all projects. In addition, we don’t need to worry about the version numbers for each dependency – although if we wanted to, we could simply look in the lock file for a more detailed analysis.

You’re now in a position to run commands such as outdated to see what’s out of date, or update to bring everything up to the latest versions. Alternatively, you could just update a single package, one at a time. As an example, running outdated tells us that we could update XUnit to 2.2.0, whilst Azure.ServiceBus is no longer in preview and could be upgraded to 1.0.0.

Conclusion

When converting NuGet repositories over to Paket, you may encounter warnings and errors along the way. We covered two main types here: inconsistencies and conflicts. These are not issues with Paket per se, but rather Paket highlighting to you issues that the repository already had – issues that had previously never been identified.

We also saw how moving to a single dependencies file not only removes the possibility of inconsistent package versions, but also shows how it makes things much, much easier to reason about.

Paket’s workflow makes it very easy to achieve consistent behaviour, but migrating from a NuGet solution means that as part of the conversion process you’ll have to take a little time fixing some of the issues that may have developed beforehand. Thankfully this is a one time only task and should be well worth the effort.

In my final post on the series, I’ll illustrate some of the features that Paket introduces that might be completely new to you, and could open up entirely different ways of developing your projects. See you then.