NuGet packages folder

So yesterday I tweeted:

why doesn’t #NuGet allow me to specify where the ‘packages’ folder goes??

and @davidfowl from the team asked me a great question:

Why do you need that?

144 characters just ain’t enough so I figured a quick blog is the best way to answer.

I’ve been using NuGet for about 6 months or so and love the whole concept of picking what I want from a list and let it do the boring stuff (download, unzip, add references etc) is awesome, plus of course the uninstall works brilliantly.

One little thing concerned me but did not stop me from using it…after a while I noticed I had multiple ‘packages’ folders through my source.

Now what I have done for years with 3rd party libraries is put them into a ‘SharedComponents’ (or similarly named) directory which sits in my root Dev folder, easily accessible by all my projects. This means I have one place only where components are referenced. This means all of my projects have the same version, for me this is good (I know if you’re a big dev shop you may need different versions for different projects).

NuGet changed this, giving me a packages folder in ‘a few’ different places. This was not a problem until recently when I added Rx to a project using NuGet and then I started getting a weird ‘co-varient contravarient something’ error (I don’t remember the specifics of the error). The problem was that one of my shared projects was referencing a different (older) version of Rx (installed by NuGet) some time ago.

My fix was to go back to my ‘old school’ way of referencing the Rx .dlls, all projects pointing to the same dlls in the same folder. Worked immediately.

But that sucks. NuGet is a great concept and saves me plently of time, I should be able to and want to use NuGet but version issues is pretty much a show stopper for me, these problems usually surface as very obscure bugs that _can_ take alot of time to narrow down to the fact that there is a versioning problem.

So if I could specify where the ‘packages’ folder always ’lives’ (ie override the default, which seems to be where the .sln file is?), then my versioning problem would no longer be a problem.

Another advantage is keeping source control of components to a minimum. With multiple ‘packages’ folders with multiple versions (often the same) of the same 3rd party components, all of these get stored in source control or the projects won’t build on a build box. Better to have one source….unless you have a compelling reason to keep multiple/different versions.

So for my usage it would be really helpful to be able to specify where the ‘packages’ folder lives :-)  Anyone else agree/disagree…with reason?

ClickOnce – File Already Exists

The Problem

If your deploy a new version of your app and your users see “Cannot Start Application” when updating:

There could be many reasons, this quick post provides a few places to look to find what may be causing the File Already Exists problem.

Click on ‘Details’…and if your error looks something like this…then this most may help:

ERROR SUMMARY
	Below is a summary of the errors, details of these errors are listed later in the log.
	* Activation of ...\Start Menu\Programs\MyApp\MyApp\MyApp.appref-ms| resulted in exception. Following failure messages were detected:
		+ The file '...\Local Settings\Temp\Deployment\GOYP6PP3.TRR\61YND8Y2.Q2A\NaughtyFile.dll' already exists.

Where to look

Open up your ..\Application Files\MyApp1.0.0.0 directory (where 1.0.0.0 is the current build number)

Find and open the .manifest file, should be called MyApp.manifest.

Search for NaughtFile.dll (in the error above, NaughtyFile.dll was the name of the file that already exists), if your problem is the same as mine then you should find two references to this file in the manifest. In my case I had one reference which was a dependentAssembly and another which was a file.

Now, the project you are publishing may not have a direct reference to a dll and yet it still appears in the list of dependentAssemblies because a project it references does directly reference the dll. The easiest way to check this list is to go to the project properties, Publish Tab and open Application Files…

From there you see the list of files that are included, but I’ve noticed sometimes the list is not 100% truthful, if weird stuff is happening, open your project file in Notepad and closely examine the list of included and excluded files.

Note that you can only exclude a file in this list if the project has a direct reference to the file (this may not be true, but I just tried it and it didn’t exclude and I don’t have time for further investigation).

So our trouble, our extra ‘file’ reference came from an extension to msBuild that we have to explicitly include certain files (needed if you need the dll but cannot add a reference for example), that looks a little like this….

  <ItemGroup>
    <AdditionalPublishFile Include="..\..\NaughtyFileLocation\*">
      <Visible>False</Visible>
    </AdditionalPublishFile>
....

By changing the explicit inclusion to ‘not’ include that specific file (we needed other files in that directory), our problem went away. The key is ensuring that your manifest only has one entry for each file that is included…the trick is figuring out why there are multiple entries!

Good luck!

file:///C:/Dev/Release/Master/PmsMasterBuild_2.0.1.11.7zERROR SUMMARY
	Below is a summary of the errors, details of these errors are listed later in the log.
	* Activation of C:\Documents and Settings\mark.wallis\Start Menu\Programs\PMS Smart Client\PMS Smart Client\PMS Smart Client.appref-ms| resulted in exception. Following failure messages were detected:
		+ The file 'C:\Documents and Settings\mark.wallis\Local Settings\Temp\Deployment\GOYP6PP3.TRR\61YND8Y2.Q2A\EcsLite.Downloader.dll' already exists.

2010 in review

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here’s a high level summary of its overall blog health:

Healthy blog!

The Blog-Health-o-Meter™ reads This blog is on fire!.

Crunchy numbers

Featured image

The average container ship can carry about 4,500 containers. This blog was viewed about 20,000 times in 2010. If each view were a shipping container, your blog would have filled about 4 fully loaded ships.

 

In 2010, there were 8 new posts, growing the total archive of this blog to 32 posts. There were 19 pictures uploaded, taking up a total of 483kb. That’s about 2 pictures per month.

The busiest day of the year was February 5th with 153 views. The most popular post that day was Software Project Failure.

Where did they come from?

The top referring sites in 2010 were stackoverflow.com, blog.ysatech.com, smartclient.codeplex.com, forums.silverlight.net, and social.msdn.microsoft.com.

Some visitors came searching, mostly for wcf concurrencymode.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

1

Software Project Failure February 2010

2

ClickOnce – Creating Publish Page from msbuild December 2009
11 comments

3

MSBuild and Multiple Environments December 2009
2 comments

4

Silverlight Communication Exception March 2009
17 comments

5

SVN Merge Without Conflicts March 2009
5 comments

WCF in Azure – Lessons Learned

Recently I’ve been trying to get my WCF services working in Azure, this post shares some lessons learned. Some of these lessons will likely be obvious for those of you familiar with hosting WCF services in IIS, however I personally haven’t written web sites for about 5 years so even if my memory served me well (which often it doesn’t) things have changed somewhat.

WebRole vs WorkerRole

Probably one of the first questions you will run into is ‘which role do I choose to run my services in?’

WebRole – Choose this if you want your services hosted in IIS
WorkerRole – Choose this if you want to spin up your own ServiceHost instances.

Before trying to make my services work in Azure I was hosting them in a simple Console app and was therefore creating my own ServiceHost instances. So at first I tried getting my existing code working in a WorkerRole.

I could make it work in the local devfabric but I couldn’t make it work in the cloud, the worker role seemed to start ok but the endpoints were not being exposed publicly, or maybe they failed to be created?

I’m still not sure why because I changed tactics and changed my services run in IIS. The main reason for this is I figure running the services in IIS will be more robust and will provide better reporting.

My change of tactics brought on the next problem…

Getting WebRole Working with Service implementation in Different Project

My service implementation code was in it’s own separate project, the service contracts were in another project as well. All WCF in Azure examples I found had the service, the contract and .svc in the WebRole. I wanted to keep my existing structure and it turned out to be very easy to do.

All that was needed was to put some .svc files (one for each endpoint) into the WebRole project and voila, endpoints available in the devfabric and in the cloud.

Contents of the .svc files….

<%@ ServiceHost Language="C#" Debug="true" Service="ServiceNamespace.ServiceName" %>

Easy!

Note that the .svc files must be in the root directory of the WebRole project as Azure does not have a concept of Virtual Directories.

Ports

I still need to more play time to understand how this works more clearly. However what I did find is that if when I changed the port from 80 to 81 my endpoints were no longer visible in the cloud (they were visible in the devfabric).

Cloud Cover – Channel 9

I’d say this is an invaluable resource, lots of useful Azure info for developers….

Making Azure Services Available to Silverlight Clients

This is another thing that turned out to be quite simple.

To make any WCF service available to an SL client you must have a crossdomain.xml file. Normally you need to put this file in the IIS root directory, the way to do this for Azure is to simply drop the file into the root directory of the WebRole project.

Easy! Though I’d suggest host the services in ‘non-azure’ IIS and get it all working before switching to Azure. The crossdomain.xml thingy can be tricky, I’ve posted about wrestling with crossdomain.xml before.

Service Constructor

My web services were using an IoC container for various things and so the constructor of each service took a container. For an IIS hosted service you need to have a parameterless constructor so I had to make some semantic changes to the code to allow for this.

Summary

It took a bit of trial and error to get things working but overall I’d say most of the challenges I faced were more about my lack of knowledge rather than any real problems with Azure. I like it….and am looking forward to learning more :-)

XAML Attribute Formatting

Is it just me or does the default setting for the XAML editor in Visual Studio make working with XAML more difficult?

If you want to make your life easier, change the Tools->Options setting to ‘Position each attribute on a separate line’.

What difference does it make? As you can see, below are two text boxes with all of the same properties set….

No wait, you can only see the properties of one, to see the properties of the other one you need to scroll horizontally, no problem I’ll just use my mouses horizontal scroll wheel….

No wait, it doesn’t have one of those, only a vertical scroller. Damn those mouse makers!

In C# it is possible to create a new object and set all of its properties all on one mega long horizontal line, but we don’t. Why? Because the code is more readable when you break it up vertically over a few lines. You can see all of the properties at once and you usually don’t need to scroll…and if you do you have a vertical scroll wheel on the mouse :-)

ClickOnce Master Build

Previously I have posted about how to publish a ClickOnce release for multiple environments. Whilst this works well there are two reasons (that I can think of) why it is not appropriate or good enough.

First, you or your manager may be a bit of a purist in terms of releasing to Production ‘exactly’ what has been tested, specifically the exact same compiled assemblies.

Second, you may need to create a release for several different customers who each have several of their own ‘environments’ but you do not want to provide them with your source code.

Third…both of the above. :-)

When you ‘Publish’ with MsBuild it requires the source code and it recompiles your assemblies. So how can we easily create releases for multiple environments but only Publish once? This posting offers one solution which has worked for me in the past, it’s not a complete ‘how to’ guide I just wanted to cover the idea and problems I ran into along the way.

Master Build Environment

Make sure you take a look at my previous posting as this process builds on those ideas.

One of the final steps of this process is to re-sign the setup.exe, however in order for that final step to work you have to change one of your ClickOnce deployment settings in your .csproj file.

<SignManifests>false</SignManifests>

SignManifests must be false! Otherwise the signtool gets confused when re-signing the setup.exe. Check this out for a bit more info on this known issue.

So the first thing to do is create a BuildEnvironment called Master. The configuration details of this (web service addresses or database connnection strings etc) should point to nowhere, i.e. if someone deployed this build then it should not work. Publishing this is easy just follow the steps defined in the multiple environment posting, the process of creating a release with environment specific information that is tricky.

MsBuild Changes for Master Build Publish

When you publish with ClickOnce it creates a folder ‘Application Files’ where it puts the contents of the release. The space in that file name causes a problem with MAGE when re-signing the app. Now the only way I could see to change this was to cut and paste the _CopyFilesToPublishFolder target from Microsoft.Common.Targets and modifying the _DeploymentApplicationFolderName property in my version to remove the space. I think it is a really BAD idea to change the Microsoft.Common.Targets directly so that option was ruled out immediately and as you can see in the code below, Application Files (version below already has the space removed) is a hardcoded value, not a settable property…therefore even though it is ugly, cut and paste seems the only viable solution.

      <!--
	  This Target code has been cut and paste from Microsoft.Common.Targets.
	  Only ONE thing has been changed.
	  Application Files changed to ApplicationFiles (i.e. the space removed)
	  The only other way of changing that value is to modify Microsoft.Common.Targets
	  which is a really bad idea (would have to ensure this was changed on ALL machines.)

	  The space causes a problem when re-signing the app with MAGE
    ============================================================
                                        _CopyFilesToPublishFolder
    ============================================================
    -->
    <Target
        Name="_CopyFilesToPublishFolder">

        <!-- Compute name of application folder, which includes the assembly name plus formatted application version.
             The application version is formatted to use "_" in place of "." chars (i.e. "1_0_0_0" instead of "1.0.0.0").
             This is done because some servers misinterpret "." as a file extension. -->
        <FormatVersion Version="$(ApplicationVersion)" Revision="$(ApplicationRevision)" FormatType="Path">
            <Output TaskParameter="OutputVersion" PropertyName="_DeploymentApplicationVersionFragment"/>
        </FormatVersion>

        <PropertyGroup>
            <_DeploymentApplicationFolderName>ApplicationFiles\$(AssemblyName)_$(_DeploymentApplicationVersionFragment)</_DeploymentApplicationFolderName>
            <_DeploymentApplicationDir>$(PublishDir)$(_DeploymentApplicationFolderName)\</_DeploymentApplicationDir>
        </PropertyGroup>

        <!-- Copy files to publish folder -->
        <Copy
            SourceFiles=
                "@(_ApplicationManifestFinal);
                @(_DeploymentResolvedManifestEntryPoint);
                @(_DeploymentManifestFiles);
                @(ReferenceComWrappersToCopyLocal);
                @(ResolvedIsolatedComModules);
                @(_DeploymentLooseManifestFile)"
            DestinationFiles=
                "@(_ApplicationManifestFinal->'$(_DeploymentApplicationDir)%(TargetPath)');
                @(_DeploymentManifestEntryPoint->'$(_DeploymentApplicationDir)%(TargetPath)$(_DeploymentFileMappingExtension)');
                @(_DeploymentManifestFiles->'$(_DeploymentApplicationDir)%(TargetPath)$(_DeploymentFileMappingExtension)');
                @(ReferenceComWrappersToCopyLocal->'$(_DeploymentApplicationDir)%(FileName)%(Extension)$(_DeploymentFileMappingExtension)');
                @(ResolvedIsolatedComModules->'$(_DeploymentApplicationDir)%(FileName)%(Extension)$(_DeploymentFileMappingExtension)');
                @(_DeploymentLooseManifestFile->'$(_DeploymentApplicationDir)%(FileName)%(Extension)$(_DeploymentFileMappingExtension)')"
            SkipUnchangedFiles="true"
            OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"/>
        <Copy
            SourceFiles="@(_DeploymentManifestDependencies)"
            DestinationFiles="@(_DeploymentManifestDependencies->'$(_DeploymentApplicationDir)%(TargetPath)$(_DeploymentFileMappingExtension)')"
            SkipUnchangedFiles="true"
            OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
            Condition="'%(_DeploymentManifestDependencies.DependencyType)'=='Install'"/>
        <Copy
            SourceFiles="@(_ReferenceScatterPaths)"
            DestinationFiles="@(_ReferenceScatterPaths->'$(_DeploymentApplicationDir)%(Filename)%(Extension)$(_DeploymentFileMappingExtension)')"
            SkipUnchangedFiles="true"
            OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"/>
        <FormatUrl InputUrl="$(_DeploymentApplicationUrl)">
            <Output TaskParameter="OutputUrl" PropertyName="_DeploymentFormattedApplicationUrl"/>
        </FormatUrl>
        <FormatUrl InputUrl="$(_DeploymentComponentsUrl)">
            <Output TaskParameter="OutputUrl" PropertyName="_DeploymentFormattedComponentsUrl"/>
        </FormatUrl>
    </Target>

Note that the space only causes a problem with MAGE.exe, it does not cause a problem with MAGEUI.exe! MAGEUI.exe happily re-signs the release even with the space in the directory name. We cannot use MAGEUI.exe however because this process needs to be automated, if you use MAGEUI.exe someone has to run it and manually set the values…not good enough.

It took a while to figure out that the space in the dir name was causing a problem so I hope this saves someone else some time…

Creating a Release

Ok so now we have a published build that no one can use, the first place we want to release to will be QA, then from there UAT so what we need now is an automated way of creating a release configured for these environments.

Creating a release should be as easy as publishing a build, it should be a one step process.

Create a batch file ‘createRelease.bat’ which performs all of the necessary steps. Those steps are:

  1. Create or clean the ‘working’ directory.
  2. Copy the master build into the working directory.
  3. Run the MsBuild steps.
  4. Create a zip file of the new release. (not necessary just makes it a little easier to move the releases around)

Run the MsBuild Steps

Add a new target in your customized.targets file, ModifyMasterBuildForEnvironment.

<Target Name="ModifyMasterBuildForEnvironment" DependsOnTargets="SetPropertyValues; ConfigureForEnvironment; RecreateManifests" />

SetPropertyValues

There are a few properties which need to be set for the resigning process.

	<Target Name="SetPropertyValues" >
		<Message Text="FullReleasePath $(FullReleasePath)" />

		<MSBuild.ExtensionPack.Framework.TextString TaskAction="Replace" OldString="$(ApplicationVersion)" OldValue="." NewValue="_">
			<Output PropertyName="buildX" TaskParameter="NewString"/>
		</MSBuild.ExtensionPack.Framework.TextString>

		<PropertyGroup>
			<WorkingDirectory>WorkingDirectory</WorkingDirectory>
			<FullReleasePath>$(WorkingDirectory)\ApplicationFiles\$(AssemblyName)_$(buildX)</FullReleasePath>
			<ApplicationManifestName>$(AssemblyName).exe.manifest</ApplicationManifestName>
			<ApplicationManifestPath>$(FullReleasePath)\$(ApplicationManifestName)</ApplicationManifestPath>
			<DeploymentManifestPath>$(WorkingDirectory)\$(DeploymentManifestName)</DeploymentManifestPath>
			<CertFileName>cert\pmsKey.pfx</CertFileName>
			<ProviderFullUrl>$(ProviderBaseUrl)$(DeploymentManifestName)</ProviderFullUrl>
		</PropertyGroup>
	</Target>

ConfigureForEnvironment

Add your own target(s) here that does the work to configure the release for a given environment. e.g. update the config file setting database connection strings or web service urls.

RecreateManifests

Now that you have modified the published release it is no longer a valid ClickOnce release. Try to install it and it will fail. What you need to do is update and re-sign the manifests (the Application Manifest and the Deployment Manifest). You also need to update and resign the Setup.exe!

	<Target Name="RecreateManifests">

		<!-- Application manifest update and resign -->
		<Message Text="Removing .deploy suffix..." />
		<RenameFiles DirectoryPath="$(MSBuildProjectDirectory)\$(FullReleasePath)" RemoveSuffix=".deploy" IncludeSubDirectories="True" />
		<Message Text="Updating application manifest" />
		<Exec Command="mage -Update $(ApplicationManifestPath) -FromDirectory $(FullReleasePath) -ToFile $(ApplicationManifestPath)" />
		<Message Text="Adding .deploy suffix..." />
		<RenameFiles DirectoryPath="$(MSBuildProjectDirectory)\$(FullReleasePath)" AppendSuffix=".deploy" IncludeSubDirectories="True" ExcludedFiles="$(ApplicationManifestName)" />
		<Message Text="ApplicationManifestPath $(ApplicationManifestPath)" />
		<Exec Command="mage -Sign $(ApplicationManifestPath) -CertFile $(CertFileName)" />

		<!-- Deployment manifest update and resign -->
		<Message Text="Updating Deployment manifest (ProviderFullUrl=$(ProviderFullUrl))" />
		<Exec Command="mage -Update $(DeploymentManifestPath) -AppManifest $(ApplicationManifestPath) -ProviderUrl $(ProviderFullUrl)" />
		<Message Text="DeploymentManifestPath $(DeploymentManifestPath)" />
		<Exec Command="mage -Sign $(DeploymentManifestPath) -CertFile $(CertFileName) " />

		<!-- Setup.exe update and resign -->
		<Message Text="Updating $(WorkingDirectory)\Setup.exe..." />
		<Exec Command="$(WorkingDirectory)\Setup.exe /url=$(ProviderBaseUrl)" />
		<Message Text="Signing $(WorkingDirectory)\Setup.exe..." />
		<Exec Command="signtool sign /v /f $(CertFileName) $(WorkingDirectory)\Setup.exe" />

	</Target>

Note that before Updating the Application Manifest we have to remove the .deploy extension from the file names, then add the .deploy extension back onto the file names before Signing. (if you have configured ClickOnce to NOT append .deploy then obviously just remove these steps).

RenameFiles Task

To get the renaming behaviour I wanted I had to create my own MsBuild task. You will need to do the same, it’s a little crude as I had limited time but this is the simple little task I created….

    public class RenameFiles : Task
    {
        public override bool Execute()
        {
            var directory = new DirectoryInfo(DirectoryPath);
            if(directory == null)
                throw new ApplicationException(string.Format("Directory path is invalid: {0}", DirectoryPath));
            Console.WriteLine(string.Format("DirectoryPath:{0}", DirectoryPath));

            ProcessDirectory(directory);

            return true;
        }

        private void ProcessDirectory(DirectoryInfo directory)
        {
            RenameAllFiles(directory);
            if(! IncludeSubDirectories)
                return;

            var subs = directory.GetDirectories();
            foreach (DirectoryInfo sub in subs)
                ProcessDirectory(sub);
        }

        private void RenameAllFiles(DirectoryInfo directory)
        {
            var files = directory.GetFiles();

            if(files.Length == 0)
                return;

            foreach (FileInfo file in files)
            {
                string fileName = file.Name;
                if (ExcludedFileList.Exists(name => name.Equals(fileName, StringComparison.InvariantCultureIgnoreCase)))
                {
                    Console.WriteLine(string.Format("{0} is excluded from renaming.", file.Name));
                    continue;
                }
                Console.WriteLine(string.Format("Renaming {0}", file.Name));
                RemoveExtensions(file);
                AppendExtensions(file);
            }
        }

        private void RemoveExtensions(FileInfo file)
        {
            if (string.IsNullOrEmpty(RemoveSuffix))
                return;
            file.MoveTo(file.FullName.Replace(RemoveSuffix, string.Empty));
        }

        private void AppendExtensions(FileInfo file)
        {
            if (string.IsNullOrEmpty(AppendSuffix))
                return;
            file.MoveTo(string.Format("{0}{1}", file.FullName, AppendSuffix));
        }

        public string DirectoryPath { get; set; }
        public string RemoveSuffix { get; set; }
        public string AppendSuffix { get; set; }
        public string ExcludedFiles { get; set; }
        public bool IncludeSubDirectories { get; set; }

        private List<string> ExcludedFileList
        {
            get
            {
                if (excludedFileList == null)
                    excludedFileList = CommonFunctions.CreateListFromCommaSeparatedString(ExcludedFiles);
                return excludedFileList;
            }
        }
        private List<string> excludedFileList;
    }

Summary

There are a number of other little steps we do to make this release process work here but these are really specific to how we want it to work. What I have outlined above is the core steps involved including some problems you will face and their solutions. I hope this posting helps you get over the most challenging of the hurdles you will face in setting a smooth ‘Master Build’ release process but there is still alot of extra steps you will need to create yourself to get it working end to end.

One thing I would recommend is, once you have a working createRelease.bat (note it doesn’t have to be a batch file, could be an exe or powershell or whatever works best in your situation) then create a createReleaseALL.bat file. In there make n calls to createRelease.bat…one for each environment you need to create releases for. That way if you have a dozen environments you can still create a release for all of them with a one step process :-)

I have had to do this fairly quickly so if something is not clear please let me know and I will see if I can improve the post over time.

Final note – It does take time to set this up and get it working correctly, but it is so worth it! Over time you will save much more time than you spent setting it up, guaranteed!

Additional Resources

A few links which helped me to piece this puzzle together…

ClickOnce with bootstraper setup.exe need to change URL without any build

packaging-a-clickonce-server-deployment

http://wallism.wordpress.com/2009/12/21/msbuild-and-multiple-environments/

Batch File Error

Weird Characters Causing Batch File Error

is not recognized as an internal or external command, operable program or batch file

That’s the error I was getting, weird considering the only thing (after removing everything else) in the file was @ECHO OFF. The ‘weird’ characters are shown in the little snippet below

After a bit of head scratching I opened notepad++, switched on ‘Show all characters’…nothing strange there. I had seen this before but I can never remember….then I did! Remember that is, the problem was the file, when it was created (by visual studio…as a txt file), was created with UTF-8 encoding, needs to be ANSI for batch files (or at least it can’t be UTF-8).

Green indicates ‘good’ encoding for batch files, red bad.

With notepad++ you can see what the encoding is (under Format) and change it. I changed my batch file to ANSI and voila it works!

I’ve blogged it so I can remember next time :-)

Avoid Data Loss – Version Upgrade with ClickOnce

Avoid Data Loss when Upgrading App Version

If you have an occassionally connected smart client, how do you ensure that your clients do not lose any data when they upgrade the application version? This posting has a tip about the UpdateMode property that helps with this question.

We use the Disconnected Service Agent (DSA) to help manage the occassionally connected scenarios. The DSA queues services calls in a local SQL Compact Database, the items in the queue get sent when a network connection is detected.

The Problem Scenario

But there is a problem with the following scenario:

1. User processes data in disco mode. Closes the application.
2. We put a new release on the server.
3. User re-connects to the network then starts the app.
4. ClickOnce advises the user that there is a new releases and asks if they want to upgrade or skip. (at this point there are queued service calls in the local database)

5. User chooses one of the following paths

Skip (good path)

- App starts with the old version
- Data captured in disco mode is sent to the server
- All good! n days later they will be prompted to upgrade their version again (define number of days in your .csproj)

Upgrade (bad path)

- Upgrade is downloaded from the server
- Local version is upgraded, which equates to a new install of the new release (and therefore a new local database)
- App starts with the new version
- Data captured in disco mode is lost, the new version has a new database.

So data loss can be avoided if the user chooses the right path, i.e. skips the upgrade the first time. But the last thing you want to do is rely on the user to make the right choice, if one path leads to data loss it’s really our fault as the developers for making the option available. So how do we take away the option?

ClickOnce UpdateMode

The problem scenario described above will only be an issue if you have set the UpdateMode property to ‘Foreground’.

UpdateMode may either be Foreground or Background. If it is set to Foreground the prompt above will appear when ClickOnce detects a new version and before the application starts. If it is set to Background, if ClickOnce detects a new version it:

  1. Starts the app with the currently installed version
  2. Downloads the new version, ready to go for the next restart.

Importantly what it means is that the first time they reconnect after working in disco mode, the upgrade will automatically be skipped, they’ll login with the old version and the captured data will be sent to the server. Then next time they start the app they will then be prompted to upgrade.

The ‘Background’ mode has the (minor) added advantage of the new version being downloaded in the background. In Foreground mode the user must wait for the download to complete when they click Upgrade.

So unless you have a very strong business case for it using Foreground, I’d recommend always setting the UpdateMode to Background!

Finding a Memory Leak in a .Net Application

Do You Have a Leak?

What do you do when you think your application has a memory leak?

I’m sure there are many options but this posting is about my experience with Red Gates ANTS memory profiler 5.1.

About Memory Management and the GC

There have been books written on this so I’m not going to attempt to explain. However I can’t stress enough how important it is to have at least a basic understanding about how the Garbage Collector works.

To compliment their products Red Gate have created a series of easy to understand on-line video’s about .Net Memory Management nicely broken down into several 5-10 minutes segments. These video’s are FREE so watch them!

If the video’s are not in depth enough for you, any article on the subject by Jeffrey Richter should satisfy your need for detailed explanations. Start with this one.

My Leak

Every time I opened and closed one of my controls, which was getting opened in a modal window, the memory used by the application increased and never went back down again. Which is weird because doesn’t .Net handle all memory management for me?

Do it does not.

So what is going on? The control in question has a dispose method, there’s nothing really tricky going on (no database connections etc) so how do I find out exactly what is causing the leak?

Finding the Leak

Enter ANTS memory profiler. I fire up the profiler, select my .exe and click ‘Start Profiling’.

Easy.

The app starts like normal but now the profiler has it under a microscope. Expect your app to run more slowly and make sure you have plenty of RAM.

I navigate to the screen where I can ‘launch’ the control that is leaking, then click ‘Take Memory Snapshot’.

Then I open and close the modal window say six times then click ‘Take Memory Snapshot’ once more.

Now we have two memory snapshots which we can compare. Stop Profiling for now, two is enough.

Click ‘Class List’, then in the grid click on the ‘Instance Diff (+/-)’ column header to sort by instance differences.

What we can clearly see now is a list of objects that are ‘alive’ in the program. What we care about is the difference in live instances from the first snapshot to the second. The ‘Instance Diff’ column provides this info. Now if we have a leak we should see an extra 6 live instances for our leaky control. So scroll down to the group of +6 items.

There are a couple of things of interest here in the profiler. First is yes we can see an extra six instance of our leaky control. Secondly, notice that ANTS puts your controls in bold. In this case we are looking for a specific problem but if you wanted to check for general memory leaks you can easily use the app for a while then scroll down until you see which of your objects is regularly increasing its instance count.

Ok great, we know it leaks, now what?

Dig a Little Deeper

Highlight the leaky object and click Instance List.

That will then show all 6 instance, just select one then click the ‘Object Retention Graph’ button.

In my case the graph appears empty, until I uncheck the ‘Hide’ box.

Now there’s some useful info…

In fact this tells me that there are several KeyEventHandlers that are stopping the GC from collecting my control.

Cool so back to the code and I add some code to unregister the handlers in the Dispose method.

_previousButton.Click -= PreviousButtonClick;
 _familyNameText.KeyDown -= Navigate;
 _givenNamesText.KeyDown -= Navigate;
...

Rebuild the solution and re-run the profiler. Take the initial snapshot, open close the leaky form half a dozen times, take the second snapshot.

Compare.

Now there are no more live instances of the leaky control! Our memory leak has been resolved.

Very easily!

Summary

It is almost certain that somewhere in your application there is a memory leak. With the right tool finding and resolving these leaks can be really easy, it just takes a little bit of time to familiarise yourself with the way the tool works.

The tools aren’t cheap but for any commercial application they are most definately worth the money. In this posting I used ANTS profiler from Red Gate, I am sure there are other tools out there that also do a great job, I just haven’t had the opportunity to use them. ANTS though is excellent (and no I do not have any association with them…beyond having bought a licence).

Events wired up to their handlers are often the ‘hidden’ cause of leaks. But even those of us who know how to manage memory in .Net often forget to do the right thing for whatever reason…usually time pressures. So I’d recommend explicitly building in some time into your project schedule to go hunting for memory leaks, find them before they go into production and everyone will sleep easier at night :-)

Software Project Failure

A How-To Guide

This is a posting for all software development or project managers who cannot stand their job, detest their company, hate their life or any combination of all three and who are actively seeking revenge against all who have made their life so miserable.

One simple means of getting the revenge you crave, of exacting pain and torment on those around you is to guide your development project to failure and I’m here to tell you some simple little things you can do to your developers that will almost guarantee your failure…

Cast Them Into a Dark Dingy Dungeon

The first thing you need to do to your developers is the worst thing you can do to your developers.

Restrict internet access.

Simply claim to your manager that your team wastes too much time on the internet so a restricted policy is required. Tell them to lock down access to all sites except those authorized by you personally.

Then don’t authorize any except Google…that way your team may still search for answers to their technical question but never get to the actual answers (evil laugh). Of course, tell them you are talking to your management to have this restriction removed.

If that is not possible, simply ensure that your team has the exact same restrictions as the rest of the company, that is usually enough to cause major frustration, especially if they cannot download anything (they like to download all sorts of helpful things).

Gouge Their Eyes Out With a Stick

Whenever a team member asks for an additional monitor or even just a larger monitor, be ready with your new favourite saying (you will be using this alot);

“sorry, there’s nothing in the budget for things like that”.

Larger monitors and especially dual (or triple) monitors often lead to increased productivity and worse, makes the developers job more enjoyable so you must be very vigilant in keeping them restricted to one monitor…preferably  a 14 inch number.

If anyone ever asks you:

“may I bring in my own monitor from home then?”

immediately ask them to pack their things and be on their way. This kind of ‘go the extra mile’ behaviour can be infectious and may allow a thread of work enjoyment to seep into the team which, naturally, is intolerable.

Strap a Giant Ship Anchor to Their Waist

Developers think they are special. They are not. As such they should have the same machines as everyone else in the company. If they say they need a more powerful machine, tell them if they were a half way decent developer they should be able to work with any old machine, even a 386 just like you used to do back in the day…

Ensure any servers you have are maxed to capacity, especially the database server, there are few enjoyments in life that beat watching a developer lose more hair because the database server has run out of disk space…again. Crying, “I have more disk space on my mobile phone than this stupid server…which century are we living in here!”, they will beg for a new machine or at the very least a bigger hard drive. Enjoy the show and once they’ve cleared some space and got things working again, copy a massive file onto the server and watch the spectacle again.

Oh yeah!

Oh and of course if they happen to have a build machine, make sure it is the slowest machine you can find. A slow build is a frustrating build and a frustrating build is easier to get rid of (you want to stop any continuous integration which may be going as this kind of setup takes away ALOT of developer pain).

The beautiful thing about this is you will be seen as a genius by upper management because of the vast amounts of $$$ you appear to be saving them! Secretly laugh at them behind their backs because the reality is very different, though they are saving money on better machines, developers are taking 2-3 times as long to complete their tasks…and developers are expensive…much much more expensive than ‘decent’ machines.

Start planning an amazing holiday, your bonus should be huge this year!

Divide and Conquer

If your team gets together on Monday mornings for a coffee and a chat about the week ahead, it may seem like a good time waster to you but don’t be deceived! Such meetings are great for morale and team building, making your team happier and therefore more productive.

They must be stopped at all costs.

Watch your bonus grow as you put a stop to these ‘meetings’, putting an extra twenty minutes of work back into each developers Monday. Senior management will be talking about your genius.

Bore Them to Tears

Documentation, Documentation, Documentation!

Reams of it!

Any new feature request demands at least a few pages to fully describe the change…and that is just to fix some spelling.

As for the developers. Insist on a tech spec for every piece of work. Then insist that spec must be approved by the team lead…before it goes to the architect for approval…which then must finally be approved by you. Make a point of rejecting at least half.

Ensure the documentation gets done in something like MS Word. The business (and senior management) will be happy with your choice and developers will grumble.

They do complain alot, developers.

They will grumble and tell you that a Wiki would be a better place to document things because it’s easier to search, keep updated, manage change history and ensure that the latest version is easily available to everyone. Indeed they are right but that would involve change and who wants change? Change means risk and you have a bonus you are working towards.

Drive Them Insane

Say things like, ‘ok, to improve productivity we are going to be doing daily SCRUMs’ (developers love SCRUM’s, well good ones do anyway).

But then in each meeting YOU do all the talking, dish out tasks, berate people for taking too long, tell them how poorly the team is perceived by the rest of the company then finish the meeting with a (contrasting) positive and resounding, “OK team, Let’s go GUYS”.

This is so completely not how SCRUM works. Your good developers will point this out…make sure you keep calling the meetings a SCRUM and they will slowly (but very surely) go insane. Eventually they won’t be able to stand it any more and will leave.

Tell senior management the deserters couldn’t handle the new tighter controls you have placed on the project and replace any good developer that leaves with one that is half the salary.

Get an extra bank account, one is not going to be big enough to fit your bonus!

Give Them Enough Rope

WARNING this is pure evil…so read very carefully.

Allow on the fly changes to production servers.

Grant full access to production databases, i.e. so developers can run any kind of script at any time.

Like a spider in a web, you just sit and wait…disaster will happen.

Throw Them to the Lions

If senior management approach you about problems they have been hearing about the system, always direct all blame to the best developer on your team (if he/she was so good the problem should have never arisen).

Deny all suggestions that your team has been doing many extra hours (with no overtime paid) to stay on top of things, suggest that even if that were true, once again, if they were half way decent developers extra hours would not be needed.

You are doing the best you can with the limited abilities which exist in your team.

If you start to feel uncomfortable with how you are treating your staff (what the!), just remember…your bonus.

The Perfect Team

Do all of these things and over time you will find that strangely not all of the developers leave! Some stay on enduring the pain, walking the tightrope over the raging river of suffering. No matter what you do they keep coming back for more and the great thing is this, they do not leave because they are rubbish, they have as much talent at software development as you do at managing projects, are entirely in it for the pay packet and are slowly but surely turning your codebase into a festering cesspool of ick.

So keep up all the good work you have been doing and your team will soon be full of these blundering fools and you will have complete success at total failure!

Authors Comment

I just wanted to be clear here, this is not a rant about a place I have worked and most certainly not about any individual manager. Just a bit of fun exaggerating some of the frustrations we have all experienced as developers.

you need to immediately put a stop to
Follow

Get every new post delivered to your Inbox.