Doodle Graphics

Yay for short working weeks! I started writing this post last week, and then Easter came along and this week's even shorter as we've been given an extra bank holiday so we can all go and watch the Royal Wedding, which will be fun for all concerned I'm sure - hope the weather holds, but it's not looking great at the moment…

Last time we tried Killing off TFS 2005

We'd started the dry run of our TFS Integration process on the test server TFS10. This had started migrating some of the work items, but also had issues with a missing Work Item Type. I then left it running, and went on holiday…

The full series

• Part 1: In which the scene is set.
• Part 2: In which the Hands on Labs are completed.
• Part 3: Where we attempt the migration.
• Part 4: In which conclusions are drawn (this post).
• Part 5: In which errors are noted.

State of the nation

I got back to discover that the server had been restarted at some point during the last week (thanks for the mandatory Windows Updates on the servers chaps ) but after some digging discovered that the process had crashed with an "Out Of Memory" exception. I suppose trying to migrate part of an 11GB TFS instance to another database, on the same server was a bit much to ask. Luckly, TFS10 is a virtual machine, so a quick shut down/re-configure/restart we were back in operation with 6GB of RAM. and I tried again.

I imported the missing Work Item Template to the target project, and then all 2,930 Work Items imported without any problems.

However, I had a number of issues with the version control migration, many of which in the end, I've not been able to resolve:

1. The workspace names are a combination of the server name, the current instance GUID and your Session Names, and this must be less than 64 characters or errors will occur - The work arounds for this is are:
   
   1. Change the WorkSpaceRoot application setting in the MigrationToolServers.config file in the TFS Integration Tools install directory to be as short as possible
   2. Ensure that the "Name" fields of the Left and Right Version Control Session Sources are short but meaningful.
2. Related to 1 - we've got some historical check-ins to folders that resulted in a total path length of over 260 characters, for various reasons TFS can't cope with this, and other than carrying out the steps in 1, there's nothing you can do about it.
3. Regardless of what the tool states, you can edit the Version Control filter pairs after the configuration has been saved to the database if you're happy to edit the XML directly, just remember to leave off the trailing "/" - however this will result in the Discovery and Analysis phases running again.
4. We've already performed some sort of merge from different projects into one of the projects we're trying to merge, and this caused an issue because the TFS Integration Tool couldn't find a workspace mapping for these old, no longer existing projects - I was hopeful that this could be resolved by adding a new filter pair to try and map them in manually, however I've not been able to confirm this completely.
5. Various Null Reference Exceptions stating that the Path could not be null that caused the migration to stop.

So, in the end, I've had to pull the plug on this experiment, and live with a seperate Team Project Collection for the legacy code base - I'll keep telling myself that "It's probably ok, we're not going to be working on them side-by-side" and things like that, but in the end I'm a little sad that it didn't work.

If you've got a fairly simple, or small history (or you're happy to drop a good portion of it), you know that file paths are short enough then I think the Team Foundation Server Integration Platform is a great tool, but as they warn you throughout their documentation, it really should only be used if you absolutely have to, and you can't live with an upgrade (a note on that - For some reason, on the live TFS2010 box, I actually had to restart TfsJobAgent to get the upgrade to complete, as the new Project Collection wasn't created until I restarted it).

Filed under: TFS

Previously on Killing off Team Foundation Server 2005

As you no doubt recall, I've now got the hang of simple migration projects with the Team Foundation Server Integration Tools. I'm therefore trying out the migration of the actual data (from a backup; remember the /confirmed switch we added to the Upgrade command in Part 1?).

The full series

• Part 1: In which the scene is set.
• Part 2: In which the Hands on Labs are completed.
• Part 3: Where we attempt the migration (this post).
• Part 4: In which conclusions are drawn.
• Part 5: In which errors are noted.

Part 3: Dry run with upgraded project

The first step was to Download the Process Templates (in Visual Studio, go to Team | Team Project Collection Settings | Process Template Manager) that had been used from the MissouriImported Project Collection, and Upload them into the DefaultCollection Project Collection.

I then tried to create a new Project using the Process Template I'd imported, but it failed on the "Populate Reports" task. As is increasingly the case, the logs are your friend. The cause was that the Reporting Services Data Source names have changed since TFS 2005 - they were originally called TfsOlapReportDS and TFSReportDS - however in TFS 2010 they are now Tfs2010OlapReportDS and TFS2010ReportDS respectively.

This was fairly simple to fix - navigate to the Reports folder in the downloaded process template, open the ReportsTasks.markup file and do a Find and Replace on "/TfsOlapReportDS" and "/TfsReportDS" for the 2010 versions - note the leading slash in there - you want to be replacing the "name" attribute, while leaving the "dnsname" attribute as is - this will allow TFS to find the incorrect version, and update it with the correct one. Save the file, and re-upload the process template. To be safe, I then restarted Visual Studio.

With the updated Process Template in place, I successfully created a couple of new projects using it, and decided to try and migrate the larger of the two projects.

It all started so well. I set up the TFS Integration configurations as per my learnings from the Hands On Lab (HOL), saved it, and started the migration. The tool soon found 1,699 Work Item Change groups (work items, changes/updates and links - there are about 600 work items in that project, we weren't really using it properly), analysed them, and started migrating them - that's when the trouble started.

It turns out that someone added a new Work Item Type to the project without updating the Process Template - so there are just over 200 work items of type "Defect" that have no where to go, which is odd, as the (admittedly out of date) HOL had implied that "the default is to move all work item types" and this is only disabled if you define a mapping - I've not defined any mappings, but clearly it's not going to create it on it's own. It's all the more tedious as the preferred fix is to create a mapping rule, and resolve the issue that way - however it appears at the moment that I have to do this one conflict at a time, and there are currently over 400 and counting so it could take some time. That said, the whole purpose of this test is to iron out these sorts of issues, so what I'm going to do for next time is import the missing Work Item Type.

At the time of me writing this (18:13), the process has been running for about 1.75 hours, it's migrated 1,236 of 1,699 work item "change groups" (after discovering and analysing them) - which is about what I'd expected and discovered over 12,000 Version Control "change groups" - the change set ID's on that server go up to 53,000, however they aren't all in the projects I'm migrating - Sidekicks seems to think it's around 33,000

There now follows a short intermission.

Filed under: TFS

Following on from where we left off in Part 1.

Recap

We've upgraded a copy of the Team Foundation Server 2005 databases to TFS 2010, and connected to it using Visual Studio 2010 to confirm I was happy that the source code, history and work items had indeed been migrated successfully.

I've also installed a copy of VS2003, and the latest version of the MSSCCI Provider to confirm that I'm happy with it in that scenario, as well as confirmed that users that aren't project administrators can also see the history, etc (we'd had issues with this on the VSS - TFS upgrade).

The full series

• Part 1: In which the scene is set.
• Part 2: In which the Hands on Labs are completed (this post).
• Part 3: Where we attempt the migration.
• Part 4: In which conclusions are drawn.
• Part 5: In which errors are noted.

Part 2: Merging Projects

Now comes the more complicated part: Merging the projects from the upgraded Team Project Collection (MissouriImported). This is made more complicated by the fact that I didn't come up with a sensible name for the new instance of a project in the original TFS 2010 instance (Newyork) - so I now have two projects with the same name in different collections that I'd like to have in the same collection, but with different names.

After I'd installed the TFS Integration Tools (note to self: When it asks for the database server, and defaults to a SQL Express instance, if you're connecting to the default SQL Server instance on a box, you don't want to add the instance name - so just TFS10 and not TFS10\MSSQLSERVER ) I then tried going through the Getting Started with a TFS to TFS Migration Hands On Lab (HOL) that comes as part of the guidance - I strongly recommend that you do as well to familiarise yourself with the process, however it's not all plain sailing.

Running through the Hands On Lab

Basic Scenario

This worked out quite nicely, but then it's designed to minimise and work around all potential conflicts - as I'm not using the "Visual Studio ALM Rangers HOL base image" mentioned in step 4 of the HOL, I obviosuly had to set up some projects - this I did in much the same way the HOL described. Note that the HOL is a little out of date compared to the tools:

• When you configure the Version Control and Work Item Tracking sessions, the screen-shots imply that the name of each session is of the form "Project Name (ServerName)", but the tool appears to use "ServerName (Session Type)", so instead of getting "TP-A (tfs10)" I got "tfs10 (VC)" for both the Left and Right sources. You can (and probably should, especally if you're planning on doing a two-way sync) change these names to something more useful.
• When the HOL asks you to Save and Close you should Save to Database.

As I'm planing on running a "One-way migration" I skipped step 7.3, 7.5 & 7.4 [sic].

Advanced Scenario

This is where the issues really started. As I'm going to be importing into a project with the same team process, I wasn't too fussed about changing the WIT types, however it's good to read through it, as this is the easiest way to move projects to a different process template - something I may be doing at a later point for some of our old projects.

I basically hit an issue with the Version Control (VC) migration, in that the HOL assumes that the VC conflict you're going to see with the BuildProcessTemplates folders will result in the error "A content conflict has been detected…" with an option to use the "Always take the source system's changes" resolution, however this wasn't available to me, as the error I received was "A namespace conflict has been detected between the source and the target system…" which meant I had to provide "merged changesets for the source and target system (requires a manual merge)". This is rather tedious as the files themselves are identical when compared beyond their check-in timestamps, and I couldn't work out what it really wanted - providing some rather arbitrary changeset ids resulted in no source code at all being migrated.

In the end I worked around it by effectively ignoring the BuildProcessTemplates folder:

In the Version Control Session definition, I added a new Paths rule, set it to "Cloaked" for the BuildProcessTemplates paths:

With this rule in place, the duplicate BuildProcessTemplates folder and XAML files were ignored by the migration, and it completed smoothly.

As an example, here's the work items list for a check-in on Project1:

As you can hopefully see, this check-in has a comment, and is associated with a closed task work item (2).

Here's the equivalent for the migrated project:

This is associated with the migrated task (14), and the check-in comment has been carried across, and had the details of where it came from, along with the original check-in details (changeset ID and timestamp).

I'm therefore fairly optimistic now for the dry run of the main project migration, even though it's got over 3GB of "stuff" under source control (I use the word "Source" advisedly there, there's a lot of other things in there, from binaries to SDO packages and beyond).

I'll let you know how it goes.

Filed under: TFS

I'm in the process of upgrading an instance of TFS2005 to 2010, with the added complication of wanting to combine the projects into an existing TFS 2010 Project Collection.

The full series

• Part 1: In which the scene is set (this post).
• Part 2: In which the Hands on Labs are completed.
• Part 3: Where we attempt the migration.
• Part 4: In which conclusions are drawn.
• Part 5: In which errors are noted.

What am I doing?

As I've had to hunt around on the internet for various steps in the process, I thought I'd try and pull it all together in one place, with the caveat of "it worked for me on my servers", your mileage may vary (considerably) and I take no responsibility for any loss of data, source code, etc. that you may experience (however, I do have to take full responsibility for any loss of data, source code, etc that my team experiences ).

To set the scene then, we have the following servers involved in this process, because I want to minimise the downtime that the teams experience while I'm doing this, and the upgrade can't be undone:

• Missouri - Legacy codebase running on TFS 2005 (upgraded from VSS), using a local instance of SQL Server 2005.
• Newyork - Current codebase running on TFS 2010 (upgraded from 2008), using a remote SQL Server 2008 instance.
• TFS10 - Clean install of TFS 2010 using a local instance of SQL Server 2008 for me to play around on.

I found the following resources quite useful as well:

• TFS Install and Administration guides
  The main help file that takes you through installing, upgrading and administering your TFS instance. The sections that I found most usefull were:
  
  • Checklist: Upgrade Team Foundation Server By Using More Than One Server
  • How to: Upgrade Team Foundation Server Using the Team Foundation Server Configuration Tool
• Visual Studio 2010 TFS Upgrade Guide
  This contains supplemental documentation for upgrading previous versions of TFS to TFS 2010.
• How to: Move Your Team Foundation Server from One Hardware Configuration to Another
  TFS 2005 version of the documentation on moving TFS from one server to another.
• How to: Move Your Team Foundation Server from One Hardware Configuration to Another
  TFS 2010 version of the documentation on moving TFS from one server to another.
• TFS Integration Platform
  A set of tools and documents to aid in moving bits of TFS around, especially the Migration Guidance document from the "Downloads" tab - to be honest, I'm using the TFS Integration installer from the VS Extensions site.

Part 1: Migration Upgrade from TFS 2005 to TFS 2010

These are the steps I took to migrate our TFS 2005 instance (Missouri) to the clean TFS 2010 instance (TFS10):

1. Copy the latest backup files from Missouri to TFS10 of the following databases:
   
   • TFSActivityLogging
   • TFSBuild
   • TFSIntegration
   • TFSVersionControl
   • TFSWarehouse
   • TFSWorkItemTracking
   • TFSWorkItemTrackingAttachments
2. Stop the TFS services using the command line tool found in Tools folder of your TFS installation:
   TFSServiceControl quiesce
3. Create new databases on TFS10 with the same names as the backups, and restore the backups into these - Note to self: when restoring a database over an existing one, make sure you change the file path on the options pane to the correct destination file path.
4. Because my friendly SysAdmin had kindly installed TFS for me, opening the TFS Administration Console didn't give me the option of starting the TFS Configuration Tool that's mentioned in the upgrade steps, however, you can run the upgrade process from the command line (again, in the TFS Tools folder):
   tfsconfig import /sqlInstance:tfs10 /collectionName:MissouriImported /confirmed
   Obviously, your SQL Instance and Collection Name will be different, but you get the point.
   
   This will take a while to run - anecdotal reports are "between 30 and 180 hours", depending on the size of your repository, and the amount of processing resources available. Mine took around 15 minutes on a 10GB version control database and a 76MB work item tracking database.
5. If all goes well, the command window should report that it's completed "Collection creation step 208 of 208", and that "The Upgrade operation on the ApplicationTier feature has completed. There were 0 errors and 0 warnings.". I can't really advise on what to do if either of those don't happen for you, other than to say "Take a look in the log file", which has quite a lot of detail in it.
6. Restart the TFS services:
   TFSServiceControl unquiesce
7. Connect an instance of Visual Studio at the new Project Collection and verify that things look ok. You might also want to try connecting VS2003 (yes, some of the code in there is indeed ASP.NET 1.1, which pretty much requires VS2003, some of it is older still, and is classic ASP/VBScript )

More to follow shortly.

Filed under: TFS

Update: There's now a "How-To" guide to Doodle Grouper 28/11/2011

A quick update to say that I've recently pushed out an updated version of my Seesmic Desktop plugin Doodle Grouper - note that the marketplace version is a bit old, but is the quickest way to install it, and it will auto-update to the latest version.

Key changes since the last update:

1. Much smaller download - easy mistake to make, I'd left the references as "Copy Local", seeing as I'm either referencing core framework libraries, or the main Seesmic SDK libraries, there's no need for me to include a copy of them.
2. Controllable memory footprint - one of the issues with this plugin is that it delivers updates to new timelines, which seems to have a fairly impressive impact on memory usage in the SLLauncher process. I've added a plug-in wide setting that will limit the app to only keeping a certain number of updates in each group.
3. Controllable logging - as I've added more logging to the plug-in, which no-one else should care about, it makes sense to disable this by default, obviously now that Seesmic uses a database file for persistent storage, it means I can't hack around in an XML file to change this, so I've had to add a check-box to my settings page, and I guess I'll be in trouble if I break the settings template at the same time as something else.

Enjoy.

Filed under: Plug-ins, Seesmic, Silverlight