rollbacks

Jun 8, 2012 at 2:22 AM

Hello,

I have been thinking about how to perform rollbacks.  There doesn't seem to be a good way to do this.  Since I'm using vsdbcmd and msdeploy as the actual deployment tool, I was thinking to create a presync script to generate backups.  Then since TFS Deployer simply calls a script based on a transition, I could then run the same deployment but with a postsync script which would restore the backup.

Anyone ever implement rollbacks?  Any help would be great.

Dave

Jun 8, 2012 at 2:31 AM
Dave,

Since TFS deployer uses PowerShell script, in my company, I use the PowerShell to call MS deploy for the deployment.
You could avoid log in many servers using RDP, I put the script the back up automatically and manage the IIS app pool, or windows services like when the script runs, it does backup --> stop app pool or windows services --> call MS deploy to deploy --> start app pool or the services ---> compare target and source. Since MsDeploy creates the log file, you can save the log file for any track records I need. I am sure you can create the function to rollback in the powershell script in the same way. MSDeploy and TFSDeploy saves lots of time of my deployment time.

John.




From: [email removed]
To: [email removed]
Date: Thu, 7 Jun 2012 18:22:15 -0700
Subject: rollbacks [tfsdeployer:358798]

From: dpachla
Hello,
I have been thinking about how to perform rollbacks. There doesn't seem to be a good way to do this. Since I'm using vsdbcmd and msdeploy as the actual deployment tool, I was thinking to create a presync script to generate backups. Then since TFS Deployer simply calls a script based on a transition, I could then run the same deployment but with a postsync script which would restore the backup.
Anyone ever implement rollbacks? Any help would be great.
Dave
Read the full discussion online.
To add a post to this discussion, reply to this email (tfsdeployer@discussions.codeplex.com)
To start a new discussion for this project, email tfsdeployer@discussions.codeplex.com
You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.
Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com
Jun 8, 2012 at 1:44 PM

Hi John,

Thanks for getting back with good information.  That's exactly what I'm looking to do.  You say you use Powershell to call MSDeploy.  I have not been able to successfully get Powershell to call MSDeploy due to the poor handling of quoted strings in Powershell.  It just will not work if I try to call it directly nor if I wrap it in a string and launch it as an argument for cmd /c "" from Powershell.  I have to have the Powershell script call a cmd file which then in turn calls vsdbcmd and msdeploy.  The rollbacks could then indeed be set up in a similar way.

Thanks for the help.

Jun 8, 2012 at 4:58 PM
Dave,

It's in the discussion forum section that I got help from Jason too. There is an example. If u can't find it, let me know I will send my script.

Sent from my iPhone

On Jun 8, 2012, at 5:44 AM, "dpachla" <notifications@codeplex.com> wrote:

From: dpachla

Hi John,

Thanks for getting back with good information. That's exactly what I'm looking to do. You say you use Powershell to call MSDeploy. I have not been able to successfully get Powershell to call MSDeploy due to the poor handling of quoted strings in Powershell. It just will not work if I try to call it directly nor if I wrap it in a string and launch it as an argument for cmd /c "" from Powershell. I have to have the Powershell script call a cmd file which then in turn calls vsdbcmd and msdeploy. The rollbacks could then indeed be set up in a similar way.

Thanks for the help.

Jun 8, 2012 at 5:59 PM
In the powershell, you could call the msploy like below.


$MsDeployDir = (Get-ItemProperty -Path HKLM:\SOFTWARE\Microsoft\.NETFramework\AssemblyFolders\MSDeploy).'(default)'

# join MSdeploy.exe and it's physical location - C:\Program Files\IIS\Microsoft Web Deploy V2\ + msdeploy.exe
$MsDeployExe = $MsDeployDir | Join-Path -ChildPath msdeploy.exe

$BackupLogfile = "$ServerName1-$PublishedPath"+"-PROJECT-"+"$CreatedDate-Backup.xml"

cmd.exe /c $('" "{0}" -verb:sync -source:contentPath="{1}" -dest:contentPath="{2}" {3} -xml -debug > Y:\DeploymentsHistory\"{4}" "' -f`
$MSDeployExe, $applicationUNCPath, $backupto, $exclude, $BackupLogfile)



John Roh





From: [email removed]
To: [email removed]
Date: Fri, 8 Jun 2012 05:44:45 -0700
Subject: Re: rollbacks [tfsdeployer:358798]

From: dpachla
Hi John,
Thanks for getting back with good information. That's exactly what I'm looking to do. You say you use Powershell to call MSDeploy. I have not been able to successfully get Powershell to call MSDeploy due to the poor handling of quoted strings in Powershell. It just will not work if I try to call it directly nor if I wrap it in a string and launch it as an argument for cmd /c "" from Powershell. I have to have the Powershell script call a cmd file which then in turn calls vsdbcmd and msdeploy. The rollbacks could then indeed be set up in a similar way.
Thanks for the help.
Read the full discussion online.
To add a post to this discussion, reply to this email (tfsdeployer@discussions.codeplex.com)
To start a new discussion for this project, email tfsdeployer@discussions.codeplex.com
You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.
Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com
Jun 8, 2012 at 6:24 PM

John,

Excellent!  I'll have to try this out and see if I can find the previous discussion.  Thanks for the help!

Dave

Jun 8, 2012 at 6:51 PM
Good and let me know if u have any Qs.

Sent from my iPhone

On Jun 8, 2012, at 10:24 AM, "dpachla" <notifications@codeplex.com> wrote:

From: dpachla

John,

Excellent! I'll have to try this out and see if I can find the previous discussion. Thanks for the help!

Dave

Jun 8, 2012 at 7:25 PM

I did find the discussion you mentioned earlier.  Thanks.

Now, I'm having problems getting TFS Deployer installed/configured.  I followed the steps on the Installing TFS Deployer link.  Then ran Test-TfsDeployerInstall.ps1, which succeed only after adding the port number and a "/" at the end of the base address.  So I'm now at the same place as the person in this discussion.

http://tfsdeployer.codeplex.com/discussions/251516

One thing that I haven't seen is any information as to what is supposed to happen or where the downloaded files go.  Running tfsdeployer.exe -d says the following:

C:\Program Files\TfsDeployer>TfsDeployer.exe -d
NotificationServiceHost: Subscribing to Notification event at address http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/event/BuildStatusChangeEvent
NotificationServiceHost: Subscribed to Notification event at address http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/event/BuildStatusChangeEvent as id 77
Hit Enter to stop the service

Then my mappings file is such:

<DeploymentMappings xmlns="http://www.readify.net/TfsDeployer/DeploymentMappings20100214">
  <Mapping BuildDefinitionPattern="BuildTest"
           NotificationAddress="me@mydomain.com"
           Computer="mymachine.mydomain.com"
           OriginalQuality="*"
           NewQuality="Under Investigation"
           RunnerType="PowerShell"
           script="Test.ps1"/>
</DeploymentMappings>

Now, when I change the build quality for BuildTest to Under Investigation, nothing happens at all.  I imagine that something should be happening in the console app, or some files should be getting downloaded somewhere, or I should receive an email.

Any ideas?

Jun 8, 2012 at 7:29 PM
Before you start modify your script, please run the example one to see if that works first.
Also, in your server where TFS Deployer is installed, you can check the even viewer to see any typo path or something.
Right now, I'm swap by the requests, I can take a look when the traffic gets slower down.
I think you're almost there :)

John Roh





From: [email removed]
To: [email removed]
Date: Fri, 8 Jun 2012 11:25:55 -0700
Subject: Re: rollbacks [tfsdeployer:358798]

From: dpachla
I did find the discussion you mentioned earlier. Thanks.
Now, I'm having problems getting TFS Deployer installed/configured. I followed the steps on the Installing TFS Deployer link. Then ran Test-TfsDeployerInstall.ps1, which succeed only after adding the port number and a "/" at the end of the base address. So I'm now at the same place as the person in this discussion.
http://tfsdeployer.codeplex.com/discussions/251516
One thing that I haven't seen is any information as to what is supposed to happen or where the downloaded files go. Running tfsdeployer.exe -d says the following:

C:\Program Files\TfsDeployer>TfsDeployer.exe -d
NotificationServiceHost: Subscribing to Notification event at address http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/event/BuildStatusChangeEvent
NotificationServiceHost: Subscribed to Notification event at address http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/event/BuildStatusChangeEvent as id 77
Hit Enter to stop the service

Then my mappings file is such:

<DeploymentMappings xmlns="http://www.readify.net/TfsDeployer/DeploymentMappings20100214">
  <Mapping BuildDefinitionPattern="BuildTest"
           NotificationAddress="me@mydomain.com"
           Computer="mymachine.mydomain.com"
           OriginalQuality="*"
           NewQuality="Under Investigation"
           RunnerType="PowerShell"
           script="Test.ps1"/>
</DeploymentMappings>

Now, when I change the build quality for BuildTest to Under Investigation, nothing happens at all. I imagine that something should be happening in the console app, or some files should be getting downloaded somewhere, or I should receive an email.
Any ideas?
Read the full discussion online.
To add a post to this discussion, reply to this email (tfsdeployer@discussions.codeplex.com)
To start a new discussion for this project, email tfsdeployer@discussions.codeplex.com
You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.
Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com
Jun 8, 2012 at 8:28 PM

Yes, I must be very close!  There's nothing in the Windows Logs ->Application other than the subscribing and subscribed messages.  Here is the config:

<?xml version="1.0"?>
<configuration>
  <configSections>
    <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
      <section name="TfsDeployer.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
    </sectionGroup>
  </configSections>
  <applicationSettings>
    <TfsDeployer.Properties.Settings>
      <setting name="KeyFile" serializeAs="String">
        <value />
      </setting>
      <setting name="SignDeploymentMappingFile" serializeAs="String">
        <value>False</value>
      </setting>
      <setting name="TfsUserName" serializeAs="String">
        <value />
      </setting>
      <setting name="TfsDomain" serializeAs="String">
        <value />
      </setting>
      <setting name="TfsPassword" serializeAs="String">
        <value />
      </setting>
      <setting name="TeamProjectCollectionUri" serializeAs="String">
        <value>http://tfs:8080/tfs/MyProjCollection/</value>
      </setting>
      <setting name="BaseAddress" serializeAs="String">
        <value>http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/</value>
      </setting>
    </TfsDeployer.Properties.Settings>
  </applicationSettings>
  <system.net>
    <mailSettings>
      <!-- TODO update mail settings for each environment -->
      <smtp deliveryMethod="Network" from="tfs@mydomain.com" >
        <network host="netv" defaultCredentials="true" />
      </smtp>
    </mailSettings>
  </system.net>

  <system.diagnostics>
    <!-- Uncomment the switches section to increase trace verbosity -->
    
    <switches>
      <add name="Readify.Useful.TeamFoundation.Common" value="Verbose" />
      <add name="TfsDeployer" value="Verbose" />
      <add name="TfsDeployer.DeployAgent.DeploymentHostUI" value="Verbose" />
    </switches>
    <trace autoflush="true" />
  </system.diagnostics>

</configuration>

Jun 8, 2012 at 8:37 PM
can you call this service from your browser?
http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/

Rather you use the fully qualify domain name, you can use IP address to call.

<setting name="BaseAddress" serializeAs="String">
<value>http://xxx.xxx.xxx.xxx/Temporary_Listen_Addresses/TfsDeployer</value>
</setting>

Once the setting is correctly done, you should be able to call the Powershell Script that it comes with the TFSDeployer.
You will see something on the event viewer otherwise it is more likely the config issue.

Before you try to call your powershell, please use the example so that you can troubleshooting faster.

Good luck.

John Roh





From: [email removed]
To: [email removed]
Date: Fri, 8 Jun 2012 12:28:05 -0700
Subject: Re: rollbacks [tfsdeployer:358798]

From: dpachla
Yes, I must be very close! There's nothing in the Windows Logs ->Application other than the subscribing and subscribed messages. Here is the config:

<?xml version="1.0"?>
<configuration>
  <configSections>
    <sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089">
      <section name="TfsDeployer.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false"/>
    </sectionGroup>
  </configSections>
  <applicationSettings>
    <TfsDeployer.Properties.Settings>
      <setting name="KeyFile" serializeAs="String">
        <value />
      </setting>
      <setting name="SignDeploymentMappingFile" serializeAs="String">
        <value>False</value>
      </setting>
      <setting name="TfsUserName" serializeAs="String">
        <value />
      </setting>
      <setting name="TfsDomain" serializeAs="String">
        <value />
      </setting>
      <setting name="TfsPassword" serializeAs="String">
        <value />
      </setting>
      <setting name="TeamProjectCollectionUri" serializeAs="String">
        <value>http://tfs:8080/tfs/MyProjCollection/</value>
      </setting>
      <setting name="BaseAddress" serializeAs="String">
        <value>http://mymachine.mydomain.com:8881/Temporary_Listen_Addresses/TfsDeployer/</value>
      </setting>
    </TfsDeployer.Properties.Settings>
  </applicationSettings>
  <system.net>
    <mailSettings>
      <!-- TODO update mail settings for each environment -->
      <smtp deliveryMethod="Network" from="tfs@mydomain.com" >
        <network host="netv" defaultCredentials="true" />
      </smtp>
    </mailSettings>
  </system.net>

  <system.diagnostics>
    <!-- Uncomment the switches section to increase trace verbosity -->
    
    <switches>
      <add name="Readify.Useful.TeamFoundation.Common" value="Verbose" />
      <add name="TfsDeployer" value="Verbose" />
      <add name="TfsDeployer.DeployAgent.DeploymentHostUI" value="Verbose" />
    </switches>
    <trace autoflush="true" />
  </system.diagnostics>

</configuration>


Read the full discussion online.
To add a post to this discussion, reply to this email (tfsdeployer@discussions.codeplex.com)
To start a new discussion for this project, email tfsdeployer@discussions.codeplex.com
You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.
Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com
Jun 8, 2012 at 9:22 PM

No matter what I use in the config for the BaseAdress, the URL in the browser returns:

Service Unavailable


HTTP Error 503. The service is unavailable.

 

Why wouldn't the service start?  This is certainly the problem.

Jun 8, 2012 at 10:03 PM
yes, the service needs to be accessible and that's why you can't see any on the event viewer.
If you search further, it asks you to use the IP addresses.
Without resolving the issue, 503 is the service not available, it wouldn't work.

I haven't implemented TFS Deployer yet in my company since I'm waiting couple more build servers to work on.
But you can always deploy from the Binaries folders as long as you deploy it from it when there is no other queue gets inserted.

John Roh





From: [email removed]
To: [email removed]
Date: Fri, 8 Jun 2012 13:22:05 -0700
Subject: Re: rollbacks [tfsdeployer:358798]

From: dpachla
No matter what I use in the config for the BaseAdress, the URL in the browser returns:

Service Unavailable


HTTP Error 503. The service is unavailable.

Why wouldn't the service start? This is certainly the problem.
Read the full discussion online.
To add a post to this discussion, reply to this email (tfsdeployer@discussions.codeplex.com)
To start a new discussion for this project, email tfsdeployer@discussions.codeplex.com
You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.
Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com
Jun 8, 2012 at 10:12 PM

I actually did try to use the IP address as well.  It gives the same message.  Maybe one of the developers can give some more insight.  Thanks for all your help.

Jun 11, 2012 at 2:53 PM

I decided to just download the source code and compile it to try to debug it.  The current source code works fine.  The TfsDeployer release 1.4.0.54837 does not.

Coordinator
Jun 12, 2012 at 1:06 AM

Hi Dave,

Rolling back deployments, regardless of chosen technology, can be a very complex field and varies depending on the nature of the system you are deploying. I can however offer some pieces of advice that I have found to be very useful in my experiences deploying a variety of applications to different kinds of environments.

  1. Every time a deployment fails in such a way that a roll back is required, ensure you follow up and investigate the root cause of why the deployment didn't fail in the previous environment and fix the discrepancy. I.e.: if deployment to UAT succeeds but deployment to Production fails, and it turns out it was due to IIS being configured differently (for example) then change UAT's IIS configuration to be more Production-like so that you are more likely to detect deployment failure scenarios earlier in the deployment pipeline.
  2. The preferred way to roll back a deployment is not to undo the deployment action but instead to redeploy the last known good build over the top of the failed deployment. This does rely on you tracking and retaining the last known good build for each environment.
  3. Rolling back a database can be hard. For example, if the failed deployment introduced a new column and new data gets written to that column before a problem with the deployment is discovered, it may be wrong to remove that column and discard that data. Similarly with new rows for an unchanged schema. I find using Agile Database Refactoring techniques to be very useful in having a database schema remain compatible with multiple versions of an application and avoiding the need to roll back schema changes.

As your automated deployment pipeline matures this should inherently reduce the transaction cost of performing a deployment which will then enable you to more efficiently deploy more frequently with smaller batches of changes in each deployment, thereby reducing the risk of any deployment failing. Additionally when you can efficiently implement, deploy, and verify a change quickly it becomes more effective to fix a problem with a failed deployment and roll forward with a new build than it is to roll back.

Regards,

Jason

Jun 12, 2012 at 5:34 PM

Jason,

What you've described are exactly the concerns that I have too.  I agree with each point completely.  It's difficult to keep track of what was deployed where, other than by the current build quality.  And not only that, but using TfsDeployer for deployment, it might make it more difficult to keep track of this since there is no history of deployments.  What would be really cool would be a UI for this to control what gets deployed where and when and with a secure storage mechanism for passwords versus clear text in scripts.

For the database, we have been making good use of the VS database project and VSDBCMD.EXE.  If you haven't looked into these tools, I highly recommend doing so.  These tools do what you mentioned for schema comparisons versus static one-way scripts.  Of course, some work has to be done at some point in the process to make sure rollbacks work correctly.

Our build process is a little more complex involving a single tfs build process for android, ios, and the web/server components.  So automating that alone was quite a task.  Setting up the automated deployment is another huge step forward.  We're dying to get to a point where we have more frequent, smaller changes added to the application and deployed.  It will most certainly help QA verify code faster and easier, allow for a deployment manager to push things around as he/she sees fit (without interrupting anyone else... unless something goes wrong), and allow development to concentrate on the software functionality and not on build/deployment issues.  And yes, your very last sentence is actually the key to it all.  Always move forward.

This really is a kick butt tool.  It's clear there has been a lot of work put into this.  Thanks for making this public.  It's definitely saving me from wasting vast amounts of time on deployments.

Dave

Jun 12, 2012 at 6:47 PM
Sorry for interrupt again but Dave, your concern can be solved using msDeploy since it allows u to create the log file which contains the detailed information about ur deployment.

There is a book called continuous integration.net guiding doing the deployments introducing a few good nice tools. I'm currently working on the database CI tool from the book.


Sent from my iPhone

On Jun 12, 2012, at 9:34 AM, "dpachla" <notifications@codeplex.com> wrote:

From: dpachla

Jason,

What you've described are exactly the concerns that I have too. I agree with each point completely. It's difficult to keep track of what was deployed where, other than by the current build quality. And not only that, but using TfsDeployer for deployment, it might make it more difficult to keep track of this since there is no history of deployments. What would be really cool would be a UI for this to control what gets deployed where and when and with a secure storage mechanism for passwords versus clear text in scripts.

For the database, we have been making good use of the VS database project and VSDBCMD.EXE. If you haven't looked into these tools, I highly recommend doing so. These tools do what you mentioned for schema comparisons versus static one-way scripts. Of course, some work has to be done at some point in the process to make sure rollbacks work correctly.

Our build process is a little more complex involving a single tfs build process for android, ios, and the web/server components. So automating that alone was quite a task. Setting up the automated deployment is another huge step forward. We're dying to get to a point where we have more frequent, smaller changes added to the application and deployed. It will most certainly help QA verify code faster and easier, allow for a deployment manager to push things around as he/she sees fit (without interrupting anyone else... unless something goes wrong), and allow development to concentrate on the software functionality and not on build/deployment issues. And yes, your very last sentence is actually the key to it all. Always move forward.

This really is a kick butt tool. It's clear there has been a lot of work put into this. Thanks for making this public. It's definitely saving me from wasting vast amounts of time on deployments.

Dave