From a quick test, executing "Get-ChildItem -Path $Destination" when the $Destination variable is not defined will return the contents of the current directory. Piping this to Remove-Item will then attempt to delete everything in the current directory.
When running TFSDeployer.exe -d, the current directory for the script is most likely the TFS Deployer install location (hence all the files except the DLLs being deleted [the DLLs would be locked while executing]).
When running TFSDeployer as a service (especially when running as the Local System account) the current directory would probably be the Windows System folder.
I've raised an Issue to fix TFS Deployer so that the current directory is set to a safer location to avoid this kind of problem in the future. http://tfsdeployer.codeplex.com/workitem/8105
Until then I recommend using the following statement at the beginning of your PowerShell scripts to cause it to error when using an undefined variable (great for detecting typos):
Set-StrictMode -Version Latest