I've read that TFS Deployer can run with least privileges than TFS Admin (http://www.codeassassin.com/blog/default,month,2008-07.aspx)
As described, I put my TFS user in the "Team Foundation Valid Users" server group and in the "Readers" project-level group.
I'm still getting the error:
System.Security.SecurityException: TF50309: The following account does not have sufficient permissions to complete the operation: DOMAIN\USER.
Check the permissions for the account and grant the appropriate permissions to perform this operation.
When user is in TFS admin group, it works perfectly.