I wrote this in response to an email question, and I thought it would make sense to post the answer here as well.

The question is, how do you clean up the SQL database after a failed install?

If you load the SQL Management Console in SQL Server 2005, you may see a number of databases there: BisDB, BisDWDB, CurrituckDB, ELeadDB, psshared, VSTEAMSCC and VSSTEAMSCCAdmin. Those should all be deleted. The only databases to keep would be ReportServer and ReportServerTempDB and anything under System.

(If you do NOT see any databases after a failed install, that could be a sign that the user installing Team System on the application tier does not have the proper SQL Server permissions.)

There might also be new directory called C:\VSTS on the database server. This should be deleted as well.

I have found if the databases are created but the C:\VSTS folder has not, your problem might be one of permissions. The TFSIdentity user needs admin priveleges on the SQL Server machine. Or, in my case, I was specifying the domain wrong - I used VSTS.x4.ca instead of just VSTS.