When working with the Silverlight client object model in SharePoint 2010, I like to break my projects up into two pieces. First I create the Silverlight application UI and interaction as much as I can before I start wiring SharePoint into it. Then I add the client OM code later to bring it to life. This makes the development cycle a bit faster as I can avoid the SharePoint debug and spin up time.
However you may have some problems when you start debugging your custom Silverlight apps deployed to SharePoint. By default Visual Studio is going to try to debug script errors which is where Silverlight exceptions are bubbled up to. In addition, breakpoints in your Silverlight code won’t get hit.
What you want to do is look at the SharePoint project’s properties, click on the SharePoint tab and look at the bottom of the dialog. Make sure the Enable Silverlight debugging (instead of Script debugging) option is checked. This will allow you to have the full debugging experience of your Silverlight apps when testing from within SharePoint.