Topic: When do you stop unit testing?

We've been trying to implement at least some XP principles at work, the main ones being unit testing and continuous integration.  The benefit of unit testing seems to be that you always know when the system works, but there's an ancillary benefit in that if you have tests for an interface, you can refactor freely under that interface, and your tests will tell you whether your refactoring has affected the public contract.  But the question seems to be where you put the public interfaces.

I argued that in .NET, the assembly is the natural boundary, and in Java, it's the package level.  This is also beneficial in deployment, because you're not shipping unit test classes (whether that's desirable is another argument).  And there's ways to separate classes from their assembly/package at release time, but to me, this was the easy answer.  However, others argued pretty successfully that on a big, chunky interface like a Web Service, it would be nearly impossible to get good test coverage simply by testing that interface; while good XP discipline says that you build the simplest thing and test first; I agree that this leaves a big hole. 

So then you've got a problem with testing against non public (or internally public) interfaces.  What do you do if you refactor those interfaces?  Furthermore, what do you do if you find you want to merge two classes?  My first reaction was that you need to refactor the tests too.  Hopefully the higher level interfaces tests would detect whether your refactoring affected those interfaces, but there's several examples of testing falling short on truly big systems due to seemingly minor flaws deep in utility routines; not systems developed with XP, admittedly, but I think the point remains valid.  It's simply too difficult to get good test coverage at the system level, or even at the level of major components in a big system.

Here's a more concrete example.  Say you're implementing a facade type of pattern, where you have an interface that forwards calls to one of several private implementations, depending on the configuration of the system.  Where do the tests target: the facade or the implementation?  The facade doesn't do anything particularly interesting, except read the configuration and load the correct implementation.  My gut says that testing the facade probably falls into the domain of integration testing, while private implementations should be unit tested.  But I think that we must somewhere draw a line and say "no unit tests below this line", otherwise, we'll cripple the refactoring process.  You might say that conservative coding will protect you, and that by making as little public as possible, you at least reduce this risk.  I still believe that the risk is real, and that in a large system, you will require refactoring of internally public interfaces. 

What do people think about this?  Is this a valid concern?  How do you determine what interfaces get tested?