should I send a JSON instead of HTML from the server and construct the
page at the client using jQuery templates - or can this be achieved
even if the Controller returns a ModelAndView?
No. That would be reinventing the wheel. There are already frameworks for working in this way. Although, somewhat incompatible with the kind of application that Spring MVC proposes.
Spring MVC was not designed to be implemented in this way.
I have considered having the login JSP and the contacts JSP combined
into one and hide/unhide based on response
From the front-end design point of view, that's a terrible idea. One more time, you are trying to solve something that has been already solved by other frameworks. And trying to defeat the application design proposed by Spring MVC.
The main problem I see here is that you have misused the framework.
Spring MVC applications fall into the group of web applications in which the server controls the state of both applications, server and client.
Briefly summarized, client is "compiled" (generated) by the server and supplied to the client in its final state. The browser just render the content as it's and wait for the user interaction.
For Spring MVC to control the state of the whole application, it's required to send requests to the server, so that the server calculates the new state and return the representantion (mainly the view as a html document) back to the browser.
This is applied to the navigation too. Every time we navigate, we change the state of the application, hence the request and the page reload.
That said, and back to the question, your implementation is defeating the above design, hence the complexity in almost everything you try to do.
In Spring MVC applications, JQuery and AJAX play a different role. They are used to provide the view with some degree of dynamism; e.g updating small sections of the page. Think in paginations, dynamic lists and combos, look ups, visual effects, background processes, etc. Features that would cause the server to rewrite and refresh the whole page just for so little (or unnoticeable) changes. As you guess, that's quite ineficient. And tedious, from the user experience standpoint.
I think this is a very common problem so there should already be best
practices/design approaches for handling this.
Yes, it was 10-15 years ago; when AJAX and JQuery (later), revolutionized the market of web applications. These two caused a sort of neurosis that led many people to develop hilarious web applications. Keep in mind that JavaScript libraries (and browsers) were not so sophisticated as they are these days.
Fortunately, these technologies have evolved according with the trends and the needs of the market. More or less, towards to allowing decoupled web clients.
Summarising, I would suggest two possible solutions:
Keep implementing Spring MVC, but stop using JQuery and AJAX as you are doing it now. Assume that JQuery should not to do in the client side what should be done by Spring in the server side.
Change Spring MVC by Spring Web. Replace @Controllers by @RestControllers, take views out of the server, make the server statless and implement the front-end independently from the back-end.
Regardless the above options, what matters is to make a proper usage of the tools. Or in other words, to choose the right tool for each problem.