Java keeps track of the objects that have been written to the stream, and subsequent instances are written as an ID, not an actual serialized object.
So, for your example, if you write instance "a" to the stream, the stream gives that object a unique ID (let's say "1"). As part of the serialization of "a", you have to serialize "b", and the stream gives it another id ("2"). If you then write "b" to the stream, the only thing that is written is the ID, not the actual object.
The input stream does the same thing in reverse: for each object that it reads from the stream, it assigns an ID number using the same algorithm as the output stream, and that ID number references the object instance in a map. When it sees an object that was serialized using an ID, it retrieves the original instance from the map.
This is how the API docs describe it:
Multiple references to a single object are encoded using a reference sharing mechanism so that graphs of objects can be restored to the same shape as when the original was written
This behavior can cause problems: because the stream holds a hard reference to each object (so that it knows when to substitute the ID), you can run out of memory if you write a lot of transient objects to the stream. You solve that by calling reset()
.