I think your main mistaken idea is this:
a fundamental change or redesign, which is basically the opposite of maturity, stability and well-roundedness
Stability and maturity are only ever going to be relative, and temporary. It is very, very rare for a piece of software to reach a certain point and become "finished" for all time, without any new ideas, new feedback, and new requirements meaning it needs to change. It doesn't sound like you believe your library is in that rare category, so you'll never meet this ideal of a "finally stable" version.
If you leave the first digit as 0 forever, it simply becomes meaningless - version 0.23.5 is actually just version 23.5. It also suggests to users that at some point the library will reach version 1, and maybe they should be wary of using it in production until it does.
If you think your library is ready for production use, and the fundamental design is stable, call it 1.0, and have a plan in mind for that would be a "big enough" change to go to 2.0.
I would also like to challenge your understanding of Semantic Versioning:
But every change "breaks the API", because my specific library is basically "just an API" for something else.
Changing an API can be done without breaking it - if you add a new function, or a new optional parameter, that is not a breaking change; and if you change the recommended easy to do something, but leave the old way in place with extra wrapping code, that is not a breaking change. Regardless of version numbers, if you want people to actually use your library, you should always be thinking about how to avoid breaking changes, because a library that I can only use by rewriting my code every two weeks is a library I will throw away as soon as possible.