knowing that this C project will not have a Symfony kernel (no KernelTestCase available)
Well, you still need a framework to run C 's tests anytime. Don't you?
I could remove the doctrine dependent code from the library and mock
some repository interfaces, but I would then need to re-implement them
in both projects A and B and this does not seem convenient.
I would dare to say it's very convenient because the opposite imposes Doctrine to A and B as a transitive dependency. What if A and B need different data access frameworks. What if they both use the same but different versions.
You could implement the D library which is a concrete implementation of C with Doctrine. You can move here Doctrine's code and adapters.
Finally,
How should I organize this C project so that I can still run my functional tests against a real database ...?
Why do so? Why you want to perform C's test with concrete databases (A or B). Ideally, you should not test C with databases C doesn't own or have any control over.
It would be more convenient if C runs (start and stop) a database during tests so that every use case populates the database with the precise information required for the test to pass. If all you need is some tables and rows mapping, an in-memory H2 DB should be enough.
That said, mocking repositories would be ok too. Both tests are not mutually exclusive.
However, If you need to test vendor-specific features (data types, functions, etc), say features only available in Oracle, MySQL, PostgreSQL, MongoDB, Casandra, etc. You can follow the same strategy by replacing H2 with virtualizations of these databases. For example, running docker containers.
Note that this approach would be a lot easier if you implement the project D I suggested because it already provides you with Doctrines the data access abstraction.
To sum it up
Project A --> c_Doctrine --> C
Project B --> c_Doctrine --> C
Project X --> c_OtherFwk --> C
Project Z --> c_CustomFwk --> C
