The stakeholders have their say at the end of the Sprint Review, which is their point of time to voice concerns, influence the possible forecast the upcoming sprints and give feedback.
The Sprint Retrospective is for the Scrum Team. In some cases only the Development Team and the Scrum Master are present at this meeting. The product owner should join, often the relationship with the product owner is part of what can be improved, so inviting him/her should make that process easier. Also, any changes to the DoD (that can be made as outcome of the Retrospective) need to be taken past the Product Owner, maybe the team missed an important reason (like legislation) for the things to be as they are. In the end the Product Owner is end-accountable for the product and it's quality, so he should have a say.
In the Professional Scrum Developer course we also include a remark that you might get a more open communication going by asking the Product Owner to leave the meeting at some point. If that is the case, the lack of transparency and trust should be addressed at some point in a retrospective in the future, I'd say...
Stakeholders have no place at the retrospective. If they want changes to how things are going they will need to go through the Product Owner. If there are issues between stakeholders and the team, it might be a good idea to do a separate meeting (not a retrospective) with the whole scrum team present (incl SM and PO) to put the cards face up on the table and work out the issues.
Have you asked the manager why he thinks these stakeholders need to be included? What does he want to get out of that meeting? Figure out which meeting would be the right place for that concern to be addressed and who should be the one addressing it. There might be a need to plan something that's outside of the standard scrum meeetings, which of course is allowed.
Relevant passage from the scrum guide:
The Sprint Retrospective is an opportunity for the Scrum Team to inspect itself and create a plan
for improvements to be enacted during the next Sprint.