Here is the SAML flow
The user accesses the remote application using a link on an intranet, a bookmark, or similar and the application loads.
The application identifies the user’s origin (by application subdomain, user IP address, or similar) and redirects the user back to the identity provider, asking for authentication. This is the authentication request.
The user either has an existing active browser session with the identity provider or establishes one by logging into the identity provider.
The identity provider builds the authentication response in the form of an XML-document containing the user’s username or email address, signs it using an X.509 certificate, and posts this information to the service provider.
The service provider, which already knows the identity provider and has a certificate fingerprint, retrieves the authentication response and validates it using the certificate fingerprint.
The identity of the user is established and the user is provided with app access.
So its point 5 which you are questioning? Here the Identity provider is Pushing the token to the Service provider.
So in theory, I could fake up a athentication response and post it directly to the Service Provider. Even if in your particular flow the token is requested directly from the identity provider, I could still potentially intercept that request and substitute my own reply.
What stops me (amongst other things) is that the message has to be signed by the Identity provider and the Service provider has the key required to validate that signature.