"CAN in Automation" (CiA) is an association that tries to standardize CAN in industrial environments. They publish documents on various parts of CAN networks and CAN related protocols. An example:
CiA 303-1 version 1.9.0
Recommendation - Part 1: Cabling and connector pin assignment
Description
This document provides device and network design recommendations for CANopen physical layer. Additionally, it provides the guidelines for selecting cables and connectors for use in CANopen systems.
Many of the documents published by CiA are downloadable from their web-site. Unfortunately, not all of them.
In an industrial environment, in my experience vendors of automation products usually stick to such documents, because they want to be compatible to other products.
Where the connector pin-out in the automotive world comes from, I do not know. My suspicion is, that one of the major vendors started in that way and the others simply followed. Maybe there is some standardization too, but I doubt that we are able to get access to such documents easily.