The only thing that should be validated is the number of digits - does it match what you would need to call?
Access codes can/should not be allowed to be entered by a user (i.e +1 for the US). I'll tell you why - if it is an American entering it, and they are not all that savvy, they will probably want to enter 001 instead of 1. Reason being that 00 is how you access an international number from within the US. They don't know what the difference is between that and just +1. If you're calling landlines between area codes (and sometimes in the same area code), you'll also enter a 1 before that number. Just like Japan.
Take Japan. Provincial land lines are accessed with a 0 on the front of them. But if you're dialling from outside of Japan, you don't need that 0.
Mobile numbers in China are 10 digits and in Hong Kong they are 8. If your form captures country, then don't even ask for country code at the beginning of the number - just auto-show it in read-only text right next to the number they need to enter.
The list goes on.
Just validate the number of digits (make sure no letters are accepted) and make it easy for an end-user to enter the right number of digits. I.e. they shouldn't have to enter country code and if your form is super smart, they may not even need to enter area code if it's a landline number. Or if they start to enter things like an access code, like 0 in front of a Japanese number, just let them know they don't need to.
Every other character, besides digits, should then be a moot point - they don't have a chance to enter them.