This is a complicated subject, but I'm going to take a shot...
As with many things used in the electronics industry, cadmium plated components aren't without risks. The highest risk being the industries that use the materials in high concentrations and large quantities. It is important though, to consider route of exposure as well as if the exposure is significant compared to background exposure. Cadmium is a naturally occurring element and is present under many circumstances in the natural environment.
As with all electronics components handling, you should always wash your hands after handling them, especially before eating and not touch your eyes or mouth while handling electronic components.
Cadmium exposure from those connectors is probably well below detection limits and you would need to be exposed in the worst case scenario for long periods of time to see any effect. Cadmium in high doses through ingestion tends to accumulate mostly in the kidneys causing renal tubular dysfunction and is thought to contribute to hypertension. You would probably need to lick 8-10 connectors a day to get a significant dose. Please don't do that. This type of connector has been used in naval electronics and avionics for many decades and though we haven't made any direct connections to contact exposure causing disease, I don't think it could be completely ruled out. The risk of handling cadmium plated parts is very low if you practice good hand hygiene.
Personal safety is an important consideration and everyone should spend a bit of their time understanding the environment in which they find themselves. Don't rely on an industry that uses a chemical to tell you how dangerous that chemical is. At the same time, it's important not to go overboard with demonizing any one safety concern. Understanding the molecular consequences of everything you are exposed to is nearly impossible.
Luckily, we are provided with an array of protection mechanisms that can help us tolerate certain exposure levels to chemicals. Cadmium in small doses up-regulates a protein called metallothionein which can bind the cadmium and prevent toxicity. Making metallothionein can be problematic to folks that have cysteine related disease and they need to be more careful with metal exposures of all sorts.
I'm pretty sure someone could write a book on this and maybe they have already. The problem is that most of us don't have time to read the thousands of volumes it would take to understand every molecular interaction that may be dangerous to us so we often rely others to tell us one way or the other. It's hard to say how well we've been served by this method of learning, but it allows us time for pursuits other than constant reading.
In the end, you need to make your own assessment of the risk vs. reward. What's right for one may not be right for the other. We each need to find our own balance that allows us to sleep at night and live our life without thinking we need to be locked in a bubble, but not become so complacent that we seal our fates with ignorance. It's a fine balance.
This document may be a useful resource to understand the relative risk of cadmium exposure. You can also visit the pprc.org website for other valuable resources.
Edit: It's important to note that the corrosion plating used on connectors is very thin, usually measured in nano meters or microns at best. Being so thin, they need to be very stable in order to maintain the characteristics of the coating. If the entirety of the coating were to just "rub off" then the coating would be useless. These coatings are very stable under normal use and it's the reason that they last for decades without corrosion.