Why would patient management systems not assert limits for certain biometric data?

Question

I think many people with even a small experience in designing UI/UX to handle user data will be familiar with the perils of putting in input field/database limits for personal data, such as names. However, when it comes to storing biometric data, such as in medical/patient management software, I might've assumed that there was some validation on input given the intended use-case!

That seems as though it might not always be the case, having recently seen this tweet, in which someone was invited for their COVID-19 vaccine prematurely, apparently due to his GP surgery storing his height as 6.2cm, giving a BMI of 28,000.

Questions:

Is this just a flaw in their particular software? Is it possibly just the case that many of these systems were never intended for the mass selection of patient groups?

Or are there valid reasons that you might not want to introduce input ranges and sanity checks to biometric data?

---

Colour me only mildly concerned, given the AI-based future of medical decision making!

Answer 1

Is it possibly just the case that many of these systems were never intended for the mass selection of patient groups?

This has absolutely nothing to do with it. Even if this software was only used in scope of retaining patient information when the patient visits their GP, the calculated BMI would've been incorrect.

The issue with mass harvesting of data is that people don't invest any time in looking at specific entries anymore, and therefore they don't see obviously wrong data. In comparison, that doctor who looks at the patient info for the patient who is in front of them will notice that 28000 number.

Is this just a flaw in their particular software?

If the software was never required to put boundaries on data input, then not having boundaries isn't a flaw in the software. At best, it's a flaw in the requirements.

The 28000 also wasn't a bad calculation either. It was a correct calculation based on the data that was input. You cannot blame a calculation for the correctness of its input, or what I like to refer to as "shit goes in, shit comes out".

So you want to limit the height input then (and weight, but let's focus on height for now). What should the minimum limit be?

Well, the shortest person recorded is about 62 cm. But what about when that record is broken? Because most records tend to get broken once in a while.

Also, babies are generally 50cm, so maybe that's where the limit should be. But what about premature babies? Even only accounting for the viable range of premature births who have a reasonable chance at survival (which is 24 weeks), they can be as small as 22cm.

So if you want to account for all humans, we could argue that 22cm is a reasonable minimum boundary.

You should already notice that 22cm is still close to the 6.2cm figure we started with.

I reverse engineered your example. For a 28000 BMI and a height of 6.2cm, you'd need to weigh about 108kg. But even if you disallow this height, yet still allow a height of 22cm, that still leads to a BMI of 2231.4.

The BMI data is still nonsensical, even though both input values are within their individual normal ranges. We established that a height of 22cm is possible, and a weight of 108kg is also realistic.

Your question is built on the assumption that such data validation would be trivial to implement without fault. The above calculation shows you that this assumption is incorrect.

Or are there valid reasons that you might not want to introduce input ranges and sanity checks to biometric data?

While people's height and weight isn't going to change overnight, it's generally inadvisable to add more restrictive validation to data than what was asked, based on nothing more than what a developer thinks might be a possible reasonable restriction.

For example, my country's license plates used to be of the format AAA-000 (and initially, vanity plates weren't legal). Should software have only allowed this format?

Well, it seems like you would have forced that. But when those license plates ran out, we started using 000-AAA. And when that ran out, we've started using 0-AAA-000.

If you had written those validation checks, you would've had to change and redeploy your application every time the format changed. And this is a relevant topic, because that is precisely what happened in my country. They had to manually update thousands of devices (speed cams, parking lot cameras, police vehicle cameras, ...) because they were unable to register these new license plates.

Had they not bothered with this format validation, they wouldn't have had to update their software. Given that in this case it was embedded software on devices, having to redeploy is a cumbersome and expensive task.

Similar issues could be encountered with:

• Landlines are 9 digits here, whereas cell phones are 10 digits
• Postal codes here are 4 digits, but they've introduced 5 digit codes recently
• House numbers are numeric, but there is a fringe case whereby a property that is split into two properties will get a "A/B/C/..." suffix. So what once was number 1 becomes numbers 1 and 1A. This is not the same as a box (i.e. number 1 box A). For example, we live at address Redacted Street 14A, but the building next door (Redacted street 14) is an apartment building, and labels their apartments A/B/C/... 14A is my house number. 14 box A is the nextdoor apartment on the first floor. You can imagine my frustration whenever I fill out a form and notice that the developers needlessly decided to enforce a numeric format in the number textbox.

Colour me only mildly concerned, given the AI-based future of medical decision making!

You're putting the cart before the horse here. Even if the patient info registration tool allows for inputting nonsensical data, that doesn't inherently mean that the interpretor of this data must blindly believe anything it is told.

If you only could implement one validation, you'd put the validation on the AI, not on the data collection tool. If you blame any mistakes your AI makes on the input data rather than the AI, then your AI isn't an AI, it's just an algorithm.

Answer 2

I have worked in this industry, and there are several popular patient management systems which simply accept whatever number the doctor enters for the patient records.

In practice this means that if you looked at a thousand patients' data, you would typically find one or two where the doctor had entered meters or feet into a field that was meant to store centimeters.

One place where you will often find input validation is in the calculators that some of these systems have for generating figures like BMI, eGFR, etc. But this SMS wouldn't be generated from the interactive BMI calculator window, it would be generated from the height and weight in the patient database, which is full of non validated data.

When I developed a chart that would show the average BMI of a patient population, I found that to get a reasonable looking chart I had to program it to filter out extreme outliers that were likely to be data errors, because the data had not all been validated on input. For example, I did not include people less than 10cm tall in the BMI chart.

To answer some of your specific questions:

Is this just a flaw in their particular software?

It's like this in several systems, not just one.

Is it possibly just the case that many of these systems were never intended for the mass selection of patient groups?

Yes. The patient database was never intended to be used like this.

The primary use case of the patient database is storing your doctor's notes within the clinic so your GP can read them again next time you visit. Other functions, such as sending SMS, are added in later versions as a bonus feature, or implemented through third-party add-on software.

Mass selection of patient groups is not a common feature of patient management systems, but there is third-party software available that can do it.

Basically, the data was entered into software A, and the SMS was sent by software B, and software company B have no say in whether software company A perform input validation.

Answer 3

Perhaps the assumption is that the doctor knows more than the programmer.

Would you want to be the doctor to tell your patient that you can't treat them because the IT department thinks the patient can't exist?

"I'm sorry Mr SuperFitAthleticRunnerPerson, we can't treat you because our system won't allow a resting heart rate less than 60."

Answer 4

Certainly, something went wrong to allow this notification to go out, but it's not necessarily a lack of input validation.

Strictly forbidding invalid input often sounds simple, but is actually an extremely difficult problem. A classic example is validating names - surely a one-letter input is someone typing their initials, and should be rejected? Not if they have the common Korean surname O.

Sometimes, the programmer might be able to codify likely mistakes, and have a "soft validation" that triggers a message like "this is an unusual value, are you sure?" For all we know, this happened in this case, and the user accidentally clicked "Yes".

Data is also copied between systems frequently, so it may be that the data was entered in one system that lacked validation, and then was imported into another. Again, the import system couldn't know for sure that the data was bad, but could have triggered a "soft validation" - "import includes suspicious data on rows X, Y, and Z". Again, a human operator needs to correctly act on this information.

Finally, the data was used to produce a report that was going to be acted on. Something as simple as sorting that report by BMI would immediately have made this result stand out, and a check against the input data would have discovered the cause. In my opinion, this is where this case should have been spotted - but we don't know if this was a missing feature in the system, or operator error using it.

Checking for every possible way a complex system can fail is hard, but providing ways for someone to spot that it has failed can be essential.

You are quite right to be concerned about this in the context of "AI" systems, which is a rather vague term currently used mostly for "machine learning" algorithms. Because these systems aren't built from individually tested components, but "evolved" or "trained", it's even more important that they have appropriate supervision.

Answer 5

This might have been caused by too much input validation of the wrong sort. When a form doesn't let you enter information in the way you need, and the information is useful or important to your job, users will invent their own awkward conventions in order to record the information. In the words of Ian Malcolm, life finds a way.

Perhaps in this case, the patient only knew their height in feet and inches, but the system only accepted metric, so the nurse or doctor invented 6.2 cm as the closest he could get to 6 feet 2 inches, rather than wasting time doing the conversion.

So input validation, especially for data entry intensive applications, must strike a careful balance. Make it too permissive, and it's too difficult to use for any sort of aggregation. Restrict use cases the user actually needs, and users will invent bizarre ways to do what they want. In these sorts of applications where it's difficult to anticipate every conceivable situation, it's usually better to err slightly on the side of being too permissive.

Answer 6

You might argue that the failure of validation is not at the input stage, but at the statistical reuse stage.

It's not illegitimate for the system to allow the entry in principle, as it may not be clear what a sensible maximum BMI ought to be, and sticking your finger in the air to gauge a maximum doesn't increase the reliability to 100%, so further checks should still have to be done.

Certainly there are people of the given weight, and there are quite possibly patients of the given height (perhaps premature babies, for example), so what would have to be validated is particular combinations of values.

Rather, those making use of unvalidated data, including processed data where additional assumptions are introduced, ought to have familiarised themselves with the data and investigated extreme values.

Even moderately extreme values, those still within the realm of medical possibility, may have implied that it was pointless to invite an immobile person to get a jab, or that a conversation should have been started first with the GP to exercise judgment on the medical sensibility of giving a jab. This is really a story about those analyses and steps being omitted from the process.

And if those additional steps are seen as too much hard work and cost to implement, relative to what is at stake by sending an erroneous invite, then why will data entry validation involve any less hard work and cost overall relative to the stakes involved?

Answer 7

The GP and the software engineer are both subject to the same ethical principle:

• The GP took the Hypocratic Oath:

  I will do no harm or injustice to [the patients]

• The software engineer does not have to take an oath, but there is a code of ethics and professional conduct:

  Avoid harm

So, medical practitioners and software engineers who worked together on this medical system should have talked about expected validity range of input and vital parameter, brainstormed on risks (medical risks for the GP, technical risk for the engineer, user confusion risk for the UX designer), and agreed how to best mitigate them.

I understand the argument that a patient record cannot be rejected because of wrong or missing input, because lives are at stake. But some warnings could easily be issued:

• at data entry, to avoid typos risk ("are you sure that ...."),
• at end of the day, to avoid bias under time-pressure ("today you had a patient with..."),
• before using the data on a connected system for public health ("x people in risk group A present anomalies, please check").

This case shows that the system, as it is, can cause harm (i.e. someone vaccinated early without need whereas a weaker person who desperately need the vaccination doesn't get it in time).

P.S: Sorry to add yet another answer, but when talking about a life critical processing, I personally cannot agree with commercial arguments such as: "It was not part of the requirements".