Too small:
This causes excess current draw when you want to "counter-act" the pull-up/pull-down.
Say you have a pull-up circuit like this:

simulate this circuit – Schematic created using CircuitLab
When M1 is "closed", you're effectively shorting output to ground. So the current flowing though R1 is:
\begin{equation}
I = \frac{Vs}{R1}
\end{equation}
Depending on what resistor power rating you have, limitations of Vs, and limitations of the switch M1 (doesn't have to be a MOSFET or even a transistor), you can figure out a lower resistance limit which will start to cause problems. You can do a similar calculation for pull-down resistors. When R1 gets way small (or the on resistance of the switch gets large), you start getting voltage-divider circuits so all of a sudden the output level isn't a digital signal anymore. You could use this to improve the lower limit value, but in reality there's usually no need to get anywhere near this lower limit.
Too large:
Large pull-up resistors have will excessively limit the amount of current which can be provided to the output. There are a few consequences to this:
- If the output has some capacitance (trace parasitics, gate capacitances, etc.), the output voltage could have some significant rise time similar to a RC circuit.
- The assumption that all external "sources" and "couplings" are insignificant start to break down. For example, cross-talk or antenna-like effects could become significant.
I can't really say how to account for the second point or other points in general, but let's take a look at the first point:
An RC circuit has a time constant of t=RC. If you have a signal line which relies on pull-up/pull-down resistors (I2C), the fastest signal rise time you can transmit is limited by this time constant.