I guess the max mm depth would depend on the carat weight of the stone. Also, if the setting has scroll work and such around the basket, then a gemstone with a fat pavilon might not fit.
There really isn''t a set depth that would force things into the "custom" arena. Different mountings are constructed differently and would have varying depth allowances. Also, it isn''t always depth that complicates things. For example, some stones are cut with such steep angles near the girdle that they cannot fit standard mounts, even if overall depth is normal (think of toumalines with a closed "c-axis," which requires the gemcutter to use extremely steep angles on the ends of the stone).