Agnostic kind of nonuniform learnability

Structural risk minimization is an algorithm that can nonuniform learn nonuniform learnable classes.

What about the algorithm: Check the size of the training set, $m$, and see what is the largest $H$ in the collection such that $m$ guarantees generalization gap smaller than $\epsilon$. Then just do ERM with that $H$? Would that work for nonuniform learnability?