Revert "Don't try to remove decls that are not part of the lookup."

This reverts commit 180cd90a.
It breaks lookup of functions with 'using ParentClass::Func'
in combination with modules which reuse the hidden flag for
module purposes.

See https://reviews.llvm.org/D37180 for a clang test case that
tests for regressions like this in the future.