[VecOps] Avoid -ffast-math

The performance gains are unclear and the option can harm
users linking against the library with gcc<13, see also
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522 .