disable fastmath for fallback backend #3563

Job	Run time
macOS / release / homebrew-clang	5m 41s
Windows / release / msvc	24m 48s
macOS / debug / homebrew-clang	4m 11s
Windows / release / clang-cl	22m 31s
macOS / release / system-clang	4m 19s
Windows / release / llvm	27m 19s
macOS / debug / system-clang	3m 38s
Windows / debug / msvc	24m 13s
Windows / debug / clang-cl	23m 35s
Windows / debug / llvm	27m 15s
	2h 47m 30s

Provide feedback