It is possible to colonize the galaxy going well under the speed of light. I have heard numbers bantered such as being able to colonize the Milky Way in a million years, and is one of the reasons why Fermi asked where is everyone, because even one species a few millions years in existence should be everywhere. Carl Sagan wrote a paper on that, suggesting we should be looking for artifacts in history because aliens would have visited in the past. So, theoretically, aliens could be here, even today, and not because of FTL capabilities, but just because they're older, smarter, live longer, and have tech millions of years more advanced than ours. Tech that far advanced means they could be right in front of us, and we not see them.

I think the 'faster than light' space travel is a distraction. Like you said, we all are in agreement aliens exist. The only question is "are they here?"