"Ancestors"? Do you really think those protesting the taking down of racist memorials were all the ancestors of said racists? Wow, those racists must've really gotten around.
Furthermore, those racist protestors were protesting the free rights of institutions under the First Amendment to take down their own statues. So if protesting someone exercising their First Amendment rights does "suppress your free speech rights," then the racist protesters were the violators. But protesting doesn't violate the First Amendment, as everyone knows, but is an exercising of it. That's why neither the protestors nor the counter-protestors were violating the First Amendment. Now racist protestors mowing down counter protestors against their racism with their car, that's a violation, but not of the First Amendment but rather of the federal code that outlaws murder.