Exactly! This trend of silencing those with whom we disagree is troubling. Imagine where we'd be as a society if every idea that went against the prevailing norm was quashed and silenced.
Do not let the trolls silence you. It's exactly what they want. No thinking person could possibly conclude that you're racist on a reading of your work. I think we both know exactly what they really want to say. Keep being you and keep speaking up.