Mayor Michael Signer said he was disappointed the white nationalists had descended on his town and said responsibility lay with Mr Trump for inflaming racial prejudices during his presidential campaign last year.
He said: “I’m not going to make any bones about it. I place the blame for a lot of what you’re seeing in America today right at the doorstep of the White House and the people around the President.”
The clashes broke out at a “Unite the Right” protest over the decision by the city’s government to remove a statue of a Confederate general.
Mr Trump has now issued his first tweet condemning the violence, several hours after the violence began.
He followed this up with another, calling the clashes which are centred around the city’s university, “sad”.
Some on Twitter accused the President of not being specific enough in his tweet as he did not specify that he was referring to white supremacists or to far-right groups.
Several members of the Klu Klux Klan joined in with the protests as well as various alt-right activists, militia groups and Confederate heritage groups.
Mr Trump has repeatedly been accused of having stirred up racial tensions in the country through his rhetoric on immigrants and his vow to build a wall between the US and Mexico.
Officials in Charlottesville declared a state of emergency shortly after 11am local time as the clashes became more violent, with punches thrown between rival groups and pepper spray released in the crowd.
The clashes are the latest since the city, situated around 100 miles from Washington DC, voted to remove the statue of General Robert E Lee from a park.
Alt right journalist Jason Kessler said the protest was partly about the removal of Confederate symbols but also involved wider issues around free speech and “advocating for white people”.
He said in an interview: “This is about an anti-white climate within the Western world and the need for white people to have advocacy like other groups do.”
More attendees are expected to flock to Charlottesville – normally a quiet, liberal-leaning university city – by tomorrow.
Both local hospitals said they have taken precautions to prepare for an influx of patients and had extra staff on standby.
The Anti-Defamation League and the Southern Poverty Law Centre, which track extremist groups, both believe the event has the potential to be the largest of its kind in at least a decade.