Police discovered the damage in a cemetery in the town of Labry, the statement said. The ministry said it was the fifth time a cemetery has been desecrated in the area this year.
"These unacceptable acts require a firm response from the law," the ministry statement said, adding that an investigation was underway to find out who was responsible.
Interior Minister Bernard Cazeneuve strongly condemned the vandalism and offered support to the families and the local authorities.
The mayor of Labry also expressed his anger and shock at the vandalism.
Luc Ritz told Europe1 radio he was “stunned and disgusted” by the desecration of the tombs. “It was only crucifixes that were knocked over and some were moved from one grave to another.”
He asked how anyone could “dare to attack grave stones of people who should normally rest in peace.”
Later on Tuesday authorities said that three "minors" had been arrested in connection with the vandalism.
France has seen numerous incidents of graves being vandalised in recent months including an incident in February that saw five teenagers charged for vandalising hundreds of Jewish graves in a suspected anti-Semitic act that shocked the country.
The boys, aged 15 to 17, were charged with "the desecration of burial places due to the religion of the deceased" and with deliberately vandalising property on public land, according to local prosecutor Philippe Vannier.
Just days later dozens of graves were vandalised at a cemetery in Normandy.