The violent attack occurred in Toulon, in southern France, after two couples, a friend, and three children were out on bikes and roller skates on Sunday afternoon.
The group was confronted by "a dozen" youths. The incident quickly degenerated and ended up with the youths kicking and punching the three men in the group.
One of the victims, a 33-year-old man, was sent to hospital and was ordered to take a month off work.
Another, aged 46, suffered a broken nose.
Two teenage children in the group were left suffering from "severe psychological trauma", reported BFM TV.
It was initially claimed by the local prosecutor that the youths first targeted the women and insulted them because of the clothes they were wearing.
But one of the victims later reported that the way they were dressed was not the reason for the attack.
"It was purely gratuitous and our clothes were not the cause," the mother told Nice Matin newspaper.
She said the aggressors hurled "vulgar insults" at the women, prompting the three men in the group to react.
"One of the youths insulted us and then it quickly degenerated from there," she said.
The females in the group were reportedly shoved to the ground and their male companions suffered the brunt of the attack. CCTV footage allowed police to track down and arrest two of the assailants, who were aged 17 and 19.
When it was claimed the fight had started because the youths had taken issue with women wearing shorts it sparked uproar in France.
"You cannot attack a woman because she is in shorts. It's an odious act that we need to condemn in a firm manor," said Hubert Falco, the right wing mayor of Toulon.
The incident was interpreted by some as the growing influence of extremist Islamism in the country and how certain violent Islamists were acting as the clothes police for women.
In June this year women in Toulon held a rally (see photo above) to show support to a young woman who was attacked on a bus.
She said she was targeted because she was wearing shorts.