"We will no longer keep quiet" the former ministers, all women, including current IMF chief Christine Lagarde, ex-health minister Roselyne Bachelot and ex housing minister Cecile Duflot said in a statement published by the Journal du Dimanche weekly paper.
They promised to "systematically denounce all sexist, remarks, inappropriate gestures, inappropriate behaviour."
Their cross-party stand comes after French judges on Tuesday opened an investigation into multiple claims of sexual harassment against former deputy parliamentary speaker Denis Baupin, a move which experts welcomed as an end to the "omerta" around alleged abuse by politicians.
Baupin, 53, vigorously denied the claims and has instructed his lawyers to sue two French media outlets for defamation, calling the allegations "mendacious".
The same day Finance Minister Michel Sapin became ensnared in a sexual harassment scandal after admitting that he acted "inappropriately" towards a female journalist after twice denying any improper conduct.
The ex-ministers in their joint comment Sunday said: "Like all women who have entered into previously exclusively male environments, we have had to either submit to or fight against sexism.
"It's not for women to adapt to these environments. It's the behaviour of certain men that need to change," they added.
"Enough is enough. Impunity is over." The former ministers encouraged all victims of sexual harassment and aggression to speak up and lodge complaints.
"Today the judicial arsenal exists but the laws are not sufficiently applied".
Labour laws protect employees but are not respected, with few women lodging complaints and very few of those complaints leading to convictions, they complained.
Women's rights minister Laurence Rossignol, speaking on the France 3 television channel later, outlined two moves to tackle the problem.
"The first is to swiftly adopt the draft law currently going through parliament... which lengthens the time limits for a range of offences from three to six years, " she said, calling upon the Senate, or upper house of parliament, to pass the measure.
The second initiative "is to allow associations to file complaints instead of the victims," through an amendment which will be examined by parliament next month, she added.