The court ruled that while there could be limits to military personnel exercising freedom of association, "a blanket ban on forming or joining a trade union encroached on the very essence of this freedom".
Michel Bavoil, vice-president of the Association for the Protection of Rights of Military Personnel, hailed the decision as "great victory that is extremely favourable for soldiers", in comments to French television.
The court examined the case of a French gendarme who had created an association called "Gendarmes and Citizens", as well as an internet forum of the same name, where members could exchange views.
Shortly after Jean-Hugues Matelly informed his superior officers that he had created the association, in 2008, he and its members were ordered to resign.
"This senior commanding officer considered that the association resembled a trade-union-like occupational group," which is prohibited by the armed forces, read the judgement.
Matelly then filed a complaint with the European Court of Human Rights which backed his claim that the acts of his superior officers had violated his right to freedom of association.
The court said that while France had put in place institutions to take into account the concerns of military personnel, these did not replace the right to form and join trade unions.
France was ordered to pay Matelly €1,400 ($1,760) and the judgement could require Paris to change its legislation.
"It is clear that this ruling will require a modification of French legislation" said legal expert Nicolas Hervieu.
Judgements by the court, which rules on violations of the European Convention on Human Rights, are binding on the country involved.
The defence ministry said in a statement it would study the judgement in order to pinpoint "what changes must be made to French law" while respecting the unique status of the military, such as its neutrality.