The move has been planned since August last year when train passengers overpowered a man who had opened fire with a Kalashnikov on a train between Amsterdam and Paris.
From October, armed guards in civilian clothes will discretely travel among other passengers, according to the RTL newspaper.
The move is similar to the “sky marshals”, plain clothes officers who are based on certain flights, particularly on US airlines.
Introduction of the changes took so long due to the complicated process of rewriting French law, but also the time needed to train and equip the guards.
The guards will be on patrol in pairs on regional trains, TGVs, and even on commuter lines.
Guillaume Pepy, the head of SNCF, said the guards will be trained to shoot but that the weapons are to be be strictly reserved for emergency situations or in cases of attack.
The guards will also have the right to ask for passengers' identity papers or to search luggage upon arrival at stations.
Their work will be carried out in cooperation with police.
In general, the presence of firearms is only to reassure other passengers, reported RTL.
Exact figures for the number of armed guards on French trains remains unknown as yet, although Le Figaro newspaper reports that there could be as many as 3,000.
France increased security on its train lines following the country's terror attacks, introducing metal detector gates, additional security checks, and more CCTV cameras.