The gun went off when the first policeman was checking his weapon at the Mulhouse police station in eastern France, reported L'Alsace newspaper.
The bullet hit the second policeman directly in the chest, the paper reported.
Emergency services – who are stationed just 200 metres from the police station – were at the scene immediately but were unable to revive the officer.
The officer, who worked in the local drug squad and was in his thirties, died at the scene.
Local prosecutors are treating the shooting as an accident, and have launched a full investigation.
The officer who fired the weapon is being treated for shock.
The incident is not the first time in recent months that a policeman has accidentally fired a service weapon.
In late February a French military policeman accidentally fired his weapon during a speech by French president François Hollande, leaving two injured and the president briefly shaken.
See footage of Hollande's reaction below.