Rovers took the lead from the penalty spot after Stuart Callaghan handled a David McFarlane shot on the goal-line.
McFarlane was subsequently given his marching orders and Bobby Barr converted the resulting spot-kick.
Alan Brazil's dipping volley levelled the game against the run of play after 21 minutes, but Chris Boyle pounced to score after a Barr shot hit the post.
Albion: Gaston, Reid, McGowan, Canning, Benton, O'Byrne, Boyle, Tyrrell, McFarlane (McCusker 78), Walker, Barr. Subs Not Used: McKeown, Stewart, Crozier, Ewings.
Goals: Barr 7 pen, Walker 62.
Berwick: Peat, Notman, Smith, Guy, McLean, Callaghan, McLaren, Currie, Brazil, Little (Radzynski 88), David Greenhill (Gray 78). Subs Not Used: McMenamin, Russell, Ellison.
Sent Off: Callaghan (6). Booked: Smith, Little.
Goals: Brazil 21.
Att: 322
Source: BBC Sport
Source: BBC Sport