Third-round leader Kenneth Ferrie struggled to a 5-over 77 in the final round to end in second place at minus-8.
The dunhill links is much like the Pebble Beach Pro-Am on the U.S. PGA Tour, as players were paired with amateurs for the first three rounds. The top-20 amateur teams played in Sunday's final round.
Three courses were used for the first three rounds -- the Old Course at St. Andrews Golf Club, Carnoustie Golf Club and Kingsbarns Golf Links. The final round was contested at St. Andrews.
Montgomerie started the day five strokes behind Ferrie. Monty birdied the second and fifth to get within four of Ferrie, who also birdied No. 5.
Ferrie three-putted for bogey on the sixth and stumbled to a double bogey at the next as he slid to minus-11.
'It was tough to start five behind,' Montgomerie said. 'I said to myself if I could get two back in the first five, I could do this. I got two back after six, and seven was a dramatic change.'
Montgomerie sank a 3-foot birdie putt on the ninth to tie Ferrie for the lead at 11 under.
Trouble loomed for Montgomerie however. As he battled Tiger Woods in the final round at this year's British Open, Montgomerie bogeyed the 11th to start his slide away from Woods. The same nearly happened on this day.
Montgomerie bogeyed the 11th and double bogeyed the 12th after driving into a fairway bunker. However, Ferrie bogeyed the 13th, but still led by two.
Ferrie faltered to a three-putt bogey on the par-4 15th, while Montgomerie ran home a 45-foot birdie try on the same hole. So the duo went to the 16th tee tied at minus-9.
Both Ferrie and Montgomerie stumbled to bogeys on the 16th to drop back to 8 under. Each safely two-putted for par on the famous road hole, No. 17 on the Old Course.
At the last, Montgomerie drove left of the green. Ferrie's tee shot came up short, right of the putting surface. Ferrie chunked his chip and left it well short. He two-putted for par from 35 feet out.
Montgomerie, meanwhile, rolled in his second shot to within 3 feet of the hole. He calmly drained that putt for birdie and the win, his first individual win at St. Andrews.
'I gave it all back at the 12th, then it was just down to the last hole,' said Montgomerie. 'Very pleasing to say the least. It's never easy. I think experience meant a lot to me. I was using the yardage book from the Open for club selection. It was worth the shot that I won by having experience around this course that Kenny Ferrie might not have had.'
'To have been five in front and then Colin to just shoot 71 to win is pretty poor,' Ferrie said. 'Colin did play pretty poor, but I needed to shoot 75 to win basically and to not do that after playing so well all week is very disappointing. I feel that I let him win. I could have put more pressure on him when I had a chance. It's my own fault, I have got nobody to blame but myself.'
Stephen Gallacher, the 2004 champion, posted a 4-under 68. He was joined in a tie for seventh place by Darren Clarke, Pierre Fulke, Ricardo Gonzalez, Titch Moore and Lee Westwood at 6-under-par 282. Soren Kjeldsen, Maarten Lafeber and Miles Tunnicliff were one stroke further back at minus-5.