Karlsson and Ross Fisher of England had 7-under 65s at The Old Course to finish at 10-under 278 and tied with overnight leader Martin Kaymer of Germany, who shot 68.
Fisher drove into the Swilcan Burn at the first playoff hole to bogey. Kaymer missed from 10 feet for birdie before Karlsson made a three-footer for the victory.
Kaymer also missed birdie chances from four feet at No. 17 and eight feet at No. 18 that would have given him a third European Tour win this year.
The victory boosted Karlsson into the lead of the Order of Merit ahead of Padraig Harrington, who won the British Open and PGA Championship this year.
I wasnt thinking about the Order but, no question, Padraig has had the better year, said.
There are four events remaining on the European schedule.