Khan, who fired a 61 in the second round, fell three shots behind Casey heading into Sunday, but battled back to match his fellow Englishman at 21-under-par 267 after a round of 67.
Both players returned to the 18th tee of the Wentwood Hills course to begin the extra session, and they did so in poor fashion. Casey, whose game was breaking down progressively throughout the final round, missed the fairway left and Khan did the same with an errant drive of his own.
Casey punched his second shot out of the thick grass while Khan was unfortunate with a bad stroke that found a fairway bunker. Casey then played his third shot just off the green within 16 feet of the hole.
Khan's third shot hit the green, but rolled back off the putting surface. He chipped his fourth within a foot of the cup, while Casey two-putted for par to extend the playoff.
Back on the 18th tee, both players found the fairway this time around, but Casey's second shot got away from him, coming to a stop well right of the green. Khan left his second shot just short before Casey played a bad wedge that could not get out of the rough.
Casey, who was seeking his fourth career European Tour title, chipped his fourth inside 6 feet, but missed the par putt. Khan meanwhile converted for birdie and the biggest win of his career.
'It hasn't really sunk in yet,' said Khan.
Khan, who won't have to worry about Q-School for some time, was trying desperately to keep pace with Casey on the back nine when he still trailed by three despite a birdie at the 13th.
The 31-year-old persisted and hit his tee shot to 7 feet for a birdie at the par-3 14th to pull within two at 20 under.
Everything fell apart for Casey at the following hole. After a bad drive, Casey scrambled to reach the green and three-putted for a double bogey to fall back into a share of first with Khan.
'I can never be accused of giving up but it should have never gotten to that position at the end,' said Casey. 'That is what is most frustrating.'
Khan took advantage of his playing partner's misfortune's and hit a brilliant second shot to the par-4 16th that nearly found the bottom of the hole for an eagle. When Khan reached the green, he saw how close he was and tapped in for a birdie to take the lead.
Each player took a similar path to the last hole in regulation. After both found the same bunker with their second shots to the 18th, Casey played his third out of the sand to 6 feet. Khan followed and left his bunker shot just inside of Casey's.
Casey putted first and converted to reach minus-21. Khan had five feet left for a winning birdie, but failed to convert. Khan's time wasn't out yet, however, and he prevailed in the playoff for his maiden victory.
'I have heard people say it is a difficult thing to hole a putt to win and I found that out today,' said Khan. 'That was probably the most nervous I have ever felt over a putt on the 72nd hole, but anyone can miss a putt, I got over it, went on to win and I am proud of that.'