Dougherty (66) played at Carnoustie, while Otto (67) played the Old Course at St. Andrews. All three are at 11-under 133.
'When I was 9-under today, I was thinking it would be nice to shoot 62 here, but it didn't happen,' said O'Malley, who forgot his clubs in the morning and had them delivered to the course. 'It was great though.'
Ernie Els (67), a two-time runner-up, was one shot behind after his round at St. Andrews. He shared fourth place with Anton Haig (66), Rafael Echenique (67), Kenneth Ferrie (65) and Gonzalo Fernandez-Castano (67).
Steve Webster, who led after the first round, shot a 70 on the Old Course and shared ninth place with Thomas Levet (65) and Jeev Milka Singh (68).
The Kingsbarns course is also being used for the tournament.
Dougherty is chasing his first victory of the year.
'I am just going to go out there and try to enjoy it,' the Englishman said. 'I have tried so hard to win that I am now at the stage of thinking just go out and have fun.'
British Open champion Padraig Harrington (66) had eight birdies on the Old Course -- none requiring a putt of more than 10 feet.
Justin Rose (69) is one stroke behind Harrington and four off the leaders. He is tied with Luke Donald (70) and Henrik Stenson (69).