Kempston were looking to continue their winning run as they welcomed near neighbours Bedford Town to Hillgrounds.
The game started in good fashion for the hosts, but the visitors had the first chance, Tyrone Lewthwaite rattling the post from a tight angle.
Shortly after, Rovers winger Ben Baker was exploiting a lot of space behind Sam Warburton at left wing back, with a couple of half chances before he put a ball across for James Hatch, but the striker got too good of a contact on it and it flew over.
Hatch was once again given a chance to break the deadlock, Muyembe's cross finding the head of the striker but he headed wide.
Bedford mixed it up at the break, going to a back four with Summerfield replacing Moloney. The change proved effective and the visitors started controlling the game and Rovers were struggling to be as effective as they were in the first 45. Ten minutes into the second half Drew Richardson volleyed the Eagles ahead following a scramble after a corner, despite Rose's best efforts to keep it out.
It was two on 79 minutes, a ball over the top was touched by Lincoln, but ended up at the feet of Craig Mackail-Smith, who converted calmly under pressure from the Walnuts defence.
Rovers pulled a goal back with five minutes to go, Aderoju's powerful header finding the net after a pin-point cross from Simpson.
Bedford held on and progress to the second qualifying round, Rovers will now set their eyes on the FA Trophy next weekend as Dereham travel to Kempston.