Oliver came back to the manor late.

After Nathaniel and him had split, Oliver took a carriage back to Central London, and then walked blindly around the city.

This was happening.

They had had an agreement and Oliver wasn't going to just go on another mission after that.