There isn’t a “rule”, as such. To quote Ó Dónaill, in the FGB entry for “ar”:
“(In references of a general nature it does not normally affect initial letter of following noun, e.g. ar muir, ar cíos, ar cosa in airde, on sea, rented, galloping. In qualified or particularized references it lenites, e.g. ar mhuir na beatha, ar chíos mór, on the sea of life, at a high rent. Eclipses in a few instances, e.g. ar gcúl, backwards)”.
“ar ndóigh”, “ar gcúl”, “ar dtús” and “diaidh ar ndiaidh” are probably just historical anomalies.