We have only been actively trying for 6 months, with no luck, even though we had stopped taking birth control for a few years. Do we need to worry?

When a couple isn't using birth control, they frequently avoid getting pregnant by having the man withdraw before ejaculating. It is not a reliable way of contraception to withdraw the penis before to ejaculation. You haven't yet gotten pregnant, but there's something else going on.