Every site accumulates broken links over time. The question is whether you find them before search engines do, or after your rankings already reflect the problem.
Check your server logs or analytics for 404 responses on a monthly basis, then decide for each one: redirect it, restore the page, or leave it if nothing of value was lost.
Fixing 404 errors is rarely glamorous work, but it protects the crawl budget and link equity your site has already earned.