I’m currently building a plugin that crawls my site (to start with, I need to get this functionality working before adding more features). I’ve got that code working using recursion. If I crawl 1 page it works fine, 2 pages deep is fine, but after that, the process takes a while and when the operation is finished, it shows me the 404 page. I KNOW the page exists as that’s the option page I was one when I ran the function.
I just need a point in the right direction on how to make it not show a 404. I’m not entirely sure why it’s doing that either.
Read more here: 404 Page Loads After Long Operation in WordPress