AI is getting better: what it missed in May, it found in October

What happened

In early May several people wrote to me that pumpydumpy2visual.com wouldn't open for them. Only over Wi-Fi. On mobile data, on the same phone, the website loaded normally. Instead of the page, the browser showed a certificate error and wouldn't let them click through.

What AI advised in May

I described the problem to AI and got a convincing explanation: the new network protocols HTTP/3 and IPv6 and their settings were behind it. So we turned them off. Then we tried editing headers and writing scripts that were supposed to “fix” the connection. Nothing helped.

It all sounded logical, it was well written and it was completely off.

Where it went wrong

The advice itself wasn't stupid. It just answered a different question. When a protocol breaks, the page doesn't load and waits until it times out. A certificate error is something else: the phone got someone else's certificate, not ours. No protocol setting causes that.

AI skipped the exact symptom and solved a problem we didn't have. And I believed it, because it sounded sure.

What happened in October

In October I came back to it with today's AI. This time it started exactly at the symptom: not over Wi-Fi, yes over data, a certificate error. From that it concluded that something between the phone and the website was intercepting the connection. Typically a firewall in the network, for example at a company, a school or a hotel.

Then it went through how security services rate our domain. Most were fine. But one, FortiGuard from Fortinet, listed it as “suspicious”. It got that rating a few hours after registration, most likely because it was brand new and its name looks generated.

Networks with a Fortinet firewall block such a domain and show their own page with their own certificate instead. The browser recognises that the certificate isn't ours and won't let you through. There's no such firewall on mobile data, which is why the website loaded there. I don't have it confirmed directly from that network, but everything fits together. We've asked Fortinet for a re-evaluation.

What changed

Not the problem, and not me. The AI changed. In half a year it moved on so much that it solved the same mystery it had failed on before. It also helped that we had the exact symptom written down, along with everything we'd tried.

And that's the most important lesson for me. Today AI can do things it couldn't a few months ago. But it can still sound sure at the moment it's wrong. Both are true at the same time.

← All articles