Your apex A records are correct, and the health-check output already names the layer that is failing: caa_error: "Dnsruby::ResolvTimeout" together with is_pointed_to_github_pages_ip:false. A ResolvTimeout is not "no records" -- the resolver never received a usable answer. is_valid_domain and is_apex_domain are both true, so the name parses fine and the query dies on the wire.

The usual causes, in the order worth checking:

1. DNSSEC. A partial or broken chain makes validating resolvers time out instead of returning SERVFAIL, and GitHub's Pages checker validates. From a machine with dig:

dig DS heart-of-the-zone.ru @8.8.8.8
dig DNSKEY heart-of-the-zone.ru @ns1.reg.ru
dig +dnssec A heart-of-the-zone.ru @8.8.8.8

If dig DS returns a DS record but the DNSKEY lookup fails or the algorithms do not line up, disable DNSSEC at REG.RU. Plain A queries still succeed during a partial DNSSEC failure, which is exactly the asymmetry you are describing.

2. EDNS and buffer handling on the authoritative side. dnsruby sends larger queries than a default stub resolver; some edge nodes silently drop those while answering small ones:

dig +bufsize=512 +noedns A heart-of-the-zone.ru @ns1.reg.ru
dig +bufsize=4096 A heart-of-the-zone.ru @ns1.reg.ru

If the second hangs and the first returns instantly, that is the failure mode, and it is worth a ticket with REG.RU rather than a change on your side.

3. CAA. caa_error is populated as well, so check for a malformed or restrictive CAA record:

dig CAA heart-of-the-zone.ru @8.8.8.8

Pages needs 0 issue "letsencrypt.org" and 0 issue "digicert.com", or no CAA record at all. A CAA record with a syntax error causes a timeout on some resolvers rather than a clean refusal.

After any change, the health check is cached. Remove the custom domain under Settings -> Pages, save, then re-add it and save again to force a fresh check instead of waiting the cache out. Propagation alone will not re-trigger it.

To verify from a resolver that is independent of your ISP:

curl -s "https://dns.google/resolve?name=heart-of-the-zone.ru&type=A" | head -c 300

If clearing DNSSEC or fixing the CAA record unblocks the check, marking this as the answer will save the next person hitting the same Dnsruby::ResolvTimeout the same 36 hours.

All possible checks showed no problems with the configuration. A support ticket was submitted to technical support, which confirmed that everything was configured correctly and also confirmed that there was an issue with certificate issuance. They promised to look into it and keep me updated. Meanwhile, the "DNS Check in Progress" message has been there for more than 6 days.
I see that similar issues have been occurring quite frequently over the last few days, so I will keep updating this thread. Perhaps the solution to my problem (when it is finally resolved) will help someone else as well

Another affected domain here: thatsparallel.space on GitHub Pages (02coldrain-blip/02coldrain-blip.github.io). As of Oct 21, the Pages UI remains at DNS Check in Progress / 1 of 3 Certificate Requested, and Enforce HTTPS is disabled. A normal curl -I https://thatsparallel.space/ fails with SEC_E_WRONG_PRINCIPAL. Google Public DNS returns the four GitHub Pages apex A records and www CNAME to 02coldrain-blip.github.io; there is no apex AAAA, CAA, or DS record. Authoritative NS are ns1.reg.ru / ns2.reg.ru. We also removed and re-added the custom domain once, without resolving the issue.

This looks very similar to the REG.RU validation/CAA timeout reported above. We cannot access the authenticated Pages health endpoint to inspect caa_error directly. If a GitHub Pages maintainer can check the backend validation and certificate job for this domain, that would help. We are leaving DNS unchanged to avoid disrupting the mail records.