Hi Benoit
Not sure what the original problem was on the 27th of Dec but the current problem is as follow:
numberportability.ch has an NSEC negative proof at the zone apex which states that there are no other hostnames then numberportability.ch itself.
dig @dns1.swizzonic.ch numberportability.ch. AAAA +dnssec +norec ... ;; AUTHORITY SECTION: numberportability.ch. 900 IN SOA dns1.swizzonic.ch. hostmaster.swizzonic.ch. 2022121601 10800 3600 604800 86400 numberportability.ch. 900 IN RRSIG SOA 13 2 900 20230119000000 20221229000000 10556 numberportability.ch. TyEySTihvSFvdHr+AIOwYV7P/7OwnEkkKmviAfpDM7Ls/7oSkE0YWpKT rtn2OLAcGayrejP3hYYdU9cH7+DddQ== numberportability.ch. 86400 IN NSEC numberportability.ch. A NS SOA MX TXT RRSIG NSEC DNSKEY numberportability.ch. 86400 IN RRSIG NSEC 13 2 86400 20230119000000 20221229000000 10556 numberportability.ch. Cv2K3pWOJ739PgraeAseqUCXegIGJsrN5zmFRa2hKpohwKY/NCSx2RuJ q1PdHXPh6w9Es+Y6btCZNtuRfQ7iZg==
See the NSEC proof in the above query.
A DNSSEC validating resolver which supports and enables synthesized answers from cached NSEC, NSEC3 (rfc8198) will answer follow up queries for this domain name which fall outside the NSEC chain directly with NXDOMAIN. This problem only occurs if there is already a cached NSEC record. I guess this is not unlikely as most web browser do HTTPS and AAAA qtype lookups in paralell to A queries. HTTPS and AAAA do both not exist for numberportability.ch.
Such a DNSSEC validating resolver which synthesizes answers from cached NSEC, NSEC3 records will not log a DNSSEC validation error. The problem is that the NSEC proof is lying and not that it its DNSSEC signature is invalid.
All current open source DNS resolver software support synthesized answers from cached NSEC, NSEC3 (rfc8198). I tested knot-resolver, powerdns-recursor and BIND and unbound. In BIND the configuration option is called "synth-from-dnssec" which is enabled by default since BIND 9.18. In knot-resolver there is no configuration option, it is always enabled. For powerdns-recursor you need v4.5 where it is enabled by default, option "aggressive-nsec-cache-size". For unbound you need v1.17 but it is disabled by default. The option is "aggressive-nsec". Google Public DNS also supports it.
Note, synthesized answers from cached NSEC, NSEC3 is a very useful feature. To quote the unbound documentation [1]:
"Aggressive NSEC can result in a reduction of traffic on all levels of the DNS hierarchy but it will be most noticeable at the root, as typically more than half of all responses are NXDOMAIN.
Another benefit of a wide deployment of aggressive NSEC is the incentive to DNSSEC sign your zone. If you don’t want to have a large amount of queries for non-existing records at your name server, signing your zone will prevent this."
[1] https://unbound.docs.nlnetlabs.nl/en/latest/topics/privacy/aggressive-nsec.h...
It is also very effective against random subdomain attacks, very common attack vector at the moment (See also rfc8198) where rate-limiting queries does not help. A DNS-OARC talk from 2017 by Petr Špaček also compared it to running the root zone locally, https://indico.dns-oarc.net/event/27/contributions/473/attachments/430/725/D....
If you want to trigger the problem on your DNS resolver, you need to query for a NoData answer first e.g.:
dig numberportability.ch. AAAA
The resolver caches the NSEC proof. You can then query for an existing name which will be synthesized because of the "lying" NSEC proof. e.g.:
dig www.numberportability.ch. A -> synthesized NXDOMAIN instead of the answer record
If you are the zone owner of numberportability.ch, you need to tell Swizzonic that they should execute:
pdnsutil rectify-zone numberportability.ch
This will fix the problem temporarily until the zone is changed again by some users.
A possible (note I'm guessing) root problem is that Swizzonic uses a WebFrontend which directly access the Database with SQL statements. This breaks DNSSEC in PowerDNS. One has to use a WebFrontend which uses the PowerDNS API. See also https://github.com/PowerDNS/pdns/wiki/WebFrontends
Cheers, Daniel
On 27.12.22 09:45, Benoit Panizzon via swinog wrote:
Hi List
Fancy another DNS issue hunt?
We have DNSSEC validation enabled on our BIND DNS Servers.
We started seeing:
no valid RRSIG resolving 'www.numberportability.ch/DS/IN': 2a01:8100:2901::1:183:202#53 no valid RRSIG resolving 'www.numberportability.ch/DS/IN': 2a01:8100:2901::1:183:201#53 no valid RRSIG resolving 'www.numberportability.ch/DS/IN': 81.88.58.219#53 no valid RRSIG resolving 'www.numberportability.ch/DS/IN': 195.110.124.196#53
broken trust chain resolving 'www.numberportability.ch/HTTPS/IN': 2a01:8100:2901::1:183:202#53 broken trust chain resolving 'www.numberportability.ch/AAAA/IN': 2a01:8100:2901::1:183:202#53 client @0x803541d60 X.X.X.X#27325 (www.numberportability.ch): query failed (broken trust chain) for www.numberportability.ch/IN/AAAA at query.c:7724
And of course the query fails, disrupting access some some quite important API.
numberportability.ch. 900 IN SOA dns1.swizzonic.ch. hostmaster.swizzonic.ch. 2022121601 10800 3600 604800 86400
$ dig +dnssec RRSIG www.numberportability.ch @dns1.swizzonic.ch ; <<>> DiG 9.16.33-Debian <<>> +dnssec RRSIG www.numberportability.ch @dns1.swizzonic.ch ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 39132 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available
So, from my point of view, the authoritative DNS server thinks, this is a recursive query and refuses to answer with the RRSIG, breaking validation of that record.
Do you get to the same conclusion? Can you resolve this host via any other DNSSEC validating nameserver?
I had no success contacting any technical inclined staff willing to look at the issue since the issue started on 16. December via hostmaster@swizzonic.ch by phone or via support@register.it. So if anyone from Swizzonic is reading here, it would be nice to get a direct contact to further investigate that issue.
Mit freundlichen Grüssen
-Benoît Panizzon-