Replies: 11 comments 52 replies
-
|
@boris, I will try to add some notes and commenst which may help you investigate further. You have multiple BRs in your setup within the same mesh. Note that multiple BRs can act as SRP servers within the mesh. Devices within the mesh may choose to register their service with one of the available SRP servers. There are certain rules for this, which I won't go into detail about here. The error itself indicates that the SRP server that received the request could not accept the service because the same name is owned and being advertised (over mDNS) by another entity. Some common scenarios that cause this:
|
Beta Was this translation helpful? Give feedback.
-
|
@abtink , |
Beta Was this translation helpful? Give feedback.
-
|
@3oris , are the Nanoleaf bulbs running the latest available firmware image? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, they are all on 4.1.3. At least that is the latest firmware I am aware of. Versions in CSA DCL and test DCL are even older. |
Beta Was this translation helpful? Give feedback.
-
|
If you are facing the same/similar issue with devices from the same vendor, it may make sense to investigate if there may be some issue/bug with that vendor's firmware. I know some vendors may choose to add some extra customization or optimizations which may unintentionally lead to incorrect behavior and issues.
It is configurable on the SRP client (per client and/or per individual registered service). The default values (if not specified) are as follows: openthread/src/core/config/srp_client.h Lines 152 to 169 in 111db8a |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , given the issue #12088 , the fact that that happens more often than I thought initially #12088 (comment) , plus you mention that the observations made in this conversation could possibly also indicate issues with mDNS, I might very well wait what happens after issue 12088 got resolved. That said, the OTBR is currently my only observability tool into my Thread network. It helped me a number of times, e.g. choosing a good RF channel. I usually make my OTBR the leader of the herd (for some reason it calculates the leader weight as 65 which is one higher than the nest hubs. Don't know if that is the reason, but they never refuse to hand over the leader role when I |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , I also see a lot of these: where devices just want to register a key, and that fails. No entries in And nothing to resolve in the matter service type domain. |
Beta Was this translation helpful? Give feedback.
-
|
Here an example, where while the host could not be registered, the services very well could: And it can be browsed in the mater service domain. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @3oris for doing all the tests and providing/analyzing the logs. The logs, as you also noted, indicate that there seems to be a conflict on mDNS. One aspect is that the conflict may persist in the current OT mDNS. I just submitted a PR #12128 to re-probe on a follow-up If you can maybe pull this PR into your build, it would be good to see if it helps. |
Beta Was this translation helpful? Give feedback.
-
|
I also have another PR #12129 which increases the This interval determines how long information about previously transmitted mDNS messages from the mDNS module is retained. This history is then used to detect whether a received message originates from the mDNS module itself. Any message we send should be sent back to us again (as we are also listening on the same mDNS port and multicast address). It is a rather unlikely situation, but I am thinking perhaps there are extra delays in some cases, and then an mDNS announcement from the mDNS itself is treated as if it is from another responder, causing a conflict. The original timeout was 10 seconds; I just increased it to 100 seconds. The downside is the extra memory usage (to keep the entries for 100 seconds). However, we actually just track different hashes (CRCs) of the message along with its length. So, it is not much info per message (definitely not the full payload). |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , I appreciate your persistent attention.
Going to pull these in successively... It may take a moment before I get to results. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , is there any magic btw. to I had hoped to see some key material as described in README_MDNS.md |
Beta Was this translation helpful? Give feedback.
-
|
No, there is nothing specific about the One thing to point out is that registering KEY records for the host and service instance name is part of Thread 1.4, but earlier Thread BRs (1.3.x or earlier) are not required to register them. As a result, shipped BR products (from Google and others), which I believe are still running older firmware (1.3), will not register any KEY records. |
Beta Was this translation helpful? Give feedback.
-
|
OK, that makes sense, I have 5 Nest Hubs running lastest official firmware (i.e. Thread 1.3), and one OTBR I occasionally update to latest main. When I issued the above command only Nest Hubs were having the SRP server role, hence no KEY records held. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , With #12128 pulled in I still see: But also: Wrt comment 14916689 I cannot actually tell if the recordquerier can retrieve keys only for OTBR locally registered keys, or if it should be able to show them for registrations on other routers as well. But for a successful registration: Key retrieval: and next: pull in #12129 |
Beta Was this translation helpful? Give feedback.
-
I have an idea/guess why this might be occurring (since it has no log level it may assume it is cert logs). I can try to change the code to handle it differently (forcing it to use Edit: I just updated the branch #12139. It now uses |
Beta Was this translation helpful? Give feedback.
-
|
@3oris some answer to your earlier question. Always happy to answer/share what I know:
Let me try to explain the idea behind this from the SRP and Advertising Proxy perspectives: SRP ConceptsSRP has the notion of a "lease" and a "key lease." The client sending an SRP update requests lease and key lease intervals (the key lease is required to be longer than or equal to the lease).
Advertising ProxyThe same model extends to the Advertising Proxy:
The Therefore,
Yes, definitely. But if another SRP server's Avd Proxy is still advertising key and/or service/host name, it will cause a conflict. The idea of "key lease" is for the same server to retain the ownership of the names, which indirectly requires client to regsiter with the same. one.
The
Yes. We cannot really do any of these. Regarding the offline Matter devices: if my assumption/guess is correct that we see a conflict happen after registration is successful, then it will not recover by itself. What you are observing is, unfortunately, expected behavior. We need to update the code to handle such conflicts (restore/recover and try again, or take other actions). |
Beta Was this translation helpful? Give feedback.
-
@abtink , thank you once again, that did the trick, so I have a working OTBR again. Unfortunately, it got a little late over here, so I probably wont't have results before tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , thanks again for your detailed explanations
Thanks a lot for the detailed explanation. I also think I do understand, it is pretty much in line with the RFC :)
That is actually the answer I hoped for.
I see. At the same time the RFC doesn't mention SRP servers but "SRP service"/registrar as a concept. And I do understand that the service/registrar should retain the ownership of the names, so only the very same requester can reclaim service and host names even after they "expired" (for the key lease amount of time). This is to protect the original requester from name theft by another/different requester.
OK
Let's see what the extended logs will show. |
Beta Was this translation helpful? Give feedback.
-
The RFC author renamed several terms at a later stage before the RFC was finalized. SRP client and server correspond to requester and registrar, respectively, and were the original names. I still use them, and they match what we have in the OpenThread code (which was completed well before the changes to the RFC). Changing the API/CLI names would have been a major issue, causing backward incompatibility with projects that were using them. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , I have a hypothesis (more or less guessing, I didn't look into the code very deep): Behavior without #12128:
Behavior with #12128:
Now looking at There is a small window, where This would explain why before the patch we saw devices continuously trying to register their names while at that point no other router was holding these names any longer. Of course the ot mDNS module doesn't serve names in conflict. It would also explain why after the patch registering/updating for these poisoned names became possible, but the ot mDNS module doesn't start to serve them. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. This makes sense. I think this is an example scenario where the #12128 would definitely help. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink , thanks again for your continued support -- I create a new thread here. Hosts names go into conflict right after registration I could capture 2.7G of logs and identify at least two cases where registration goes well, and straight after that (within a second) mDNS puts the host in conflict. The two cases are: and Client Then -- as observed before --, the successfully registered clients stick to this SRP server with continued successful update requests. I have stripped the log file to a time range that I thought might be relevant. If it is not sufficient I can send more. EDIT: |
Beta Was this translation helpful? Give feedback.
-
Thanks @3oris . This all seems to be good and proper recovery behavior and what we expect to see when there are changes in the mesh (e.g., BRs added/removed or mesh is partition and/or new leader is chosen).
The situation from which we cannot easily recover (today with the existing code) is detecting conflict after a successful mDNS registration and post-probe. This is effectively a misbehavior by some other BR (and its mDNS). This causes inconsistency between the SRP server and what is being advertised by mDNS. |
Beta Was this translation helpful? Give feedback.
-
|
Submitted the following PR this weekend. This should help address post-probe conflict: |
Beta Was this translation helpful? Give feedback.
-
|
@3oris , thanks again for all your help regarding this. Building custom branches, collecting all the logs, and going through them was super helpful. I am happy we got to the bottom of this and feel good about it. I submitted the PR above as a (short-term) solution to recover. Longer term, when I get some free cycles, I will also enhance Adv Proxy to help with such conflicts and try to re-register and recover periodically if a post-probe conflict is detected. Adding this in the Adv Proxy layer would have the benefit of improving the robustness regardless of which mDNS library is used, whether OT mDNS or a platform-provided one. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @abtink, for following this through to the end. This was more than I could have hoped for.
Anytime again! We may have a win-win here. Plus, I have learned a lot!
I can hardly wait ;-) |
Beta Was this translation helpful? Give feedback.
-
I am going to apply it and see what happens. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @abtink , it has bin a while, mostly due to the fact that I could not always force my OTBR into the srp server role when I wanted to. In general the OTBR works way better as an SRP server with your modifications applied. Today though, I have a scenario that (from what I have learned from you) I didn't even thought it was possible: I have a device that is being shown as registered in the mdns module, but not in srp: Is that by any chance an expected state? It is also actively responding to mdsn queries: |
Beta Was this translation helpful? Give feedback.
-
Good to hear this.
This is not a desired state, but I can see ways this might happen, although I expect them to be unlikely. If I get a chance, one area I would like to spend time improving is the handling of error situations in the Adv Proxy, specifically cleanup when the Adv Proxy (mDNS) fails (although I am not sure if/when I will have time to work on this). It is a rather tricky problem since we may receive multiple SRP Update messages from the same host/client — perhaps even with conflicting content —and we need to handle the case where any single one of them fails. For example, consider the case where a first SRP message adds a host and two services, A and B. While the Adv Proxy evaluation of this is ongoing, we receive a new SRP update where service B is removed and service C is added. We should then stop the mDNS advertisement of B, which the Adv Proxy code detects and handles. However, if the advertisement of C causes a conflict failure, we end up rejecting the entire second message. At that point, handling service B becomes tricky. I will see if we can add some simpler code regarding this, perhaps not to address this in a general sense, but to handle cases where the host is fully rejected (removed) on the SRP server. |
Beta Was this translation helpful? Give feedback.
-
|
@abtink Thanks for the explanation! Unfortunately I don't have logs available for how this particular situation came into existence. Due to the sheer amount of data the verbose mDNS logger creates I cannot let it run always for just in case something happens on my little Raspi 3b. I will keep an eye open though, maybe this is reproducible. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @abtink , I have a question: Today My assumption was that in this situation registered devices are being announced by the ot_mdns module, so they probably are in conflict on all currently active srp servers and cannot re-register somewhere else... |
Beta Was this translation helpful? Give feedback.
-
|
Sigh, I guess I have to re-think what the question is... |
Beta Was this translation helpful? Give feedback.
-
|
@abtink -- Thanks again for going through this together. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good to me. Thanks again, @3oris, for all your help with this. It was a pleasure working with you on this. Wishing you a happy New Year! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the bug
Yesterday, 11 of my 125 thread devices became unavailable in my matter controller, so I gave them a power cycle.
At that point in time the following started to show in the OTBR logs:
Over the course of the day the light bulbs did not come back online in matter while these messages continued to appear.
Today I restarted the border router at wich point another border router took the role of the SRP server which immediately solved the issue.
I quickly did a search and found: #8056
While I do understand the explanation in general, I do not understand why this all of a sudden can be a mass phenomenon with about 10% of my light bulbs. Like all of them losing their SRP Keys in a short time frame (while many/most others do not). Is that somehow rooted in the design of SRP?
Additional information
OTBR_OPTIONS:
IEEE 802.15.4 hardware platform: EFR32MG21
Git hash for OTBR is ca4ba9ae63e963f62277a2d847c9e3a5dee5ab20
Network topology:
Beta Was this translation helpful? Give feedback.
All reactions