Hi,

We are running a small mesh consisting at this point of 3 Linksys nodes, 6 Xagyl nodes (xagyl.com, OpenWRT 12.09, olsrd, manual configuration, 1W output) and one VM with Debian Linux as a VPN concentrator. Ever since we have migrated the mesh to BBHN-v2 and firmware 1.1.2, we do experience the troubles somewhat similar to the ones described in your news bulletin, and all of them seem to relate to olsrd_secure module.

1. First just like it was mentioned in the other thread, the Debian VM did not connect to the mesh with a 64 bit version of olsrd, and it was specifically olsrd_secure module that did not want to connect. Once I have rebuilt the VM with the 32 bit version of os, everything connected just fine with the same config copied over, so please consider this theory confirmed.

2. However this was not the end. Once in a while olsrd crashes at this VM. After running it with strace, it showed that it crashes with the "stack smashing detected" error in olsrd_secure module right after "[ENC]Received timestamp 14096805" message. You can google "stack smashing detected" message, there is a nice explanation to it on the web. This might be the same case with UBNT problems.

3. Another thing happens on all the nodes, including Linksys. After some point they start ignoring OLSR packets from their neighbors. olsrd is still running, but when restarted with "-d 4" parameter, olsrd_secure keeps saying: "[ENC]Timestamp missmatch in packet from 10.251.51.50! [ENC]Rejecting packet from 10.251.51.50". Strange thing, it does not help to simply restart olsrd, you need to reboot the node to fix this. Also, most of our nodes do not have access to NTP servers which might or might not be necessary for the stable operation of olsrd_secure module.

I could send you the debug logs from which I quoted the error messages if you really want to troubleshoot olsrd_secure. However I just believe the whole olsrd_secure thing is no good. This module is known for having bugs, just remember the old version of it that came with BBHN v.0.4.3 firmware - due to a bug in the module code it was incompatible with the later versions of olsrd_secure. I also wonder how many networks have actually changed the default encryption key in all of their routers? For running with the default key adds zero value to security, but brings a lot of headache.

The reason I am writhing this is to maybe help you with your troubleshooting, but also to ask you if you have any short-term plans on releasing the new FW version without olsrd_secure. Otherwise we'll probably be rolling back our network to v.1.0.0 for now.

Thank you and 73

Anthony VA3IDL

1) We have received confirmation on that from olsrd that it is not recommended on 64bit. Basically this makes this an OLSRD bug, but considering all BBHN hardware currently is 32bit we haven't focused development resources on this yet.

2) We would be happy to receive the data on this output. The more data we can get on the subject the closer we can get to resolving these tickets and get back to moving forward.

3) Sounds like it could be BBHN->ticket:62 which is  being worked on (though at last check it had been identified and was just looking for testing and final solution)

Quote:
"I also wonder how many networks have actually changed the default encryption key in all of their routers? For running with the default key adds zero value to security, but brings a lot of headache."

I believe several items in play here where value still comes in even if one doesn't change the key.

1) We want the software to be there and available for those networks whom need to have some control over who is allowed to access a mesh but still needs to run under part 97  Best way is to leave the module always on so one only needs to change the text in a single file. This goes with the goal of keeping the configuration simple.

2) This module also acts as a reasonable method for ensuring NON Amateur radio operators are not operating on your mesh environment (USA Part 97 we are not to respond to devices not operating under part 97, I can't speak for the rest of the world on the regs related to this). This is just another method of making sure we do our due diligence of ensuring we talk with only other hams (You know the secret key so you should know the rules behind using that key is the logic)

Most of the issues were seeing now seem to be related largly to the fact were now creating bigger meshes.  Were in that pain point where adoption has finally grown to a point where what once worked no longer does work.

It takes time to fix the issues and fix them right but that is what we are committed to do. This is not the old BBHN where issues were large issues (like the byteorder bug) were not dealt with for 2 years, we are actively working this set of issues to make BBHN and its related software stronger. J

As for a new release without OLSR: I can't speak to that from a project standpoint. I can say though I know making one image does take away from the time that can be used to work the issues and it becomes a balancing point  to resolve issues vs just 'rolling back' I'm sure the subject of how to handle all this is being looked at and considered from all angles.

Conrad,

Thank you for your answer. Where do you want me to send you the files?

Quote:

This is just another method of making sure we do our due diligence of ensuring we talk with only other hams (You know the secret key so you should know the rules behind using that key is the logic)

Whatever comes default takes no thinking. Most people don't know the secret key nor the fact it's even there. And if a non-ham wants to talk to the mesh, they have to download the software from this site. Without the firmware you can not join the net. And this firmware would equally let them in regardless whether the mesh is not using a key, or has a key that already comes with the firmware. Anyway, I made my point, but the decision is with you guys so I don't think we should argue on this one any more.

Andrew, I'll second Conrad's statements. We still have confidence that these issues with olsrd can be worked through to provide a suitable connect security that fulfills a need in our community. The alternative is to wait 1 to 2 years for the olsr.org community to create olsr v2 implementation that they are working on now. We know that secure plugin is "not maintained" by olsr.org developers.

Note, it is also an option to comment out the secure plug in in the olsrd config file and take it out. This enables use of the 1.1.x features, e.g. 5g, DtD, fixes, etc.

So far we've found a 32bit overflow (fix already in 1.1.x images), an uninitialized variable (just compiled a linksys fix 2 days ago to test), and we have renewed focus on the crash issue.

The description of your environment sounds like you may be able to contribute? If you can reproduce an olsrd crash, please run valgrind - http://www.olsr.org/?q=valgrind . I am working to build an environment with sufficient memory to do this. Of particular interest is a lead from a coredump we have that occurred in file descriptor functions. There is a valgrind option to report file descriptor usage. If the array length is exceeded in the code of using too many socket/files opened, stack corruption occurs like you are seeing. Need to rule this possibility in/out... Please CC me on the other logs and continue discussion directly at ae6xe@cox.net .

If you are not running this on a production (served agency) network a beta release of version 3.0.0 has been posted to the Software Downloads section of the website under "Experimental Builds".

3.0.0 uses a V3 protocol so it will not link with older networks.  It is believed to resolve a number of issues (including a removal of secure module while we spend more time looking into the issues)  however as a beta build it has not been fully tested so unknown issues may exist in it so if you have a few nodes you use for testing you may want to install it with those to see if it will fit your needs.