Hey Bart -
Good to hear you are changing things up but if you can, do not rush this. It's a PITA to migrate to new software so you don't want to do this again.
First of all, a great expert community on a poorly designed forum will ALWAYS beat a slick forum with bad content. You fall into the first category so you have a great foundation. Sounds like you are getting some leads on software, I am not familiar with too many so I can't help there.
However, as a product guy that makes consumer software for a living, I can suggest how to help quickly understand the needs of your members and select a solution.
CRITERIA
1) What activities that most interest your members? For me it would be: Finding information, tracking topics that I am interested in, participating in discussions, connecting with people.
2) How do your members use the site? Desktop, mobile, email..
3) What are the top features that support the needs & usage patterns? For example, Search (super important and most forums suck a this -- i would invest a lot here), Navigation, Browsing, tracking discussions, PM system, classifieds, mobile friendly site, overall ease of use, etc.
You could learn all of the above with a quick member survey (survey monkey is great because people can select or rank a list of 10 things you give them) and google analytics.
SELECTING THE SOFTWARE AND HOW IT SHOULD BE CONFIGURED, ETC
4) Make a spreadsheet to score the software packages out there: criteria running down, software packages running across, score each cell. Score could be high, med, low based on how well it addresses the need out of the box. You want to avoid as much custom coding & design as possible.
I have a template if you want -- it even weights criteria. Not as complicated as it sounds, actually very easy to use. The hardest part is identifying criteria and software packages.
pete