One potential change would be to divide the forums up a little more. When you create a thread, you declare what "forum" it is in, but because of the way the board functions, that tends to be irrelevant when browsing.
I don't know how easy it is to do, but I would suggest that it would be better if there were 2 or 3 forums that you couldn't browse simultaneously. 1 or 2 "serious" ones with tight moderation, and one "jokey" one with somewhat looser rules.
I would also like to pose a thought to both critics and supporters of the board that there are several tiers of "trolling". For example
1. Silly jokes.
2. Edgy jokes that could be interpreted as as sexist or racist, but are clearly not intended to be taken seriously.
3. Comments that could be interpreted as sexist or racist, and are intended to be taken seriously.
4. Personal attacks.
What tends to happen is that critics focus on category 3 and 4 and supporters focus on 1 and 2 to say that the critics don't understand. The reality is that there are both on this board.