Many have noted similarities between James' Epistle & the Sermon on the Mount, and I've heard it suggested that James may have worked partly from the same oral tradition that informed Matthew's SOTM. On that model, I'd presume, Jesus' actual preaching would be the source of both streams of 'Jesus tradition'. Well, to no one's surprise, I'd like to suggest taking that one step further.
If James & Jesus were actually brothers, who grew up in Nazareth and sat in the same Synagogue together for decades, might there have been some particular Rabbi(s?) who taught memorably during their time there? If so - and if we accept Matthew's SOTM as a faithful account of things Jesus actually taught - then it might it be possible that James' Epistle didn't borrow from Jesus' teaching, but that both built upon the same hometown source, namely, teaching from Nazareth's Synagogue.
It would be intriguing to work back from both sources to try and suppose what was being taught in Nazareth. With this model, it would also be interesting to compare both texts to Rabbinic Literature and see which stream stayed closest to the original, and which was more transformational. But my bias and hunch are revealing themselves now. I'd expect to find Jesus' treatment of that material to be more transformational, and to find James' version to be more faithful to his own upbringing. But again, that's my own bias & hunch.
There's an interesting moment in John's Gospel when Jesus' brothers take an air of superiority because they're on their way down to Jerusalem and he seems unconcerned with the festival. That suggests to me that James & Jude (et al) had established a personal religiosity (that's not a negative term) quite apart from their brother's existence, and that they'd done so long before Jesus went out to begin his public ministry. That's just another reflection that might partly bear on this hypothesis.
These questions may or may not be worth asking, but I'm guessing they're new, and for three reasons: (1) It requires us to think of James as Jesus' actual brother. (2) It requires textually-focused scholars to delve into historical reconstruction. (3) It requires someone who holds a high view of the SOTM's authenticity AND who also takes James' Epistle as more Jewish than Christian. These are the kinds of reasons why I do what I do. There's an unfulfilled niche here, I know there is! ;-)
In all seriousness, if anyone knows that this has been done this before, please let me know. And if anyone else wants to jump on this topic, please do so with no hesitation. This is one issue I will never try tackling in depth. It's way too far beyond me on the textual issues.
In historical terms, however, I'll say this much with confidence. We should take it as fact that James & Jesus did share Synagogue space for a number of decades, and that Jesus' take-home from each meeting would have been quite unique from what anyone else was perceiving, and that must have included his brothers. That difference in perception *must* have played into their respective future careers somehow, and it's possible that at least some of the similarities between James' Epistle and Matthew's SOTM are due to this common heritage.
Both James and Jesus grew up at the Nazareth Synagogue, and whatever Rabbinic teaching they got, they both heard it all.
If anyone wants to spin this another way, my ears are open...