I've been playing both guitar and banjo for about 48 years. Learned 'em together, so my playing style on each instrument is influenced by the other.
There's lot's of different styles of banjo playing. Check out
www.banjohangout.org for some lively discussions. I don't play bluegrass. I do a mixture of frailing (usually called claw hammer nowadays), two-finger picking, and three-finger picking (non-Scruggs) plus other stuff that could be classified as "folk banjo"
I do all this on a banjo that would be called a "blue-grass" banjo, but it don't care what I do with my fingers. Only the blue-grass and old-time police care about that, and I don't let 'em bother me none.
Whatever style you decide on, it'll take a lot of retraining of your fingers. That fifth string and the different layout of the fingerboard make all the difference in the world.