Rhythm is obviously the most important element for
dance music. Which is of course why the question occurred to you listening to rumba!
Rock music was also originally dance music (as was jazz originally, and R&B always). A lot of folk music too, of course.
Even classical music, such as minuets, sarabandes, allemandes - all
dances originally.
The less you care about your music being danceable, the less crucial rhythm becomes.
Timing is still important, but not necessarily
groove. I.e., tempo needn't be "strict", it can be
rubato if dancing is not required.
Dizzy Gillespie (whose bebop was always danceable) had a nice quote: "some people think of a note and put a rhythm to it. I think of a rhythm and put a note to it." So he clearly felt rhythm was more important than the notes.
I was once in a class run by Brazilian percussionist Bosco D'Oliviera, and he had another nice saying about the
clave (the rhythmic cell of both Cuban and Brazilian music). He said the
clave was a pattern that existed "in the air" - you just had to hook on to it and follow it. (Like one of those drag lifts on ski runs, I guess
)
Obviously notes matter, but what they're both saying is that the
groove comes first. The groove is the foundation on which everything else rests. Maybe the notes end up dominating, but without that firm foundation the whole thing can collapse.
My own epiphany came at a jazz gig, when I was listening to some pros play some quite simple stuff, and wondered why it sounded
so good. I realised it was because their timing was impeccable, absolutely precise. Even when they swung, or played loose behind the beat, every note was place on exactly the right nano-second. It's that precision of timing that sorts the pros from the amateurs. I'd had a lot of experience playing with amateurs at that point, and the loose timing made everything
tiring to listen to. You could admire some nicely melodic soloists now and then, but it all just dragged, even at fast tempos.
Technically, achieving crisp timing is partly about having a good internal clock - being totally
aware of tempo and groove - but also about
articulation. You not only have to know how to groove, but to have the chops to be able to place those notes exactly where you want them in the bar, and hold them exactly as long as you want. Chops in the service of
time, IOW, not
speed!