Interesting story about Jacob Collier, online collaboration, and video/audio sync.

[lkingston]

Today my 19 year old son got really excited and showed me what was currently playing on his phone. He’s a student a Berklee and one of his friends (an extremely talented girl singer) had done a Jacob Collier cover and it had caught his attention.

He did a spontaneous online collaboration with her live with both where they were both singing and he was also playing guitar. Of course it sounded great, but what really amazed me was thinking about how they could possibly be doing that live with the half second or so of Internet latency that always plagues this type of thing.

My son’s answer was just that Jacob Collier is so talented that he was probably just compensating for the delay mentally!

For anyone not familiar with Jacob Collier, that might seem like an impossible feat, but the young man is probably the most talented person alive. I’ve seen videos with him doing things like tapping 21 against 22, or tapping out 5 completely different independent rhythms: on with each finger on the same hand. His use of harmony is even more impressive. It is entirely possible (and in fact likely) that that is EXACTLY what he was doing: anticipating the delay and compensating for it.

What made it even more amazing is that he seemed to be following her lead...which would mean that he was anticipating what she was going to do next a half second or so before she did it!

Anyway, I thought that might be of interest to a few of you here! The live video feed was so natural and spontaneous, and didn’t have the feel of any effort going into it at all! It was done for an extremely small audience. My mind is blown!

[lkingston]

Thinking about this a little more:

When we do the AGF Zoom session and it gets streamed to Facebook, you can be on the original Zoom session and also see it delayed on the Facebook stream. On Facebookit is delayed, but everyone is in sync. If you were to use the mic from Zoom, but monitor it from Facebook, you would be in sync with the other performers, but the price would be the delay (latency). Mentally, working around this would be quite a challenge!

I remember when I was a teenager, the church organist at my mother’s church let me try the pipe organ. It was the first time that I realized how much delay there was between when you press a key on a pipe organ, and when you hear the sound! It was about a half second: really similar to Livestream latency. On the pipe organ, it just takes that long for the key to trigger a relay, which triggers a valve, which guides the air down a length of pipe, which finally makes that enormous sound! Suddenly I realized what a skill it is to conduct the choir with one hand at one time, while playing with the other hand and left foot a little earlier in time...and yet at one time most church organists had acquired this skill! Organists have told me that it just a matter of ignoring what you hear and plowing ahead!

None-the-less, being able to judge the latency and compensate mentally would be extremely difficult! To do that while listening attentively to a part that was largely improvised, and to respond soulfully (all while compensating for the delay) impresses the heck out of me!

[MikeBmusic]

Before I agree about Collier's time-changing ability, I would want to hear the result, and also find out what the hardware/software set up was.
I suspect that any communication was 1-way only at any particular time, and that the 2 parts were combined in software like Acapella.

[Doug Young]

Do you have a link to the video?. I have a suspicion about how this was done, but it might help to see it.

[lkingston]

Unfortunately there is no link. It was live for a minute and it was gone.

I'm pretty sure it wasn't a matter of only one person hearing. They both were interacting like they were in the same room together. I am highly aware of that because I've been trying to figure out a way around it (the latency).

I'll look for another example.

[lkingston]

A while back, Jacob Collier did this thing at MIT where he had some sort of way of signaling their choir notes to sing as he played them from his keyboard. I'm sure there were similar latency issues with that arrangement, but he seemed to handle it with ease. I really think he just is that talented.

[lkingston]

Here he is on Instagram Live with Tori Kelly. It was a lot like this, but I think he's gotten even better at it since this. Last night it was Instagram Live as well:
https://youtu.be/7uxIxthDqKM

[Doug Young]

OK, the instagram example makes sense. He's leading, she's singing along and streaming.

So he plays, it arrives at her end delayed by, let's say 1/4 second.

She sings along, and to her, everything is in sync, she streams the combination and both are in sync.

The only issue is that he hears her delayed by another, say 1/4 second coming back, so he hears her way off. He just ignores her and we don't hear how it sounds at Jacob's end. He could simply turn off his sound if he didn't want to mentally ignore her. Actually, I'm not sure about instagram, since I don't use it, but it's probably half-duplex, just like facetime, skype, and zoom, so as long as he's making sound, he won't be able to hear her anyway.

He's not time traveling :-) or anticipating what she's doing, he's just blasting ahead and trusting that she's singing in sync, even tho to him, to the extent he can hear her, she sounds like she's behind the beat.

That approach works fine with two people. Here's another way to do something similar.

You connect with me via facetime. I use OBS to capture the facetime video/audio, and add me in a second window of my webcam. I mute my facetime mic. You start playing, and I play along, and stream to facebook or YouTube. We sound like we're playing "together" and we are, it's just that you can't hear me. But I can hear you, and listeners hear us both in sync. As long as the tune doesn't require that we really hear each other, and the first person can hold the time together, it's all fine. If anyone wants to try this for an AGF Open Mic, I'd be glad to give it a try. By the way, in theory, this could be chained indefinitely, adding a 3rd, 4th.. person. The last person in the chain streams it, and the only catch is that players don't hear anyone who's later in the chain.

Incidentally, you can do better than this with JamKazam, if all conditions are right. I played with someone the other night - he wasn't far away, and we both have good internet, so the latency wasn't noticeable. We had split screen video and played together. JamKazam is definitely "beta" quality, and the internet makes it even more hit or miss, but it can work, and both people can actually hear each other.

[lkingston]

I think it is possible that he is actually listening and responding to her and just mentally working around the delay. I don’t know how much you know about Jacob Collier, but he is known for doing things musically that nobody else can. For instance, if your pitch reference is off 20 cents, he can tell you that and adjust. He can sing microtonal stuff. He can tap out polyrhythms that are just crazy: 21 against 22...stuff like that. I think he is leading, but also listening and just adjusting for the delay in his head.

Like I said before, it isn’t a completely new skill. Church pipe organists have been doing it for centuries (often while conducting).

I really think that he may just that crazy talented!

[lkingston]

I’ve gone from trying to figure out how he did this, to wanting to be able to do it myself.

Is this possible in a zoom session? If I am on a Zoom session and I was to accompany a singer, would the stream from the host be in sync, or would it have to be redirected to Facebook or YouTube from the singer’s Zoom session?

[Doug Young]

It depends on what you want to do. You want to both hear each other and be in sync? No, not going to happen with Zoom or any traditional chat program. You want person 2 to hear person 1 and stream the result, yes, as I outlined previously. For a true collaborative experience, JamKazam, sofasession or Jacktrip are currently your best bet. JamKazam supports video, the others do not. Success is highly dependent on your internet connections.

Here's an article I just wrote for Acoustic Guitar that outlines your options and explains a bit about why various chat programs won't work for 2-way collaboration (latency, and half-duplex are the big ones)

https://acousticguitar.com/virtual-j...-in-real-time/

I've helped a number of people get up and running on JamKazam now - success rate is around 50% :-) Depends on your internet, your tolerance for buggy beta software and your gear.

[Doug Young]

If you want to try any of this, I'm more than willing to be a guinea pig with you. I have never used instagram, but I'm familiar with most of the other options (and I'd be happy to figure out instagram if you prefer that). One quick test I've been using - (a nod to Stevie Coyle who suggested it) - is to just set a metronome at a slow speed and see if we can both play a simple scale together and have it be reasonably in sync, and if we can both hear ourselves in time.

[lkingston]

That sounds like fun. In the mean time I’m going over your article. Awesome!

Here’s another Instagram Live duet with H.E.R. And Tori Kelly where they are really pushing the limits of working around the latency. Check out around the time 1:45 and 2:36 for a little harmony singing! It’s certainly full duplex:

https://youtu.be/79v21534Z5w

[Doug Young]

I'm not finding anything on whether instragram chat video is full or half duplex. Virtually all are half, for good reason. WebEx is supposedly full, tho I think you have to have the paid version. Google's is claimed to be full-duplex, but people report that it's not in reality. Haven't tried it. I have some friends who did a work around, basically used Skype one way, Facetime the other, so avoided the issue (tho not latency). With full duplex, you can end up with feedback issues, so you typically need to wear headphones. Headphones definitely work best with JamKazam, (which is full duplex), otherwise you hear yourself echoing back from the other side.

[lkingston]

What you say makes perfects sense, but if I’m hearing them both at the same time, isn’t that full duplex?