Interesting story about Jacob Collier, online collaboration, and video/audio sync.

[Doug Young]

You're hearing Jacob and Tori at the same time? Of course. His sound is going to her. She's then singing and she's streaming what's on her end. It's just as if she's singing along with a recording of him and broadcasting that, except that he's streaming it to her instead of having it pre-recorded.

What we don't know is what Jacob hears. He says at the beginning something about how he'll just ignore her and trust that she'll stay in sync with him.

He'd being hearing her, at best, with double latency - his latency to her, she sings, then her latency back to him. With half duplex, however, he may not be hearing her at all, or maybe only erratically.

The half duplex cuts back and forth quickly, to allow back and forth conversation (but not simultaneously). In my experience using various chat programs for lessons, sometimes it seems like you can hear both ways, but more often it's cutting in and out, sounding like distortion or breaking up, cutting in and out as the program locks on one or the other. This also happens when just talking. You get the illusion that you can talk naturally, because the system can switch even between words. But every now and then, you realize the other person is being cut off when you say something. Music is tougher because notes then to be sustained.

The other thing we don't know is the latency between them. if it's 20-30 ms, then ignoring the half-duplex problem, they can interact pretty reasonably. As it gets longer, it gets harder. A nice thing about JamKazam is that it tells you the latency between all participants, so you have some idea if it's even possible to play together. I've seen 20ms (great) as well as 400ms (almost half a second), totally impossible to play "together". (tho one person can play along with the other, in a follow the leader fashion, as long as the "leader" can ignore the delayed sound of the follower. I've heard a lot of sessions where it's not clear if everyone just has terrible time, or if the latency is scrambling everything :-)

[lkingston]

On Instagram Live, everyone seems to be wearing headphones, so it probably is full duplex. In fact Tory Kelly was wearing Apple wireless earbuds, so she was experiencing even more latency. Though I noticed that Instagram was using the phone rather than the Ear Buds mic.

I know that my Android phone has way more internal latency than my iPad or an iPhone, so it probably works best if everyone is on an IOS device.

[Doug Young]

Originally Posted by lkingston

On Instagram Live, everyone seems to be wearing headphones, so it probably is full duplex. In fact Tory Kelly was wearing Apple wireless earbuds, so she was experiencing even more latency. Though I noticed that Instagram was using the phone rather than the Ear Buds mic.

Possibly, I can't find any info on it. (not surprising, none of these systems were designed for music, so it's never been a concern.) Headphones is a good idea in any case, someone trying to sing while listening to a tinny phone speaker doesn't sound like a recipe for success.

[dhalbert]

My guess is that the people in these online collaborations wearing headphones are not hearing each other. I think they may be listening to a click track or a pre-recorded performance, which they keep time to. Or, it is the upstream collaboator, which they add to, and send on down the line to the next collaborator. But that would add noise each time. So I was thinking it's a time-keeping track.

[Doug Young]

I'm not sure that's what's going on in this particular video, but yes, in general this stuff is being done offline. I got an email the other day titled something like "The secret to doing real-time online videos" from CD Baby. Excited to find out if there was something I was missing, I clicked - and of course the "secret" is that they're not doing it in real time, it's videos sync'd up later. It can be quite convincing of course. I got a kick out of the article in Forbes, of all places, speculating about how the Stones did their performance on that benefit a few weeks back. They seemed to think it was live (it wasn't).

Here's an example I just did with Anton Emery, which he just posted in Show and Tell:



We aren't listening to a click, but Anton started it with his rhythmic banjo, and for all the remaining tracks we were just listening to that as we added parts.

Incidentally, yet another way to do a "live" collab video is to really play live over a chat program, but for each person record the audio locally, the sync it up with the video later. I did a podcast recently, where all participants recorded our local audio, then sent recordings to the interviewer, who could sync them up. The goal there was to not all sound like we were on the telephone - better sound quality. There are also tools that support this, like Zencastr, which records each participant locally for best quality, then lets you download the separate audio files to be combined for a higher-quality, no-latency version of the conversation. Of course, you can't stream in real time with this approach.

[phcorrigan]

Both American Idol and The Voice have been doing something similar for the last weeks of their shows. It's actually pretty amazing how well they pulled this off, with contestants, musicians, and background singers all working from home.

[phcorrigan]

Originally Posted by Doug Young

Here's an example I just did with Anton Emery, which he just posted in Show and Tell...

Beautiful! Thanks for posting.

[lkingston]

OK, in this video, Jacob Collier talks about compensating for the lag. Unfortunately, time stamping in TapTalk forums doesn’t seem to be supported, so I here are the times:

The conversation about compensating for the lag happens at the 33 minute mark.
They do a song together at 36:30
Then at the end of the song, Jacob Collier talk about how he compensates at marker 41:53.

Unfortunately, there is nothing that will help us mere mortals. He just says that it isn’t hard, and all you have to do is think in to timeframes at once...

https://youtu.be/hp_cpN14tKU&=32m59s

[Doug Young]

Your link is broken, should be:



So his explanation is "I have a psychological trick I use"? Not too helpful :-) My guess is that his trick is ignoring her, but who knows. Maybe he plays slightly ahead of the beat. That would work fine if the internet is steady in its latency (but it's not). Of course, he seems to be the one streaming - he starts the call, so however he plays, ahead, behind, she'll hear his delayed, and then he'll hear her back delayed some more, so I'm not sure how that helps.

But here's a question for you to ponder - suppose there was a 3rd performer on the call. He's got say, a 50ms delay with performer 2, then 120ms delay with performer 3, how does this trick work? There's a reason where when this situation occurs in the real world, like say an orchestra with 100 or so feet between performers (i.e. 100ms latency) that there's a central coordinator - the conductor, and they don't just rely on all the musicians to compensate for delays.

You might find this article useful, BTW:

https://medium.com/@calebdolister/wh...s-44260789a721

BTW, to add to the puzzle, there's an irish guitarist on facebook that has dozens of jam duets like this, with people from all over the world performing, perfectly in sync. Exactly like this, some discussion, seemingly everyone just hanging out at home, and then they start playing. And it just works. Some minimal detective work indicates there's post processing involved, my guess is that each side was recorded, and he synch'd the sound later. The video may be live, but the sound is not, or rather, the sound gets bumped by a few ms to get it in sync. The sound is also a bit too good, no dropouts, none of the stuff I hear when this is really done. But again, I don't know. Maybe this guy has the secret, and isn't telling either.

My suggestion is to try it. Glad to hook up with you and help with hands-on exploration.

[Doug Young]

BTW, the trick may be something a bit technical, and not so "psychological" :-)


For example, I'm working on a setup right now, where I discovered for various technical reasons that my video and audio aren't in sync. There's an easy fix with my computer setup - I can add a delay transparently to the audio to compensate.

I could do exactly that in this case as well - if I was recording the video, and the audio, you and I could connect. Say I kick off, and you play along. Our latency is 50ms, say, so you hear me 50ms after what I play, I hear you an additional 50ms later. (100ms total off ) I could actually set up my recording gear to record capture the video, delay my sound (recorded) by 100ms, and you exactly as you come back (but delayed by the internet by 100ms). I could stream the result and we'd sound in sync to listeners. The only trick - the psychological part - is that I'd have to ignore that you're coming back to me delayed. The video might be off a bit, but that tends to be pretty forgiving. This would take a bit of configuration with a decent audio setup (and I'm thinking out loud here, haven't tried it), but Jacob's a pretty technically astute guy, with lots of recording and gear chops, so I'd say he could manage it.

[lkingston]

Who is the Irish Guitarist? I'd love to check him out.

[Doug Young]

Ah, I found it. The guy's name is Jim Murray.

https://www.facebook.com/Jim-Murray-...72310489517652

If you watch carefully, you may see a few cues that things are not as casual as it appears. It sort of looks like facebook live, but it's not marked as such. Sometimes there are fades between people and scenes. There's no startup fiddling with anything on either end, so the video is trimmed. and so on. It'd be nice if he said something about what he's doing, but I see no info. Looks like he does one of these every day.

He also has a website here: http://www.jimmurraymusic.com/

[lkingston]

An interesting thing about the “broken link” is that it works fine on my iPad Pro, but not on my Android phone.

The link has a time stamp on it, and while on the iPad, the link doesn’t take you to the time stamp in the video, it does take you to the video. However, on the Android phone it shows up as a broken link. Doug, what do you browse this forum wit? It would be interesting to see i the time stamp works with Macs or PCs, or if the link works or is broken on each.

[Doug Young]

Originally Posted by lkingston

An interesting thing about the “broken link” is that it works fine on my iPad Pro, but not on my Android phone.

The link has a time stamp on it, and while on the iPad, the link doesn’t take you to the time stamp in the video, it does take you to the video. However, on the Android phone it shows up as a broken link. Doug, what do you browse this forum wit? It would be interesting to see i the time stamp works with Macs or PCs, or if the link works or is broken on each.

I don't think any of that stuff works with AGF for embedded videos. You have to just put the video key, not a URL inside the You Tube tags. I'm using Chrome on a mac. You should be able to post the raw url, not embedded, but you were missing a key in the url and the form was wrong. It'd have to be like this:

https://youtu.be/hp_cpN14tKU?t=32m59s

Note the "?t", which is missing in yours.

[lkingston]

I originally tried the "?t" and it didn't work with my iPad. The format I used works with the iPad but not my phone or your browser. I guess the best thing is just to link to the video but avoid trying to put in a time stamp. Interesting.