Q1. ​The work you are doing is impressive, what have you on purpose decided to not implement for OAK-D that you think would work and be useful?:

Brandon: That’s a really good question. We’re not doing anything that has to do with security cameras. My thought behind that is like consumer electronics. You know I’ve known a lot of folks in my circle or not in my circle they’re like consumer electronics. I’m gonna do that like I know about my home in this thing that’s annoying to me in my home. So do you and so does everyone else. To me, the security cameras it’s kind like the first thing that everyone’s like, oh man AI would be useful for security cameras. It’s like yep you and every other smart person has thought of that and there’s this huge oversupply of intelligence and time being thrown at making security cameras, which just means if you throw your life into doing that or your effort in doing it, you’re just likely not going to get many fruits from your labor. You know if everyone’s all doing exactly the same thing all the time, unless you need a bajillion of them, which security cameras you kind of do. You know it just means to be hyper competitive and really difficult and you’re just not adding value and so you know we got a lot of initial pull in like, well you should make security cameras and so that was probably like if you were to rand the things of like the thing we were asked for the most and then I didn’t do it’s that right? There are some other things like, you know multi-camera like real-time stitching that actually our platform would be good for in and I think it could be a great feature but it’s a really high ask and it kind of like a limited audience we’ll probably do it later but the thing that we’re really avoiding that gets asked for a lot is building a security camera out of this.

Q2: For the hard synchronization, what is the limitation speed it can catch?

Brandon: We actually have data on that. Let me see if I can pull it up so I can just tell you actual numbers. It’s sub millisecond for sure. So the worst case is with like rgb cameras synced to grayscale that actually don’t have hard sync and so we do a firmware equivalent of hard sync in there. So this is the worst case offset but this shows the left right as well so actually just share this. So we just updated this with our gen1 api and then we’re bringing and improved version from our gen1 api for the hard sync. So it’s tricky because these differences are in milliseconds so this is 0.7 milliseconds, so 785 microseconds and this is when the thing first boots up so it hasn’t had time to really start the synchronization. So you can see that it kind of oscillates around a little bit after 12 frames or 16 frames movie or under 100 microseconds and after that looks like the worst case is maybe 70 microseconds between rgb to grayscale and right to left is what has the hardware synchronization. This comes down to measurement error as well they actually might be better sync than this but you’re looking at 11, 10, 9, 12 microseconds but it’s that comes down to our capability to measure how offset they are and then this is a view of the sink so the left and right then grayscale of literally just you know pointing at a device or at a timer.

Q3. You have selected a very challenging north star and you are well on the way. Could you provide your best estimate of the final size and battery life of the commute guardian for bikers?

Brandon: That’s a good question. This gives an idea of just kind of like the electronics heatsink. You could just build into an enclosure but to just give an idea. When you put it on a bike this isn’t really like the form factor you’d want it’d probably be, you know because generally on a bike you just want it to like adhere to the seat post so this is about as wide as a seat post. This is closer to the form factor and this has everything that you’d need when you build it into an actual like commute guardian you just make the commute guardian out of aluminum. Similar to like an OAK one, this is completely aluminum, so to give you the size, so imagine this but a little bit taller. What we’re thinking is without any optimization so you’re just running the thing full out all the time so you’re doing all these features all the time when you don’t need to, you’re looking at probably 10 hours of battery life out of a worst case out of like a 10 amp hour battery. You’re probably looking at higher than maybe like 15. 10 hours maybe is a good estimation but the thing that we want to add as well which is coming like q4 this year to the api is some more intelligent, like power thresholding modes. For example in the commute guardian, you could do an image classifier to start that’s running that takes way less power you’re not calculating 3d trajectory or platform motion or anything you’re just doing an image classifier at some frame rate, maybe five frames a second, which just detects if there’s a car behind you and when you do something like that instead you could very reasonably extend the battery life to maybe 20 hours. Because it just sipping power, telling is there just a car in the scene and when there isn’t, obviously doing a lot less it’s just effectively being a bike light with a low rate image sensor and once there’s a car in the scene then it can kick into like overdrive of doing spatial sensing, platform motion estimation, trajectory estimation the vehicles, and h.265 encoding. All those things that they can push this more up to one amp pulling off the battery. But when you’re not doing those maybe you’re taking like 50 milliamps or 100 milliamps, so I think the initial version would last maybe 10 hours. We think probably early next year it’ll be reasonable to have like production like prototypes of the actual commute guardian maybe earlier if the chip shortage isn’t super painful and we would not be building our model directly but we’d be working with a company enabling them to build it anything we build directly though we will just open source to make it easier for any company that’s building that sort of thing.

Q4. Which algorithm that you have implemented brings the OAK-D to the edge of its capacity?

Brandon: What really pushes it to the limits? That it’s a great question because there’s a lot of neural models that can actually be optimized to then allow you to go further. But in terms of like maximum limits, there’s a lot of different ways to push it to the edge. But what it really comes down to is when you run out of DDR bandwidth on the chip which is pretty fast, but you know one example is the encoder. The maximum encoding is eight megapixels at 30 frames per second so you can do a 4k video and dual 1 megapixel video. So it’s 8 megapixels plus dual 1 megapixel at 25 frames per second and then that starts to saturate the DDR bandwidth, and so then an object detector would run maybe at 20 frames per second or something like that. That’s one of the ways to maximize, to push the chip in terms of power and utilization.
I think you can run depth at that speed as well. Another is just straight up the neural networks, running some of these more complicated pipelines. Say a good example is we have this machine safety where it’s tracking where your hands are in physical space and then tracking some other object that’s dangerous, and we’re doing depth realignment, as an example on that we are running it on the RGB, and realigning the grayscale depth to be centered on the RGB. So you can get the best of both worlds, and doing multi-stage, multi-model starts to push the system where we are, maybe you’re a tad under 30 frames per second, but the interesting thing on a lot of these is that’s kind of like the naive way to use our pipeline and we’re using it intentionally in that way to push it, and it’s simpler as an example. But you can do things like lowering sensor resolutions and focusing on regions of interests and stuff like that, that then would make that pipeline actually not push the system at all. And that’s one of the things that we will be implement, like maybe like Q3 to Q4-ish is allowing those like hardware reconfigured of image sensors for, say if you’re tracking a hand. The hands can only move so fast in the frame, and so if you do the region of interest reconfigure the grayscale cameras for that region of interest, then they can actually run at much higher frame rate while taxing DDR and taxing the neural networks, everything a lot less. And so this pipeline when it’s naively created, maybe runs at 30 frames per second, that’s about all it can do. When you do those sorts of optimizations, then you can get, you know, 60 or 120 frames per second out of it, and actually have, you know, head room for other things. So it’s very multi-dimensional. The pipeline builder is one of the fundamental infinite, sort of, so there’s no rigid structure your pipeline has to be in. It’s only limited by resources, so do you run out of DDR because of the size of their role models usually that’s not what you run out of. But in some cases, if you have a very large set of models that could be usually you just start to run out of performance, and then it’s a matter of like, how do you optimize this by passing fewer pixels in or more intelligently cropping or reconfiguring the sensors. So it’s a bit hard to answer, but hopefully those give, you know, various limits.

Q5. Data Path, what current restriction does it have?

Brandon: Restrictions on data path. So the image manip node is how you convert between various formats going between, say you know, a feature tracker node or a neural inference node ,and so forth. And that’s one of the things that’s probably the most limiting right now, is it only supports specific image formats, and so we’re planning on improving that soon here to support, like various nv12 operations where a lot of its RGB right now, it does support grayscale operations, but to build to interchange between all these formats. So it is a network on chip architecture, and it has a specific CV manipulation capabilities the image manipulation node is our way of having a performance node. It does all these conversions for you, leveraging the hardware and so the hardware does support these things that we do not yet support in the node. But there’s been more critical kind of bugs and blockers that have been preventing that, and so for most customers we’ve unblocked them via another route, but the image node, let’s see, I probably have it up, is one of the last things and that when I said about the kind of the gen one feature release is it. This I’ll share my screen here. This extend image manip node…this is funny, your internet only stops working when you’re doing a screen share. So env12 to nv12, and v12 to RGB, RGB to mv12, RGB to gray, you can actually do that now, but it’s not as well supported as we want and upscaling. So this is probably the most limiting thing in the data path right now. There are a slew of ways to work around it, but it’s one of the highest priority things left from our phase one motion estimation’s mostly done. It should probably be an in-progress and then the extending image when it puts. So we’ll probably start on that next week after, and then it’ll be mainlined with a month, I think.

Q6. ​Connecting 2 Oak-D for a complete 360 spatial view?

Brandon: So we have the OAK-D but then we also have the modular camera, so let me see. I’ll bring up some photos here so if you’re wanting to do a full 360 degree capture, I would actually recommend these I’m gonna really make use of screen share here, so we have this modular version and then we’re extremely happy with Arducam uc tronix and so they’ve made custom devices for our platform, so with m12 with autofocus, m12, which is what you see on the bottom here and then we also, it’s not as clean as we want it to be, but you can technically connect multiple RGB so you could connect multiple 12 megapixel or multiple OV-9782 or AR-0234 which are 2.3 megapixel color global shutter so if you’re wanting to do a 360, I would actually recommend this board and/or, we actually have this is our R-3 it’s our oak ffc-3p, meaning for three ports, and then we have a four port version coming out which are actually ordering tomorrow, but you could use the three point port for now or multiple three ports. So the four ports largely are the same thing that you just saw there, we moved the ports to the bottom so they’re a little easier to connect and since there are four of them, and so this you’d be able to hook four cameras. You can with RG-cam modules you could use a 180 degree M-12 for maybe 120 degrees so you have overlap between all of them to give 360 degree view. So that’s what I would recommend you could definitely use an OAK-D, but these are only like 70 something degree horizontal, and you can actually buy RG-cam modules that like if you’re like a hacky type of person you could take this apart, take the front cover off and put RG-cam’s 120 degree horizontal on OAK-D and use it and recalibrate it, because we support mesh calibration, but I would probably recommend just kind of building a rig with these modular versions. Project Northstar which is open source has has done that, for example they use three cameras with very wide field of view.

Q7. ​Scooter (commute guardian). More stable than a bike, and here in Taiwan, there is a lot, any possibilities for implementing?

Brandon: I would love to see this on, it’s like scooter commute guardian. I’d actually like cold reached out to, I don’t know, like probably 10 different scooter companies but they don’t know me and our platform wasn’t nearly as far as it is now, with the idea of like you could definitely build this in, it’s more stable, that’s a really good point. And I love Taiwan by the way, I’ve been there multiple times, three times actually. I was there in January 2020, so the last time I could like really get out there, and so I’m familiar with the scooters and and I love scooter culture and everything, so yeah I think mainly it’s just getting connected with the company, so if you guys know a company that would be interested, you know I’d love to talk to him, integrate the technology, you know as I kind of mentioned in the presentation a lot of the stuff is already there, you could in this particularly with scooters since it’s a more stable platform, you could get an early alert system and at the very least you could have a really useful kind of like incident logging system already where it’s like this happened and I already have a log of the license plate 4k video of the incident and those sorts of things.

Q8. ​In terms of energy (or other restrictions), what is needed to implement the OAK-D on a drone?

Brandon: Yeah that’s a great question actually so the Myriad 2 was extensively used in dji drones, which I think is a great use of the chip’s capability in the Myriad X as well, I think fits really well, so the power modes vary right now. Our API is very basic in terms of power mode, the Myriad X is actually quite capable in terms of how much you can vary the power use, because it’s power island system, so you can literally turn off whole sections of the ship. I believe it’s around 30 parts of the chip are programmatically controllable to these actually fully powered down, but right now we don’t expose any of that, we plan to do that at the end of the year, so the only like power control right now is just how fast you’re running cameras, and then that just kind of slows everything down so you can get it to go as low as maybe 2.5 watts on an OAK-D, maybe two watts, I haven’t played with it that much. And then the max is about five watts. Once we have those power control settings, I think you’d be able to go as low, you know depending on the scenario, as low as maybe quarter watt or something like that. Where you’re just running like say an image classifier or something like that and then scale that all the way up to 5 watts depending but right now it’s around 2.5 to 5 watts.

​Q9. Hi! We’re a group of bachelor’s student planning to work on a real time video surveillance anomaly detection system. What do you think about a possible implementation of this project using OAK-D?

Brandon: I mean, I think you could do anomaly stuff with us for surveillance. I would say there’s probably other platforms that will work better, kind of what I was saying earlier for surveillance. And I dig, I dug into like one aspect of it earlier when I talked about just the moment of people working in that area and so there are good chips for it. The other aspect which is a bit more technical in the Myriad X is, you know, right now the world is kind of given a specific capability, max capability for the die of producing chips. The Myriad Xes has taken a huge portion of that die and dedicated it to things that are only relevant if the platform is moving. So, you know, a lot of those features like the tight IMU integration with everything is for a moving platform, and it’s extremely useful. When you’re doing surveillance, if it’s fixed surveillance, all those become useless or mostly useless. So you look at this die, and you’re paying a certain amount of money for the whole thing, and then this whole subset of it is effectively useless to you. So that makes the surveillance, there’s a lot of other, say, umbrella chips or other platforms that do not have all those features that the Myriad X has, that aren’t nearly as capable on moving platforms, but and aren’t intended for moving platforms, but are then specifically targeted, and a lot more useful for fixed surveillance applications. So you can do it. We have a great API, maybe that’s excuse enough to one want to use our stuff other over others, so we make things easy. But what I do not want to say our things best at, that what I would say are things best at is if you want to integrate it into some tiny thing that moves that makes sense.

More questions:

Q1. With your current experience what would you do differently if you started over from scratch again?

Brandon: Do everything faster. I questioned if what we were doing would have value, which slowed down our pace and limited how quickly we hired and accomplished. I would have hired faster, developed faster – which would have then unblocked a ton of engineers a lot faster. If I were to do it again, I’d just do it harder.

Q2. The Myriad chip has a lot of features. How many years do you think it will take before your software support convenient use of all features?

Brandon: Most are now supported. The lacking portions are customer kernel support (which is technically there, but difficult, see an example here and here or with OpenCL here). And then the last bit is power modes, which we think we will support in all the practically-useful ways around the end of this year.

Q3. Could you explain Luxonis business model? How do you make money?

We’re a software company that monetizes on hardware.

Brandon: Specifically, we sell system on modules (OAK-SOM, OAK-SOM-IoT, OAK-SOM-PRO) which allow engineers to build all the power of the DepthAI OAK ecosystem into their own products. And that is where our volume sales are. So while we sold ~20,000 OAK end products (e.g. OAK-D) last year, a single customer will order more than double that of the OAK-SOM-PRO for their own products, for example. So our model is to enable those customers, from customers who order 100 modules, to customers who order 100,000 modules. And they all benefit from the platform improving, and the combined hardware scale.