WEBVTT 00:00.000 --> 00:27.000 Hi everyone. Welcome to my talk. Thank you for joining me today. And the next about 20 minutes, I'll be speaking about a project we were working on for a couple of months now. 00:28.000 --> 00:42.000 Just a short disclaimer, the project is still going on, so I will not be naming specific things. I will leave out some specific details, but I will be talking about how we did that. 00:42.000 --> 00:46.000 And I think this is the most important and the most interesting part of the whole story. 00:46.000 --> 00:58.000 So this is a real project as I said, and this makes it so interesting for me, I think. I think also for the community and for everyone working on Android. 00:58.000 --> 01:07.000 Understanding the implications of devices being outdated or being updated on regular basis is super important here. 01:07.000 --> 01:14.000 So my name is Igor. I am working with Android for 15 years now, I think. 01:14.000 --> 01:21.000 I was one of the common trainers for Android, one of the original Android distributions for Raspberry Pi. 01:21.000 --> 01:34.000 Many years ago, I think it's eight, nine years ago, and about seven years ago, I started a new company in Syria, where we basically help companies build new products with Android. 01:34.000 --> 01:47.000 It doesn't matter which type of products, automotive, dashboard, IoT, whatever. If there is a screen, if there is some reason to have Android on it, and the company doesn't have the resources to do it themselves, we help. 01:47.000 --> 02:03.000 That's all my thing, and by help, I mean, we're doing initial onboarding, we do audits, we do automatic builds for overday updates, we have an infrastructure for distributing updates for remote management. 02:03.000 --> 02:25.000 For many, many different things. And here, the main idea, as I said, to keep the devices up to date, but this is really a hard thing to do, because many manufacturers are not really very interested in keeping the software up to date, because they make money by selling hardware. 02:25.000 --> 02:46.000 So, if you are a hard manufacturer, and you want to sell as much devices as possible, and also new CPUs and new system on chips, you really don't focus on keeping up with all that speed of the Google pushing the Android releases. 02:46.000 --> 02:54.000 So, you stay low, and you give your customers whatever they want for the initial release, and that's it. 02:54.000 --> 03:01.000 But, obviously, this is very important, right? So, we want the devices to stay secure. 03:01.000 --> 03:14.000 We want to know what we want to be compliant with all the legal obligations. Many of you probably know that, I think last year, the cybersecurity resilience act came into power, 03:14.000 --> 03:22.000 meaning that in the next year's companies were building digital products will have to provide updates to their clients. 03:22.000 --> 03:38.000 So, there are some regulatory and legal implications for that, but also there are economic and environmental benefits of keeping devices up to date instead of throwing them away and building new ones. 03:39.000 --> 04:05.000 Actually, while I was working on this presentation, I learned that building a single smartphone generates something between 30 and 70 kilograms of carbon dioxide, meaning that if we are talking about the fleet of devices of several thousand, we can save what 50,000, like 50 ton of carbon dioxide just by one project. 04:05.000 --> 04:10.000 It's kind of a cool thing to know, I didn't know that when we started this project, but I know now. 04:10.000 --> 04:28.000 So, we were approached by a company who has a fleet of devices. They purchased those devices from a manufacturer directly, and many years ago, and they were working fine and everything was well. 04:28.000 --> 04:39.000 But today, they can be considered outdated. So, the manufacture of those devices basically refused to provide any types of updates, any types of upgrades. 04:39.000 --> 04:49.000 So, the position our clients were in is either to drop them and purchase new ones, somewhere else, or to try to upgrade them to something newer. 04:49.000 --> 05:01.000 And we were talking about several thousands of devices. So, it's not just one or two or three. Maybe this little bit too much work for just keeping your own smartphone up to date. 05:01.000 --> 05:12.000 But I think this is the main story behind lineage. So, a community helping each other to maintain a set of devices up to date. 05:12.000 --> 05:20.000 But here, we were in the position where it was a dedicated custom build device, and the question was, can we actually upgrade it? 05:20.000 --> 05:30.000 So, the project though was, can we move it from Android 9 to something newer, whatever it can be, and by this also reduce the dependency on the original manufacturer. 05:31.000 --> 05:41.000 So, what was the device? We got one sample in our office to take a look at, and we learned that there was no schematic available for us. 05:41.000 --> 05:48.000 Actually, it would be available, but we would need to sign a ton of NDAs, and we were not interested about that. So, we said, no, thank you. 05:48.000 --> 05:59.000 What we knew is that the UART is available, so we can get actually bootbox from the device, and we knew that we could be able to unlock the bootloader and use the fast boot to flash. 05:59.000 --> 06:09.000 That's already a pretty good start, I would say. So, it's better than jail-breaking and routing most of other smartphones on the market. 06:09.000 --> 06:24.000 When we got the device, we learned that it's quite popular SOC from I think 2018, which was used quite widely in the market for different types of smartphones as well. 06:24.000 --> 06:30.000 The device we had had also a lot of peripherals and many different cool things in it, and it was a pretty cool device to be honest. 06:30.000 --> 06:37.000 So, I kind of like the idea of making sure it can run something newer than Android 9. 06:37.000 --> 06:42.000 By the way, can you hear me well in the back of the room? Thank you. 06:42.000 --> 06:51.000 So, this is the hardware part, then we took a look into the software part, and well, it was Android 9, you already know that. 06:51.000 --> 07:00.000 So, we learned that it was using Linux kernel version 4.9, which was okay back then. 07:00.000 --> 07:09.000 But, and we also, we were told that we will get access to the BSP, and then we got access, partially at least, and it was basically a single zip file. 07:09.000 --> 07:16.000 No history, no repair manifests, no information about what was changed and why. 07:16.000 --> 07:26.000 No support from ODM, so the original device manufacturer said, nah, to whatever you want, but we are not helping you. 07:26.000 --> 07:35.000 But, for us, it was super important that we got an expert on the site, like on the site of our client, who was working with this device for many years, 07:35.000 --> 07:40.000 who knew a lot about the kernel, who knew a lot about the hardware, so that was a huge win for us. 07:40.000 --> 07:44.000 Otherwise, it would take much, much longer, I think. 07:44.000 --> 07:50.000 So, we said yes, we said yes, we would like to try it, and we started the research. 07:50.000 --> 07:57.000 We started to, you know, by understanding what can we do, and is there anything on the market on the internet available? 07:57.000 --> 08:08.000 And obviously, there is a lot of BSPs for different smartphones, for different tablets, many open source projects. 08:08.000 --> 08:21.000 And we started looking through all of them, like everything we knew about for pin compatible or compatible socks and for similar kernel versions. 08:21.000 --> 08:35.000 Basically, we were scanning all Wikipedia pages, we were reading all articles, we were crawling through the forums and looking for unofficial and official builds for this type of the CPU. 08:35.000 --> 08:41.000 And at the end, we have compiled a list, we have compiled a pretty long list of everything we found. 08:41.000 --> 08:49.000 So, it was the list contained, like, a name of the device, which is, you know, the dedicated name of the project. 08:49.000 --> 09:00.000 The SOC, it was using the kernel Android versions, and whether it was actually maintained or not, if it was outdated already. 09:00.000 --> 09:10.000 And then, while we were evaluating those results and scrolling through this document, we noticed one of them, which was a perfect match for us. 09:11.000 --> 09:22.000 It basically had exactly the same kernel version, it had Android 14, which when we started the project was the most recent one, I would say. 09:22.000 --> 09:27.000 I had similar peripherals, small S, not really exact, but pretty good. 09:27.000 --> 09:34.000 Match, and it was officially maintained by a lineage community, and it was actively, actively underdevelopment. 09:34.000 --> 09:44.000 So, if we were successful on this project, it would mean that, rightfully, in future, we could upgrade even further and go beyond Android 14. 09:45.000 --> 09:54.000 So, we downloaded the BSP from lineage, we made sure that we can compile it and everything works, and then we thought, like, how do we approach this? 09:54.000 --> 10:02.000 So, there are many, many ways. I will present some of them. Obviously, there are more possibilities, and some of them will work in different projects. 10:02.000 --> 10:14.000 I will share some of our experiences, why would it some things? But, if you will be in this position, I'll be in future, I'll be happy to hear your opinions, I'll be happy to hear how you approach such things. 10:14.000 --> 10:20.000 If you already did that in the past, just to learn how other people approached this. 10:20.000 --> 10:38.000 So, for us, the very first step was, let's make sure that we can build a kernel. So, we had the kernel from the original BSP, and we knew that it was compatible with original BSP, but it was not compatible with lineage. 10:38.000 --> 10:47.000 The problem was that in lineage, they had different defines, they had different make files, they had different tool chains, and there were two ways for us to go. 10:47.000 --> 11:04.000 Either we can replace the lineage kernel with a kernel of the original BSP, or we can keep the kernel which was used in lineage, but then patch all the device to information, all the configuration parameters, everything into that kernel to make sure it works. 11:05.000 --> 11:23.000 We tried to do the first one, we failed terribly to be honest, so it was really, really difficult, because inside the lineage kernel, we found many dozens, I would say, of commits, regarding tool chains, regarding specific things, which lineage was using. 11:23.000 --> 11:29.000 So, backporting all of them into the original kernel was kind of stupid, so we did it the other way around. 11:29.000 --> 11:41.000 We migrated the device tree, we migrated the kernel config, we migrated some of the configuration, not all of it, because our strategy was, let's keep the kernel as small as possible. 11:41.000 --> 11:49.000 Let's remove all the stuff we don't need for the initial boot, and make sure that we can get the system to visualize somehow. 11:49.000 --> 12:01.000 So, we disabled audio, we disabled touch, we disabled networking, we disabled everything we could disable, hoping that it will give us a smooth boot for the first time. 12:02.000 --> 12:23.000 We needed to add some Android specific stuff, like you probably know some of you probably know there is this firmware, entry in the device tree where you can specify the amounts, the early amounts for the participants, so we had to add that, and once it was compiling, we thought of moving to the next step. 12:23.000 --> 12:39.000 And here in this step, I would say that without the expert, without the guy I was mentioning in the beginning, working for the company who hired us, without knowing all the details of the kernel, I think we would need months more work. 12:39.000 --> 12:48.000 So it was, we did that, and I say, I'd say maybe one month of work, I think without him, we would need at least five. 12:48.000 --> 12:57.000 If you don't know the kernel, if you don't know the hardware, if you don't know anything about the device you're working with, it's really, really hard to go like this. 12:57.000 --> 13:08.000 But we managed that part, and we thought, okay, so the kernel is compatible, we don't know yet if it boots, but it's compatible, so let's go to the next step, and the next goal is to actually make Android boot into the system UI. 13:10.000 --> 13:23.000 And here we did the similar approach, so the first step was, we went into the device 3, like the device config of Android, and we tried to make it as similar as in the original BSP, right? 13:23.000 --> 13:51.000 We made the partition layout compatible, we removed all the features we didn't need, we changed the configuration for the, for the halls, similar, and added many, many different tweaks to basically reduce the size of things which will be, which will be compiled and used by the boot during the boot, in order to avoid many of those things we are not interested in the beginning. 13:51.000 --> 14:13.000 So our main goal was to make it boot, and this was standard debugging cycle, right? So compile, flash, boot, check the logs, figure out what is, what is crashing, so if there is a freeze or boot loop or something, figure out what is the part which is not working, and then figure out why. 14:13.000 --> 14:35.000 And usually the reason was that the configuration inside the lunch was different from the configuration inside the original BSP, so our goal, I would say, or our move was to go back to see if we can change something, we can reuse something from the original BSP, understand the difference, and this is how we proceed. 14:35.000 --> 14:59.000 In the end, so for the booting device, for the very first time from the kernel until the UI, we had to swap two things, the first was the graphical stack, so there was some property drivers, some property blocks, we needed to reuse from the original BSP, and there was a USB gadget configuration thing, which was crashing the whole system. 14:59.000 --> 15:05.000 I don't know why, but then again, we swapped back with what we had from the original BSP and the devices booted up. 15:05.000 --> 15:13.000 I know it was pretty cool, experience I would say, it was the first time we did that, this type of project, so that was pretty cool. 15:13.000 --> 15:25.000 But we knew it's also more, let's just the beginning, right? Because now we are inside Android, so Android is booting we see the UI, but as I said, we disabled all the drivers, so nothing was working properly. 15:25.000 --> 15:42.000 Here we started to be adding things back from the beginning, so we started with touch, because if you have a smartphone or a device, it's easier to work with if you can just scroll around or touch it. 15:42.000 --> 16:06.000 The USB ADB was not working, as I said, we disabled the whole in the beginning, so we were going step by step, and every single time it's okay, check the logs, try to find an arrow in this specific configuration, try to understand what Linux has is using in this specific thing, figure out what was the configuration original BSP. 16:06.000 --> 16:18.000 Remember that we disabled some kernel drivers, we did not port some device three parts, so we have migrated them as well, step by step for every single feature step by step. 16:18.000 --> 16:28.000 And when the device was, I would say in a good condition, so most of the things we are working, we started finalizing and cleaning up stuff. 16:28.000 --> 16:41.000 Enabling as Linux again, try to figure out how to make Android-35 boot work, remove all the things we didn't need, all the clients didn't need, it's not us. 16:41.000 --> 16:57.000 Figure out how to sign it with a client keys to be compatible to the old device, to make sure that we can roll out this whole thing as an over-day update and future, integrate all over-day updates, logic and clients we have. 16:57.000 --> 17:06.000 And that's basically the main story until the device had Android 9 and the device had Android 14. 17:06.000 --> 17:14.000 I don't have too many things left, I would rather prefer a little bit of a discussion at the end, I have one more slide. 17:14.000 --> 17:19.000 I would consider this project a success, but it's far from being complete now. 17:19.000 --> 17:33.000 As I said, there's a whole bunch of peripherals we're still working on, so the device had multiple cameras, the device had a fingerprint sensor, and we are migrating those whole step by step. 17:33.000 --> 17:37.000 We will continue working with Linux, it was a fantastic support. 17:37.000 --> 17:47.000 I think without Linux it would not be possible for us to do this at least in this time, so maybe it would be possible if we would spend the next couple of years on that. 17:47.000 --> 17:51.000 But this was not going to happen. 17:51.000 --> 18:01.000 And I can say that there is a huge dependency on knowledge and availability of source code for projects like this for keeping devices up to date and maintainable. 18:01.000 --> 18:10.000 I think the availability of source code is a huge criterion for device longevity. 18:10.000 --> 18:23.000 So if you want to make sure the devices are actually usable, many years after they were manufactured, we have to open source the most important part of the device. 18:23.000 --> 18:29.000 It was difficult without the support of the original device manufacturer, it was really difficult. 18:29.000 --> 18:32.000 As I said, we had a lot of help from the expert. 18:32.000 --> 18:40.000 Without it, I'm not sure if it would be successful in that time, maybe it would take much longer, probably it would take much longer. 18:40.000 --> 18:48.000 I think if the device is persistent, if you have a goal, if you know what you want, you can still do it. 18:48.000 --> 18:56.000 I mean there are examples of jail breaking smartphones and keeping smartphones up to date, which we are not meant to work like that. 18:56.000 --> 19:05.000 For developers, we didn't do a lot of reusing existing USB but had to implement everything from scratch. 19:05.000 --> 19:14.000 So this is my call to action to everyone here and online and in the community, please try to contribute. 19:14.000 --> 19:21.000 We are contributing, we are a commercial company but we are contributing where we are possible. 19:21.000 --> 19:30.000 We are not violating any in the A's and we are not violating any other agreements with our clients. 19:30.000 --> 19:34.000 We will be doing so in the future as well, we will continue to do so. 19:34.000 --> 19:44.000 And I would love to see more stories like this to learn how people are actually upgrading devices, how we can keep the community rolling. 19:44.000 --> 20:03.000 And I wonder how many million devices could be actually saved from going to waste and being replaced if there would be an open source implementation, if there would be an repository where you can just download your code to some adjustments and push it. 20:04.000 --> 20:10.000 So that's the story for me. Thank you very much for your attention and I'm helping for questions. 20:15.000 --> 20:19.000 Two questions one is just for this project. 20:19.000 --> 20:22.000 We did it. 20:22.000 --> 20:26.000 So how many people worked on that as a whole time? 20:27.000 --> 20:32.000 We are a small company. We only have, thank you. So I'm being asked to repeat the questions. 20:32.000 --> 20:38.000 So the question was how many people are working for us and on this project. 20:38.000 --> 20:40.000 So we are a small company. 20:40.000 --> 20:55.000 We have, I would say, ten people in total but in many different areas, backhand fronthand apps on the A's P level, there are just three people who can work full time including myself. 20:55.000 --> 21:03.000 But since I'm also involved in the organizational part, I'm usually not working a lot on the low level stuff. 21:03.000 --> 21:11.000 So I would say it's, it's the expert I was talking about. He was doing a lot of work also hands on development work. 21:11.000 --> 21:20.000 It's one full time employee on our side and maybe one or two people jumping on and off all the time. 21:21.000 --> 21:27.000 And the second question is, do you expect to change? 21:27.000 --> 21:38.000 So with the upcoming policies that the vendor can obtain more bets, your problem, knife, or that even has to do this stuff. 21:38.000 --> 21:45.000 The question is, if I expect any change in this policies or vendors where they say, 21:45.000 --> 21:51.000 we will not provide any updates and the question was if I can expect any change in that. 21:51.000 --> 21:53.000 I hope so, I hope so very much. 21:53.000 --> 22:01.000 I still see a lot of product builders who say security is super important for us but we have no budget for it. 22:01.000 --> 22:04.000 And the policy is that doesn't matter. 22:04.000 --> 22:07.000 Yeah, but they say we will take care of that when it's there. 22:07.000 --> 22:18.000 But if I mean what people don't understand today a lot is that if you don't embed some kind of capability of installing updates later on, 22:18.000 --> 22:22.000 there will be no way of updating it even if it will be necessary. 22:22.000 --> 22:25.000 So we try to educate our clients as well. 22:25.000 --> 22:31.000 So using our clients comes to us when it's either too late or something is going really wrong. 22:31.000 --> 22:33.000 Not all of them. 22:33.000 --> 22:37.000 The companies who think five years in advance, that's really nice to work with them. 22:37.000 --> 22:45.000 But by far the most come to us and say, oh, we actually didn't think about this part about security over the other things. 22:45.000 --> 22:51.000 Can we fix it now and then they ask us how can you retrofit such things and that's not possible. 22:51.000 --> 22:56.000 So I hope Europe is moving into the right direction or I think Europe is moving into the right direction. 22:56.000 --> 23:01.000 I hope Asian manufacturers will follow, I hope they will be forced by that. 23:01.000 --> 23:05.000 But it's it's a difficult position to negotiate, I think. 23:05.000 --> 23:15.000 So because if you have product builder, a small product builder in Europe and you go to Asia, China, Taiwan, India and you say, I want updates. 23:15.000 --> 23:23.000 How the manufacturer will say, well, yeah, maybe maybe it will cost you a million, maybe two, maybe five, we don't know today. 23:23.000 --> 23:30.000 So this is the position where they are and I think building up tools, building up tools like for automatic compilation, 23:30.000 --> 23:34.000 automatic rollout will support that a lot. 23:34.000 --> 23:38.000 And forcing standards as well. 23:38.000 --> 23:40.000 Yes, please. 23:40.000 --> 23:56.000 The question was, if porting the drivers can be supported by enforcing GPL, 23:56.000 --> 24:08.000 this is something I would like to believe in, but the reality is you ask the manufacturers and they say, 24:08.000 --> 24:12.000 yeah, we know about this but we are not doing this. 24:12.000 --> 24:16.000 So and if you go to China and try to sue them, you will not be successful. 24:16.000 --> 24:20.000 So I think the importance here is to do it in advance. 24:20.000 --> 24:24.000 So when you sign a contract for a new product, 24:24.000 --> 24:32.000 we have to think about such things as advance and get the access to the BSP early on, so they won't tell it to you later on. 24:32.000 --> 24:36.000 Yes, please. 24:36.000 --> 24:38.000 Excuse me? 24:40.000 --> 24:50.000 Not the vendor, but our client, the company who hired us, they had a copy of the BSP, they had a copy of the device three. 24:50.000 --> 24:54.000 It was almost complete, I would say. 24:54.000 --> 24:58.000 So there was a lot of stuff in it. 24:58.000 --> 25:02.000 They opened sourcing part of it, yes. 25:02.000 --> 25:20.000 I tried to summarize the question. 25:20.000 --> 25:24.000 If our time is up, I will be available here if you have any further questions. 25:24.000 --> 25:26.000 So we can continue to discussion later on. 25:26.000 --> 25:30.000 The last question was, if what we will do in future, 25:30.000 --> 25:36.000 so if the road is closed for us now, what we did already and it will not be open source fully, 25:36.000 --> 25:42.000 I can say that the goal was to reuse the lineage BSP, so there are used to lineage device things, 25:42.000 --> 25:48.000 and we will try to contribute there as much as possible to be compatible with our device. 25:48.000 --> 25:52.000 This is the goal, so we can rebuild it in future from there. 25:52.000 --> 26:00.000 For the other projects, again, I think it's a question of getting the BSP as early as possible. 26:00.000 --> 26:04.000 The problem is that the device three source is not in the air. 26:04.000 --> 26:06.000 Yeah. 26:06.000 --> 26:09.000 The device three source is not GPL, that's one of the biggest problems, yes. 26:09.000 --> 26:10.000 Thank you very much. 26:10.000 --> 26:12.000 Can I take another question? 26:12.000 --> 26:14.000 Yes. 26:14.000 --> 26:16.000 Can I take another question? 26:17.000 --> 26:25.000 No, it's not for phone calls, I'm not sure I can talk about this, but it's not for no. 26:25.000 --> 26:33.000 So there is no real, so it's a B2B device, so it's a device used by a person for specific task, 26:33.000 --> 26:38.000 but not for consumer grade device. 26:38.000 --> 26:40.000 Thank you very much. 26:46.000 --> 26:48.000 Thank you very much.