WEBVTT

00:00.000 --> 00:13.520
Perfect. Thanks, everyone, for coming. My name is Sherbanko, I'm from UNESCO. This is an unofficial

00:13.520 --> 00:19.680
guide to trunk stable. For those of you that have not heard of trunk stable before, this

00:19.680 --> 00:25.800
is a new Android development model that you're going to learn a bit more about in the next

00:25.800 --> 00:33.760
few slides. So, just before we start, a little bit about myself. I've worked on Android

00:33.760 --> 00:39.700
for close 13 years now. Almost half of that time, I've worked on Android from a

00:39.700 --> 00:51.160
term. I've contributed significantly to porting Android to 264rv8, so 64 bit. I've

00:51.160 --> 00:57.800
also pmd technology called R memory safety. This is basically a hardware technology that

00:57.800 --> 01:04.800
helps with catching buffer overflows and use after freeze. And then with Google, I was the

01:04.800 --> 01:13.240
pmd for a bunch of technologies that you've heard about in some of the previous slides. I

01:13.240 --> 01:22.120
led Android virtualization. I led Android runtime and Android native teams. I've also led Android

01:22.120 --> 01:27.440
regulation and in the past few years I've been working a lot on regulation and longevity.

01:27.440 --> 01:33.680
This is basically Android's initiative to keep devices around for longer.

01:34.480 --> 01:41.280
Last year I left to start source of dev together with David who's here in the audience and

01:41.280 --> 01:49.760
also a game who's a vector where we're basically building an operating system for the operating

01:49.760 --> 01:54.880
system kind of. We're trying to make it easier for some of you folks to be able to develop

01:54.880 --> 02:01.840
but also to maintain those code bases for a long time. We're putting all the learnings

02:01.840 --> 02:08.800
with a gather across the years to build this and we're hiding. I'll tell you more about that

02:08.800 --> 02:16.400
towards the end. So just the last thing, please take this as an unofficial guide. I've been

02:16.400 --> 02:23.360
deeply involved with a lot of the decisions we'll discuss here but I'm no longer a Googler. So

02:23.360 --> 02:30.320
please take this as the unofficial guide and if we discuss it before many things are changing

02:30.320 --> 02:35.600
quite rapidly so I might want to check the code base even if I claim something's here in

02:35.600 --> 02:46.080
in this deck. So just a quick overview of what's trunk stable. So trunk stable is basically

02:46.080 --> 02:52.240
a new upstream Android development process where all the releases are cut from a single main

02:52.240 --> 03:00.160
development branch and that branch must remain stable at all times. So it must build and pass

03:00.720 --> 03:08.400
a set of tests. This basically requires that all new features are developed behind flags

03:09.280 --> 03:16.320
so that on a case by case basis those flags could be enabled or disabled to maintain code base

03:16.400 --> 03:24.480
stability. You'll see in a second why this model was very much needed but basically Google

03:24.480 --> 03:36.240
advertises this as streamlining development and improving quality. So probably the best way to

03:36.240 --> 03:44.160
describe why the new model works and why it's so useful is to take a look at the old development model

03:44.320 --> 03:53.200
and to try to understand what wasn't working in the old development model. So if you look at it

03:53.200 --> 03:58.800
from a timeline perspective the old development model kind of looked like this. You had an Android

03:58.800 --> 04:06.720
version being released then soon after that Android version would be released or actually internally

04:06.720 --> 04:13.920
even before the Android version would get publicly released we would start working on the next

04:14.000 --> 04:22.080
Android version then you would have portion of code freeze and this is also known internally

04:22.080 --> 04:28.960
at Google as Dr. Now you would need to get approvals many approvals to get patches margin to the

04:28.960 --> 04:36.800
code base and then the next version would basically get released. So behind the scenes

04:37.760 --> 04:44.400
this model kind of looked like this. Folks like me, pms would go off and write pardes and

04:44.960 --> 04:50.080
figure out what we're building and why we're building and what's the minimum viable version of that

04:51.360 --> 04:56.400
and then together with the engineering teams we would start working on those features

04:57.920 --> 05:05.840
and then almost a week or two weeks before code freeze held would break loose literally

05:05.920 --> 05:14.160
like everything would be merged into the main course. We would all be dog footing our devices

05:14.160 --> 05:20.480
and those devices would stop working and that period over their kind of correlates with the time

05:20.480 --> 05:26.640
my wife would get very mad at me because my phone would almost never work. But the important

05:26.640 --> 05:34.320
thing here to highlight is that because the timeline was somewhat fixed that meant that we had

05:34.320 --> 05:39.040
to compromise on a few things and some of those features might not have been

05:40.000 --> 05:46.960
ready to be released or would have required a longer period of time. So from a quality perspective

05:46.960 --> 05:52.800
if we look at the same thing it kind of looked like this. We would release an Android version

05:54.080 --> 05:59.760
and in the background we would still be eagerly working on stabilizing that code base

06:00.720 --> 06:08.400
we would heat some sort of stabilization point after releasing the code base and then again

06:08.400 --> 06:14.320
we would do the same thing where the quality would drop when we start merging all the

06:14.320 --> 06:21.200
features. Just to put this in perspective I think the Android engineering organization today is

06:21.200 --> 06:28.720
about 4,000 engineers at Google we would work very closely with some of the ecosystem partners

06:29.760 --> 06:35.440
so think about similar numbers from the lead associate vendors and then think about the

06:35.440 --> 06:42.960
charting that would go on in the code base I think Android AOSP it's about 120 million lines of code

06:42.960 --> 06:49.760
today. Android OSP as you folks know is not actually the full Android as you can imagine that

06:49.760 --> 06:55.040
the full Android is even bigger than that and I'm sure that some of your internal code bases are

06:55.120 --> 06:59.200
very big as well so imagine the amount of children that would go on there and how difficult

06:59.200 --> 07:08.880
they would be to get this on. So just sort of like recap if you think about this model

07:08.880 --> 07:17.440
at this model choose is a fixed timeline kind of fixed set of features and then kind of

07:17.920 --> 07:23.200
doesn't prioritize quality and this model has worked actually for many years early versions of

07:23.200 --> 07:30.720
Android didn't have these problems it probably became really significant in Android 12 if you

07:30.720 --> 07:35.920
search Android 12 online there many articles including the words should that say well this was

07:35.920 --> 07:47.040
in the greatest release and was primarily because of that reason. So by comparison this is the new

07:47.120 --> 07:54.480
Android development model from a timeline perspective. So instead of having one release you have

07:54.480 --> 08:01.760
four releases and throughout the year every quarter you have a new release this is unlike the

08:01.760 --> 08:09.760
oak all QPR model you're used to the all QPR model would primarily be bug fixes this is actually

08:09.760 --> 08:19.200
features all the time features get released obviously bug bug fixes and security fixes and then

08:19.200 --> 08:26.640
you have these two distinct releases so you have Q2 which is a major SDK release that comes a lot

08:26.640 --> 08:32.800
earlier then all the Android would come would get released so you get probably Android around Q3

08:32.880 --> 08:40.960
now you get it around Q2 because that basically expedites all the development towards products

08:40.960 --> 08:47.680
in Q4 which is the peak of product sales so that gives folks one more quarter to work on that

08:48.720 --> 08:57.920
Q2 has API changes so new features developers could use Q2 also has behavior changes so

08:58.240 --> 09:07.520
changes in the API surface that developers would use and also not mentioned here but Q1 or Q2

09:07.520 --> 09:15.040
have sort of like the hard changes so if you're a SOC vendor you probably just have to focus on one of

09:15.040 --> 09:20.080
those that will be your major release and obviously you need to iterate with the others but those

09:20.080 --> 09:27.600
will be one the hard changes happen and then Q4 is effectively think of it as developer preview

09:28.400 --> 09:37.440
it's APIs and features no behavior changes but developers would have sort of like a minor version

09:38.080 --> 09:43.680
that they could target in their application so that they provide earlier feedback towards the next year

09:44.000 --> 09:53.040
towards Q2 so for this model you're obviously going to have features that take a lot longer

09:53.040 --> 09:58.240
than a quarter to build most of the Android features take a lot longer than a quarter to build

09:59.680 --> 10:08.800
and you would have features behind flags only the features that are ready to be released would be

10:09.760 --> 10:16.640
turned on by the flagging system and everything else would remain turned off and you basically

10:16.640 --> 10:23.360
have every release would probably have a slightly different set of features enabled a slightly different

10:23.360 --> 10:28.800
set of flags where probably every feature would inherit from the previous feature and other

10:28.800 --> 10:36.480
few more on top of it so from a quality perspective assuming you've got like the right

10:36.480 --> 10:43.120
system is in place this kind of looks like this and so you'd be able to quickly revert features

10:43.120 --> 10:49.280
that don't work turn off flags and things like that actually if you follow closely the Android code

10:49.280 --> 10:56.960
base some of the systems are already present in the OSP codebase you would see flags turn off automatically

10:58.000 --> 11:05.200
by an agent not an AI agent but like an agent that actually does something that's useful

11:06.480 --> 11:14.000
and that agent would basically monitor that the codebase is stable or if it's not stable

11:14.000 --> 11:19.920
it would try to bisect and identify the flags that have been turned on and then turn them automatically off

11:22.480 --> 11:31.600
so from if you look at this model this model obviously chooses features in quality prioritized

11:31.600 --> 11:39.680
as a quality which is very important and then kind of has a more granular release mechanism

11:39.680 --> 11:46.080
which gives developers more opportunities to release the features that make the Android codebase

11:47.440 --> 11:54.960
great so if you put them side by side the all development model and the new development model

11:55.600 --> 12:00.880
kind of looks like this the all development model was branch based even though

12:01.920 --> 12:06.240
David corrected me on this actually internally you don't have that many branches but you have a lot of

12:06.240 --> 12:12.400
code that's flying around so the all development model you could still say that's branch based

12:13.280 --> 12:18.560
teams work separately and then towards that called for this everything would be merged back in

12:19.360 --> 12:25.440
you have a yearly release cycle those QPRs actually are not that meaningful or didn't

12:25.440 --> 12:30.560
used to be that meaningful in the past and then it's relatively hard to maintain codebase

12:30.960 --> 12:39.440
quality and you get only one SDK release per year by comparison the new development model is flag based

12:40.800 --> 12:48.400
there's a quarter release cadence and field it's easier to maintain codebase quality and as I said

12:48.400 --> 12:54.800
you could already see the benefits of this in RSP and then you get a few more as the key releases

12:54.800 --> 13:01.280
per year which is also important because what the old model was missing was feedback from developers

13:02.320 --> 13:09.840
on API usage in the applications and so on and this model enables that so if you look at it from a

13:09.840 --> 13:17.360
downstream OS fork and you can think here of a downstream OS fork sort of like an OEM OS fork

13:17.440 --> 13:23.760
but you could also think about automotive for example or other verticals those are OS forks as well

13:25.280 --> 13:31.360
the old model had a bunch of problems around sort of like rebasing and testing cycles

13:33.120 --> 13:39.360
some of the bigger OEM so I had a pleasure to work with across the years for them this yearly

13:39.360 --> 13:45.360
rebase would be a significant productivity hit like basically the whole team would would have to

13:46.240 --> 13:52.000
rebase a massive codebase it would be difficult to identify issues I really stopped stream

13:53.040 --> 13:59.280
actually we've heard this a lot one going the details but you can imagine it's finding the needle in the

13:59.280 --> 14:08.240
haystack and by comparison the new development model in theory makes it easier to rebase and test continuously

14:08.880 --> 14:16.640
easier to identify upstream issues easier to bisect and that would hopefully reduce the time

14:16.640 --> 14:21.120
I dropped it a codebase I think the last thing I'll say on this slide is that

14:22.560 --> 14:30.080
this is primarily for android properties or android phones but you will see some of these

14:30.080 --> 14:34.640
trickling down to the other verticals so if you're doing automotive where or anything else

14:34.640 --> 14:43.040
android related sooner or later this would be coming your way so you might be asking yourself

14:43.040 --> 14:50.880
weren't went to use facts I'll say by saying most big modern codebase is used flag based systems

14:52.320 --> 14:57.680
there's almost nothing out there of the size of android and actually even order magnitude smaller

14:57.680 --> 15:04.320
that doesn't use flag based system google play gms core facebook tick talk all of those codebase

15:05.280 --> 15:14.480
flag based releases so you should use a flag wherever your change might lead to codebase

15:14.480 --> 15:21.760
instability and that most likely is for everything so all major features need to be behind

15:21.760 --> 15:29.680
this complex bags API changes behavior changes need to be behind flags the place you would not use

15:29.760 --> 15:36.640
flags would probably be comments small changes or where the flag already exists and I think

15:36.640 --> 15:43.200
this kind of requires a lot of change in the engineering mindset it's probably better to

15:43.200 --> 15:47.840
start from a place where you add flags everywhere rather than not flag flags everywhere

15:50.720 --> 15:56.480
so there are two types of flags that you could use in android there's runtime control flags

15:56.800 --> 16:03.120
something that's called aconfig this are basically used in cc++ rust and java

16:04.400 --> 16:11.920
you basically declare the flag in a dot aconfig file and then you could check that the

16:11.920 --> 16:18.720
runtime if the flag is set or not and then there are two types of flags read only flags

16:19.680 --> 16:22.480
these flags are basically the flags they would have for the release

16:24.400 --> 16:29.840
once you set the values as read only compilers would actually understand that

16:30.640 --> 16:36.640
some portion of code is is that code and would eliminate that and optimize the code for you

16:37.600 --> 16:41.200
but you also have read right flags and this is what we use during development

16:42.160 --> 16:48.720
and actually this is how you do a lot of at scale AB testing so you'd have one dog food

16:48.720 --> 16:54.080
population with the flag set one one dog food population with the flags that do often you would see

16:54.080 --> 16:59.840
what are the benefits of that change in feature and then at scale with the with a right

16:59.840 --> 17:06.880
of system behind it this helps you out you also have built-in control flags and these are used

17:06.880 --> 17:13.680
for low level firmware changes the Android virtualization framework uses this for example

17:13.680 --> 17:18.960
because it has changes in pre-built code and other changes that require build system changes

17:19.840 --> 17:24.640
I want to leave more about flags but you can take a look here how to use a flags that's a

17:24.640 --> 17:31.440
pretty comprehensive guide for that and then for build flags I didn't find the right documentation

17:31.440 --> 17:37.920
but if you look inside a build folder and just to get a lot minus p and take a look for the

17:37.920 --> 17:46.640
change that adds a build flag it's pretty self-explanatory so there are a few release flavors

17:48.160 --> 17:55.120
and if you look here I basically start a cuttlefish target and then I vote this command

17:56.000 --> 18:00.720
list releases which would basically give me the releases that are available for

18:01.680 --> 18:07.760
that target a USB current that's basically mapped to the latest release which in this case is

18:07.760 --> 18:16.160
AP4A AP3A and AP2A are releases that came before it then you have trunk staging which is basically

18:16.160 --> 18:21.360
a development configuration you could take a look at trunk staging as like an indication of what

18:21.360 --> 18:28.960
might come up in the next release and then something that's not public is this next release that

18:28.960 --> 18:35.920
Google has internally which describes sort of like the dog food population and also what I

18:35.920 --> 18:44.480
wanted to show you is if you go in in the build output I could take a look at what are the flags

18:44.480 --> 18:50.320
enabled for your particular target and what's really neat about that is that you also have a

18:50.320 --> 18:57.520
description for all of those flags so for folks that were downstream of Android this is actually

18:57.600 --> 19:04.240
quite helpful some of those descriptions are not set up until the flag actually gets publicly released

19:05.840 --> 19:10.320
but you could you could have an idea of like what the next Android might look like

19:13.200 --> 19:20.880
similarly if you look at the AIC on feed flags if you go and pick a new Android phone connected to

19:20.880 --> 19:27.200
ADB and then run this command that command would tell you what flags you have on that device

19:28.640 --> 19:32.800
if you actually run this command you'll see that there are many many pages of flags

19:33.360 --> 19:41.200
there's flags used for everything right now this is just a short shortening of that

19:42.400 --> 19:48.880
and as you can see those flags can have different default values they could be read only they could

19:48.880 --> 19:53.360
be read right and then you also have a name space the name space basically says where you could read

19:53.360 --> 20:02.240
that flag from and so on I'm not going into other details here but as I said some of these flags

20:02.240 --> 20:07.680
are serve could be server controlled so you can think of a system where you could actually control

20:07.680 --> 20:14.320
some of these flags server side so that you do those AB populations and stuff like that and

20:14.320 --> 20:22.640
that's actually what the big companies that do Android scale do here so from a life cycle perspective

20:22.640 --> 20:27.840
it kind of looks like this you have a feature flag you start developing before behind a feature flag

20:28.640 --> 20:35.120
you enable it locally for testing the new enable it dock for the team wise and then you enable it

20:35.120 --> 20:40.320
in the next release and then as I said automated systems could actually go ahead and

20:40.880 --> 20:47.520
disable that and that could also be downstream automated systems that monitor stability for

20:48.080 --> 20:55.120
whatever use case you might have so you might be thinking about this one well that's cool but

20:55.120 --> 21:02.160
when when I get that well there's actually a lot of regulation coming in that says you need to get

21:02.160 --> 21:07.920
security updates and always upgrades there's regulation coming in for all the verticals

21:08.880 --> 21:15.040
in 20th of June this year in Europe all smartphone and tablets placed on the market

21:15.040 --> 21:21.520
would have to support devices with five years of OS and security updates after the point of

21:21.520 --> 21:25.920
placement point of placement for some devices could actually mean seven years of updates

21:26.960 --> 21:31.760
and there's some companies that have already made some announcements in that direction

21:32.240 --> 21:37.440
but there's something that explicitly says that features or OS updates need to be released within

21:37.520 --> 21:42.240
six months of public release so I'm not a lawyer but I'll let you interpret that

21:42.240 --> 21:48.560
and you have a FAQ here that talks more about it and gives you a link to the law

21:50.640 --> 21:57.440
so that's it thanks I'll also have a quick shameless flag will be posting this in air

21:57.440 --> 22:04.160
speed devs but we're hiding if you want to work on something that's very exciting

22:04.800 --> 22:09.360
sort of like an operating system for an operating system we use rust we use all sorts of

22:09.360 --> 22:16.960
kind of technologies we use a brand-new lake and a bunch of other cool things come chat to us

22:16.960 --> 22:21.120
or send the email sharebundant source to dev I will also be working with

22:22.160 --> 22:26.560
former Android team members so that should be super exciting thank you