1 00:00:00,000 --> 00:00:15,520 *wikipaka preroll music* 2 00:00:15,520 --> 00:00:22,720 Hello and welcome to my talk on Remixing Linux. My name is The one with the braid, 3 00:00:22,720 --> 00:00:30,480 and today I'm going to introduce your own creation of a Linux distribution. We will 4 00:00:30,480 --> 00:00:37,280 talk on remixing Linux, on the architecture you require to run an own 5 00:00:37,280 --> 00:00:50,830 Linux distribution, and some of the usual use cases. OK, let's start. Well, what are 6 00:00:50,830 --> 00:00:57,356 use cases, what are purposes of your own Linux distribution? Of course, you could 7 00:00:57,356 --> 00:01:02,544 customize Linux distribution for your enterprise environment. For example, if 8 00:01:02,544 --> 00:01:08,152 you require special network settings, your own services to be installed, it's 9 00:01:08,152 --> 00:01:15,298 sometimes easier to create an own distribution than using a ready to use 10 00:01:15,298 --> 00:01:21,280 distribution and customizing it. Another option is that you discovered something 11 00:01:21,280 --> 00:01:27,885 fully new and decide to create your own Linux distribution. That's what we call 12 00:01:27,885 --> 00:01:34,015 yet another Linux distribution. Please, please do not do this. We already 13 00:01:34,015 --> 00:01:42,293 have enough different flavors of Linux. OK, another valid reason is missing 14 00:01:42,293 --> 00:01:46,640 software. If you require a special software for use case and you want to 15 00:01:46,640 --> 00:01:52,450 bundle it into your operating system. That's a usual use case. Moreover, 16 00:01:52,450 --> 00:02:02,251 embedding for microcontrollers is a common use case for Linux remixing, as well as 17 00:02:02,251 --> 00:02:11,204 OEM hardware support for computer vendors to bundle Linux. Some known vendors do 18 00:02:11,204 --> 00:02:18,509 this. Yeah, and of course, another desktop or in general different preinstalled 19 00:02:18,509 --> 00:02:24,760 applications are a good reason to create your own distribution. OK, what are we 20 00:02:24,760 --> 00:02:30,624 going to have a look on today? We will look on live systems, because that's the 21 00:02:30,624 --> 00:02:36,394 most common use case. Afterwards we will go through step by step instructions for 22 00:02:36,394 --> 00:02:44,131 and next, we take an example of Ubuntu. We talk on making your system installable and 23 00:02:44,131 --> 00:02:49,855 we will talk about a simple tool simplifying some of these steps. We will 24 00:02:49,855 --> 00:02:55,370 have a look on Linux from scratch, meaning compiling everything yourself without 25 00:02:55,370 --> 00:03:01,320 using any templates or anything. And we'll look at the architecture you require to 26 00:03:01,320 --> 00:03:07,354 run your own Linux distribution, a.k.a. package mirrors. OK, let's start with a 27 00:03:07,354 --> 00:03:15,080 live system. What are properties of a live system? Of course it needs to fulfill some 28 00:03:15,080 --> 00:03:22,760 requirements. If you use a tiny pen drive, I don't know, eight gigabyte or 16 29 00:03:22,760 --> 00:03:30,240 gigabyte, it's obvious you need to compress your whole system. Moreover, it 30 00:03:30,240 --> 00:03:37,022 must be writable at runtime. Just try to mount your root filesystem in Read-Only and 31 00:03:37,022 --> 00:03:43,120 run a web browser. You run into big trouble. Same for network connections. If 32 00:03:43,120 --> 00:03:50,248 you connect to wifi without writable root system, you will run into big trouble. And 33 00:03:50,248 --> 00:03:55,967 another requirement, it's not really a requirement, but it's recommended is an 34 00:03:55,967 --> 00:04:00,844 accessible live media. So if we have our pen drive containing our live Linux 35 00:04:00,844 --> 00:04:05,683 distribution, we sometimes need to access the pen drive itself in our live 36 00:04:05,683 --> 00:04:12,113 environment. For example, if we'll have packages of our distribution inside or on 37 00:04:12,113 --> 00:04:19,668 our pen drive, we may want to install them later. Then it's quite good to have your 38 00:04:19,668 --> 00:04:26,880 live media accessible. Well, of course, we could script everything on our own, but 39 00:04:26,880 --> 00:04:33,677 someone already did all these steps and has put them into a tiny software called 40 00:04:33,677 --> 00:04:40,600 Casper. Casper does exactly what we want, creates a union file system, meaning an 41 00:04:40,600 --> 00:04:48,335 overlay on the existing file system and creates a kind of fake writable system in 42 00:04:48,335 --> 00:04:53,941 memory, file system in memory. So it allows modification of files, creation of 43 00:04:53,941 --> 00:04:59,422 new files while they are actually being stored in the memory instead of on our 44 00:04:59,422 --> 00:05:05,720 physical disk. It mounts the live media and it shows up a pretty display manager 45 00:05:05,720 --> 00:05:12,035 you can see over here. That's the Ubuntu mate, it asks whether you want to try our 46 00:05:12,035 --> 00:05:19,160 system or whether we want to install our system. OK, let's talk about the 47 00:05:19,160 --> 00:05:24,935 compression, because the compression we require is tremendous. If we have a normal 48 00:05:24,935 --> 00:05:32,546 desktop distribution. Believe me, it consists of about 30 gigabyte of data. 49 00:05:32,546 --> 00:05:37,814 Our applications, as soon as you install some more applications like LibreOffice or 50 00:05:37,814 --> 00:05:46,773 a Web browser and some tools, you will reach the 30 gigabyte or 20 gigabytes, but 51 00:05:46,773 --> 00:05:53,280 if you ever downloaded a live distribution, you will realize it's not 30 52 00:05:53,280 --> 00:06:03,680 gigabyte. It's about two gigabyte or one point five. I noted in my slides. Yeah, it 53 00:06:03,680 --> 00:06:10,560 means we need to compress our file system on about 5 or 10 percent of the actual 54 00:06:10,560 --> 00:06:19,200 size. That's not difficult. There are many compression algorithms we may use, but we 55 00:06:19,200 --> 00:06:26,640 have another requirement. We need performant readability. As soon as we boot 56 00:06:26,640 --> 00:06:33,280 a system, our kernel won't stop accessing any files, reading files, writing files. 57 00:06:33,280 --> 00:06:38,890 So we need a performant readability because no one wants to wait five minutes 58 00:06:38,890 --> 00:06:45,840 to see the Web browser. And that's what the squash file system is made for. The 59 00:06:45,840 --> 00:06:52,080 squash file system is a compressed file system. It's kind of a disk image, but more 60 00:06:52,080 --> 00:06:59,440 advanced. It's used in diverse Linuxes. It's used in the operating system of 61 00:06:59,440 --> 00:07:05,840 Chromecast and in the Linux userland it's used by app images. These are these click- 62 00:07:05,840 --> 00:07:12,960 to-run Linux binaries you can download, and it's used by snap packages from 63 00:07:12,960 --> 00:07:19,840 canonical's proprietary snap package manager, which is unfortunately very 64 00:07:19,840 --> 00:07:27,200 widely spread. Yeah. Now we know something about compression. Let's talk about how 65 00:07:27,200 --> 00:07:33,600 the customization works. Well, first of all, we download a template. We create a 66 00:07:33,600 --> 00:07:40,974 base system from which we start our customization process. Well, we start 67 00:07:40,974 --> 00:07:48,925 the customization itself. We prepare a CD environment. So when you put a bootloader 68 00:07:48,925 --> 00:07:56,104 on a CD, we pack our root file system we just created, put it on the CD, put a 69 00:07:56,104 --> 00:08:05,491 kernel there, make it, harden it a bit, and at the end we will pack it. OK, let's 70 00:08:05,491 --> 00:08:16,136 go through it, step by step. In this case, we will have a look at Ubuntu, we will go 71 00:08:16,136 --> 00:08:23,150 step by step through the customization process of Ubuntu. How do you start the 72 00:08:23,150 --> 00:08:31,393 customization process? Well, you need an existing ISO file from a live environment 73 00:08:31,393 --> 00:08:38,782 of Ubuntu, because it contains the bootloader, the directory structure and 74 00:08:38,782 --> 00:08:47,942 everything, all the assets. Of course, you could create them on your own, but, well, 75 00:08:47,942 --> 00:08:54,720 it's more work than you have to do. You can easily use the existing bootloader and 76 00:08:54,720 --> 00:09:00,522 the existing files and directory structure from an existing ISO file. Well, let's 77 00:09:00,522 --> 00:09:07,440 extract it and we're fine. We're done with it. Afterwards, we need to create a base 78 00:09:07,440 --> 00:09:14,547 system. A base system is, well, the base of Linux. We have the base utils, the 79 00:09:14,547 --> 00:09:24,589 base files, the directory structure, the kernel, a shell. The package manager of 80 00:09:24,589 --> 00:09:32,670 our operating system we have. Without any unnecessary tools, without, just the base 81 00:09:32,670 --> 00:09:40,988 of everything. You can either download the prepared base system of the distribution for 82 00:09:40,988 --> 00:09:48,560 download, but you could create one on your own using tools like the other Debian tool 83 00:09:48,560 --> 00:09:55,122 to create base system or the corresponding tools from other distributions. Yeah, as 84 00:09:55,122 --> 00:10:01,356 soon as you have your base system, you need to change your root inside this base 85 00:10:01,356 --> 00:10:08,696 system. And then you start the customization process by installing the 86 00:10:08,696 --> 00:10:15,412 necessary drivers. We already talked about Casper. Casper is this tool providing the 87 00:10:15,412 --> 00:10:23,542 live software and the live scripts, the overlay file system in Ubuntu based 88 00:10:23,542 --> 00:10:31,641 systems. And it creates an initial RAM- file-system and installes the required 89 00:10:31,641 --> 00:10:37,876 kernel modules to load and to boot a squashed file system. So let's install 90 00:10:37,876 --> 00:10:45,909 these three packages, Casper, Ubiquity- Casper and Lupin-Casper. And afterwards we 91 00:10:45,909 --> 00:10:52,160 can install whatever we want. We can install our loved vim, we can install our 92 00:10:52,160 --> 00:10:59,595 loved Alpaka software, and we can install an alternative shell, the packages we 93 00:10:59,595 --> 00:11:07,220 want. In the example code I provided here we look on changing the plymouth screen. 94 00:11:07,220 --> 00:11:13,017 That's the startup screen which shows up when you boot the distribution. I don't 95 00:11:13,017 --> 00:11:19,610 know, you could put an Alpaka in there or the logo of rC3. Yeah, that's what we do 96 00:11:19,610 --> 00:11:27,280 in customization or modification of the system. Afterwards, we continue packing 97 00:11:27,280 --> 00:11:33,654 everything we created. First of all, we need a Kernel to boot , because well 98 00:11:33,654 --> 00:11:39,790 before our base system can be extracted, we need to load a kernel by the bootloader 99 00:11:39,790 --> 00:11:46,687 and the kernel must contain the corresponding kernel modules to load, to 100 00:11:46,687 --> 00:11:52,888 boot a squash file system. That's what Caspar already prepared for us, we only 101 00:11:52,888 --> 00:12:00,496 need to create the initial run file system and if you read the code I provided, you 102 00:12:00,496 --> 00:12:08,037 will see I was missing an minus between update and init RAM file system. Let's 103 00:12:08,037 --> 00:12:16,558 create our initial RAM file system and copy them or copy it together with our 104 00:12:16,558 --> 00:12:25,250 kernel to our extracted live environment. And can delete the prior kernels and so 105 00:12:25,250 --> 00:12:30,600 on. We don't need them. We have our own kernels now. Afterwards we pack our base 106 00:12:30,600 --> 00:12:36,315 system. Of course we need to unmount any mount points in the base system we created 107 00:12:36,315 --> 00:12:42,943 and create a squash file system from it. And these three files, the VM Linux, the 108 00:12:42,943 --> 00:12:50,120 initial Ramdisk and the squashfs file system, they are the major pillars of our 109 00:12:50,120 --> 00:12:55,680 live environment. If we have those three, we are actually ready to go now. We can 110 00:12:55,680 --> 00:13:00,990 either use our existing bootloader from our extracted ISO file or we, of course, 111 00:13:00,990 --> 00:13:06,961 you can install your own bootloader in this directory. But well, it's easier to 112 00:13:06,961 --> 00:13:13,874 use the existing. For security purposes and for hardening your live environment I 113 00:13:13,874 --> 00:13:22,148 would recommend to put the MD5 sums of all files in your ISO file in this 114 00:13:22,148 --> 00:13:28,375 environment, because as soon as you, for example, copy your distribution, for 115 00:13:28,375 --> 00:13:36,934 example, on a pen drive on which is fat or ext4 formatted, it could be modified by 116 00:13:36,934 --> 00:13:43,430 anyone. And that's why it's much better to verify the MD5 sums at the boot process. 117 00:13:43,430 --> 00:13:50,379 Casper will automatically detect whether the MD5 sum, the txt file exists and if 118 00:13:50,379 --> 00:13:56,564 it exists, it will automatically verify these MD5 sums. OK, now we are actually 119 00:13:56,564 --> 00:14:03,382 ready to go, ready to pack up our ISO file. There's a famous tool called 120 00:14:03,382 --> 00:14:11,029 genisoimage. So generate an ISO image, that's a command you could use. The 121 00:14:11,029 --> 00:14:20,080 command takes care of legacy booting except of output file, the disk and the 122 00:14:20,080 --> 00:14:28,400 disk image and the -r parameter, you could leave out everything if you do not require 123 00:14:28,400 --> 00:14:37,320 legacy bootloaders. But if you want to obey legacy bootloaders, you should also 124 00:14:37,320 --> 00:14:44,960 hybritize your ISO file to be able to dd the ISO file to a pen drive. Otherwise, 125 00:14:44,960 --> 00:14:51,600 without the ISO hybrid command, it won't boot from, it would boot from a CD, but 126 00:14:51,600 --> 00:15:01,280 not from a pen drive. Yeah. OK, now we created our own Linux distribution. Well 127 00:15:01,280 --> 00:15:05,360 not that much. It's actually just a repacked Linux distribution. Now we talk 128 00:15:05,360 --> 00:15:11,040 about making it a real distribution. That's why we talk about making it 129 00:15:11,040 --> 00:15:18,560 installable. Why should we make it installable? Of course we could manually 130 00:15:18,560 --> 00:15:25,280 install our system. We could manually format our hard drive, extract our file system and 131 00:15:25,280 --> 00:15:32,960 manually create users and so on, et cetera, et cetera. But most of users 132 00:15:32,960 --> 00:15:39,920 prefer graphical way except of these geeks taking part at the rC3. That's why an 133 00:15:39,920 --> 00:15:48,400 installer may be useful. An installer does exactly what we need. It creates 134 00:15:48,400 --> 00:15:54,560 partitions, creates users, installs language-packs, offers installation of 135 00:15:54,560 --> 00:15:59,760 extra software and does everything the user might want. Creates users, connects 136 00:15:59,760 --> 00:16:05,760 to wifi, whatever. And there are three very common installer frameworks: 137 00:16:05,760 --> 00:16:12,080 Ubiquity, Calamaris and Anaconda. Now we'll have a look at these frameworks. 138 00:16:12,960 --> 00:16:21,760 Ubiquity is known from Ubuntu. It's ready to use if installed, but it's actually not 139 00:16:21,760 --> 00:16:30,320 at all customizable. Anyway, some hackers at GitHub found some ways to customize it 140 00:16:30,880 --> 00:16:36,320 and provided some good information, some helpful information in their repositories. 141 00:16:36,320 --> 00:16:42,880 The next one is Calamaris. Calamiris is highly customizable, is used by many 142 00:16:42,880 --> 00:16:50,720 distributions, including SUSE Linux, I think Xubuntu, so the XFCE Ubuntu flavor, 143 00:16:51,440 --> 00:16:59,840 DeepinOS, I think I am unsure. I think elementary as well. But if you only 144 00:16:59,840 --> 00:17:05,600 install it, it won't work. You need to customize it. The third one is known from 145 00:17:05,600 --> 00:17:13,840 the Fedora project. It's called Anaconda. It's a very advanced tool. It's able to 146 00:17:13,840 --> 00:17:19,520 update existing systems. If we have an old Fedora installation, we can boot our live 147 00:17:19,520 --> 00:17:26,640 CD and update the existing installation and moreover, very useful for admins is 148 00:17:26,640 --> 00:17:32,640 the remote functionality. It can be started via SSH and automatically update 149 00:17:32,640 --> 00:17:38,400 everything and automatically install and deploy the software we need. Um, yeah. 150 00:17:40,240 --> 00:17:46,720 Well, I already mentioned, for everything you can manually do and customize, there 151 00:17:46,720 --> 00:17:52,080 is a graphical tool. If you prefer the graphical way you could use the tool 152 00:17:52,080 --> 00:18:00,560 remastersys, aka LinuxRespin. That's the the older name. It offers a simple 153 00:18:00,560 --> 00:18:07,120 graphical user interface for packing your currently installed system into a live 154 00:18:07,120 --> 00:18:13,040 distribution, allowing to install it. It does not allow a high amount of 155 00:18:13,040 --> 00:18:16,880 customization in the graphical way, of course, you can manually perform it on the 156 00:18:16,880 --> 00:18:24,080 command prompt. But it's an easy way to, for example, backup your just installed 157 00:18:24,080 --> 00:18:29,920 system or you readily configured system to a pen drive to make it installable later. 158 00:18:34,160 --> 00:18:40,480 And now, Linux from scratch. I guess most of you already heard of Linux from scratch 159 00:18:40,480 --> 00:18:45,920 from scratch means, yeah, pretty much Linux from scratch. You start without 160 00:18:45,920 --> 00:18:51,680 anything. You start on your host system and create an own system from scratch. 161 00:18:53,120 --> 00:18:59,120 What does that mean? It means you should have lots of time. Lots of RAM, and many 162 00:18:59,120 --> 00:19:05,120 processor cores. Because you need to compile everything yourself. Beside the 163 00:19:05,120 --> 00:19:10,480 kernel, you need to compile the whole userland. If you want, you can include a 164 00:19:10,480 --> 00:19:15,840 package manager or whatever. But if you make real Linux from scratch, you do not 165 00:19:15,840 --> 00:19:20,400 take any binaries but compile everything yourself, everything on your own. 166 00:19:21,440 --> 00:19:28,720 Currently, minimally, you need to compile 226 dependencies. But in some years it 167 00:19:28,720 --> 00:19:36,000 will be some more dependencies and the amount of dependencies required increases. 168 00:19:36,000 --> 00:19:40,800 Yeah, after you compile everything, of course you need to configure everything, 169 00:19:40,800 --> 00:19:47,120 need to provide a bootloader. But as soon as this is done, as soon as you created 170 00:19:47,760 --> 00:19:53,840 your own base system, your own system from scratch inside a fake root, it's the same 171 00:19:53,840 --> 00:20:02,720 as we already talked on during live system customization. As soon as you have this 172 00:20:02,720 --> 00:20:08,080 fake root, you simply pack it to an image, so a squash file system, write it on a 173 00:20:08,080 --> 00:20:13,840 disk, provide a bootloader, and that's it. OK, one more word. I will talk about 174 00:20:14,560 --> 00:20:18,960 maintaining architecture for distributions. I will talk about package 175 00:20:18,960 --> 00:20:24,880 servers. If you have a distribution, you usually provide updates, security fixes 176 00:20:25,680 --> 00:20:32,960 and whatsoever. Maybe software the users may install later. Yeah, that's a package 177 00:20:32,960 --> 00:20:40,320 mirror, a packet server. From Debian, you can simply mirror existing servers, for 178 00:20:40,320 --> 00:20:47,920 example, the Debian or the Ubuntu or the Linux Mint servers by using the tool apt- 179 00:20:47,920 --> 00:20:54,320 mirror. It simply mirrors, means it does not allow any customization. For ArchLinux 180 00:20:54,320 --> 00:21:01,440 it's easy as well. You can simply use vftp and crontab to download the packages and 181 00:21:01,440 --> 00:21:09,360 provide them on your host system, on your server and synchronize from them. That's 182 00:21:09,360 --> 00:21:15,440 much easier than in the Debian environment. But usually if you have a 183 00:21:15,440 --> 00:21:21,520 distribution, you want to customize it. At Debian, the tool aptly is very widely 184 00:21:21,520 --> 00:21:26,080 spread because it allows very advanced management of different snapshots from 185 00:21:26,080 --> 00:21:31,200 different repositories, mirrors, local repositories, single packages, allows 186 00:21:31,200 --> 00:21:36,560 diffs and so on. But it does not allow any optimization. So good luck at scripting on 187 00:21:36,560 --> 00:21:43,840 your own. If you look at ArchLinux, for example, you would usually write your own 188 00:21:44,560 --> 00:21:50,320 package build files as you know them from the Arch user repository and include the 189 00:21:50,880 --> 00:21:57,920 readily build packages into your directory from which you serve your updates. That's 190 00:21:57,920 --> 00:22:02,080 much easier than in the Debian environment. I still found one more 191 00:22:02,080 --> 00:22:08,400 graphic explaining aptly. That's what aptly can do. It can do everything. Well, 192 00:22:08,400 --> 00:22:12,400 you have mirrors, you have local repositories. You can take snapshots, take 193 00:22:12,400 --> 00:22:17,040 differences from snapshots, publish these differences, publish merged snapshots of 194 00:22:17,040 --> 00:22:23,680 them, publish single snapshots, diff everything and do whatever you want. It's 195 00:22:23,680 --> 00:22:30,640 very performant, the tool and yeah, but it's hard to use. It requires some time to 196 00:22:30,640 --> 00:22:38,720 be learned. Yeah, that's it already on remixing Linux. Now there should be an 197 00:22:38,720 --> 00:22:44,640 Q&A. Thank you very much for your attention of this talk. I hope I could 198 00:22:44,640 --> 00:22:53,040 give you a good introduction into remixing Linux and good luck at coding and enjoy 199 00:22:53,040 --> 00:22:56,904 your Remote Chaos Experience. 200 00:22:56,904 --> 00:23:00,731 *Music* 201 00:23:00,731 --> 00:23:13,160 Subtitles created by c3subtitles.de in the year 2021. Join, and help us!