0 00:00:00,000 --> 00:00:30,000 This subtitle is not finished yet. If you are able to, please support us and watch the talk in amara for the last changes: https://c3subtitles.de/talk/973 Thanks! 1 00:00:00,000 --> 00:00:19,759 *35c3 prerol music* 2 00:00:19,759 --> 00:00:26,630 Herald: So Trammell Hudson, who is standing here, he's taking things apart. 3 00:00:26,630 --> 00:00:34,370 Don't worry not life on stage, but he will give us a proof of concept and some 4 00:00:34,370 --> 00:00:39,559 details and functionalities about hardware implants. So the same things that we heard 5 00:00:39,559 --> 00:00:45,480 from Bloomberg article talking about Apple and super microcomputers with implants 6 00:00:45,480 --> 00:00:52,879 that, yeah, were implanted into those, into those computers. And I'm really 7 00:00:52,879 --> 00:00:57,680 excited to see this in action. Please give a warm round of applause to Trammel 8 00:00:57,680 --> 00:01:02,590 Hudson! 9 00:01:02,590 --> 00:01:07,510 *applause* 10 00:01:07,510 --> 00:01:11,600 Trammell: Before we begin talking about hardware implants just two quick 11 00:01:11,600 --> 00:01:16,310 disclaimers. The first from my employer Two Sigma investments as it says are 12 00:01:16,310 --> 00:01:21,910 chocolate bars. This is not investment advice. And secondly I don't actually know 13 00:01:21,910 --> 00:01:26,920 what the story is behind the super micro story. No one outside of Bloomberg and 14 00:01:26,920 --> 00:01:32,450 their sources do. But I have spent a lot of time thinking about hardware implants 15 00:01:32,450 --> 00:01:38,200 starting with the thunderstrike firmware attack against mac books as well as the 16 00:01:38,200 --> 00:01:45,420 thunderstrike 2 where we were able to get software to write into the firmware on the 17 00:01:45,420 --> 00:01:50,560 mac books. I've also been thinking a lot about how to defend against hardware 18 00:01:50,560 --> 00:01:54,420 implants with things like the heads firmware for slightly more secure laptops 19 00:01:54,420 --> 00:01:59,420 and also as part of my co-lead on the Linux boot project. We're thinking about 20 00:01:59,420 --> 00:02:10,080 how to protect servers from physical and software attacks. So with all of this 21 00:02:10,080 --> 00:02:14,910 concentrated thinking about firmware and hardware attacks, I was really excited 22 00:02:14,910 --> 00:02:20,720 when I saw the Bloomberg story back in October. But what really intrigued me was 23 00:02:20,720 --> 00:02:26,440 the animated image that they had at the header that highlighted one small part of 24 00:02:26,440 --> 00:02:32,920 the board as where the implant was, but what I found really interesting is that is 25 00:02:32,920 --> 00:02:40,250 exactly where I would install a hardware implant as they described on the SPI bus. 26 00:02:40,250 --> 00:02:44,610 A lot of other people in the hardware and from our security community thought it 27 00:02:44,610 --> 00:02:50,140 sounded plausible. Other people pointed out that supply chain attacks come up 28 00:02:50,140 --> 00:02:56,070 periodically and they are definitely a concern. Some people thought the attack as 29 00:02:56,070 --> 00:03:03,320 described was entirely implausible and in general we sort of had a Whiskey Tango 30 00:03:03,320 --> 00:03:08,160 Foxtrot moment as everybody scrambled to figure out what's going on inside their 31 00:03:08,160 --> 00:03:14,540 machines. So, let's step back very quickly and review what the key claims that 32 00:03:14,540 --> 00:03:22,340 Bloomberg alleged happened. First they said that Amazon's testers found a tiny 33 00:03:22,340 --> 00:03:27,250 microchip that wasn't part of the board's original design that had been disguised to 34 00:03:27,250 --> 00:03:33,350 look like a signaling condition signal condition coupler and that these illicit 35 00:03:33,350 --> 00:03:39,790 chips were connected to the baseboard management controller or the BMC which 36 00:03:39,790 --> 00:03:44,210 gave them access to machines that were turned off. That might sound kind of 37 00:03:44,210 --> 00:03:49,870 extreme, but that's actually what the role of the BMC is, that in most servers the 38 00:03:49,870 --> 00:03:55,280 BMC is running any time the machine is hooked up to power and it's connected to 39 00:03:55,280 --> 00:04:01,910 the power supplies so that it can turn the machine on and turn it off. Frequently you 40 00:04:01,910 --> 00:04:06,780 want to be able to do this over a network so it has its own dedicated LAN port but 41 00:04:06,780 --> 00:04:14,180 it can also share the LAN port with the with the main system. Serial over LAN is a 42 00:04:14,180 --> 00:04:19,180 really useful way to debug the systems so it provides that functionality. It can 43 00:04:19,180 --> 00:04:27,350 also provide fake USB volumes to allow to to do unintended OS installation. A lot of 44 00:04:27,350 --> 00:04:33,430 sites also won't remote KVM so it has VGA but that VGA support means that it's on 45 00:04:33,430 --> 00:04:40,370 the PCIe BUS and because some PCIe it can do DMA into main memory. It also is 46 00:04:40,370 --> 00:04:47,000 typically muxed into the SPI flash for the host firmware, which allows it to 47 00:04:47,000 --> 00:04:51,820 modify it and on some systems it's even connected to the TPM which allows it to 48 00:04:51,820 --> 00:04:59,930 circumvent the corporate of trust. So with all of this capability inside this chip 49 00:04:59,930 --> 00:05:06,919 it's really unfortunate that they are really not well put together. The head of 50 00:05:06,919 --> 00:05:11,150 Azure security says they have no protection against attacks. There's no 51 00:05:11,150 --> 00:05:15,530 ability to detect if an attack has happened and there's no ability to recover 52 00:05:15,530 --> 00:05:22,449 from an attack. So having a hardware implant on the BMC is a really big 53 00:05:22,449 --> 00:05:32,030 concern. The other claim in the article is that it affected 30 different companies 54 00:05:32,030 --> 00:05:39,930 including Apple and Bloomberg alleges that Apple found malicious chips independently 55 00:05:39,930 --> 00:05:44,980 on their super micro boards. Went to the FBI about it and that they then severed 56 00:05:44,980 --> 00:05:52,100 ties with Super Micro. This particular claim was interesting because it 57 00:05:52,100 --> 00:05:57,570 corroborated a story that had shown up back in early 2017 that Apple had removed 58 00:05:57,570 --> 00:06:03,050 Super Micro from their data centers. Apple denied that there was a firmware issue. 59 00:06:03,050 --> 00:06:10,190 But it's interesting that perhaps these two were related. The third set of claims 60 00:06:10,190 --> 00:06:16,090 is that on some of these implants they were actually put between the layers on 61 00:06:16,090 --> 00:06:23,210 the PCB and then the most explosive claim is that this was done by operatives from 62 00:06:23,210 --> 00:06:33,580 China, the Chinese People's Liberation Army. With a story with this you know this 63 00:06:33,580 --> 00:06:39,389 many claims and this significant of allegations we'd hoped that it would be 64 00:06:39,389 --> 00:06:45,430 really well sourced and for a normal story 17 independent sources that Bloomberg 65 00:06:45,430 --> 00:06:52,490 editors agreed to grant anonymity to, including six national security, two 66 00:06:52,490 --> 00:06:57,340 people inside of AWS and three senior insiders at Apple seems like pretty solid 67 00:06:57,340 --> 00:07:03,110 sourcing, except as soon as this article is published everyone denied it. The 68 00:07:03,110 --> 00:07:09,080 Director of National Intelligence said they'd seen no evidence of this. Amazon 69 00:07:09,080 --> 00:07:13,990 said that they've never found any issues of modified hardware nor have they been 70 00:07:13,990 --> 00:07:21,000 engaged with the government over it. Apple was even more blunt. CEO Tim Cook said 71 00:07:21,000 --> 00:07:27,590 this did not happen. There is no truth to this. And Super Micro wrote a fairly 72 00:07:27,590 --> 00:07:32,150 lengthy letter about what they do to protect their supply chain and why they 73 00:07:32,150 --> 00:07:38,990 think this attack did not happen. And it is worth going through to look at some of 74 00:07:38,990 --> 00:07:44,880 the things that they say that they do to protect their supply chain. They point out 75 00:07:44,880 --> 00:07:50,700 that if there's any unauthorized physical alterations during the manufacturing 76 00:07:50,700 --> 00:07:56,949 process other design elements would not match and those things would be detected. 77 00:07:56,949 --> 00:08:03,300 To sort of understand how circuit boards are made, I recently visited a PCB factory 78 00:08:03,300 --> 00:08:11,080 in Guangzhou. This is not a super micro factory. This is just a holiday photos. So 79 00:08:11,080 --> 00:08:16,760 in order to add new vias they would have to modify the drill files which would then 80 00:08:16,760 --> 00:08:22,050 get electroplated. If they had to add new traces, they would have to be able to 81 00:08:22,050 --> 00:08:29,400 subvert the masking and etching process and any changes to either the drills or 82 00:08:29,400 --> 00:08:34,909 the etching on individual layers would be caught by the optical inspection that's 83 00:08:34,909 --> 00:08:41,479 done on these bare circuit boards. Additionally the allegation that things 84 00:08:41,479 --> 00:08:47,110 were inserted between circuit boards would require that the lamination process be 85 00:08:47,110 --> 00:08:55,329 subverted and that the implant somehow aligned into the system. If that implant 86 00:08:55,329 --> 00:09:00,410 changes any of the connectivity the flying protesters would pick it up or the bed of 87 00:09:00,410 --> 00:09:05,980 nails testers which checks all of the connectivity of all the traces to make 88 00:09:05,980 --> 00:09:09,300 sure that there are no shorts and to make sure that everything that is supposed to 89 00:09:09,300 --> 00:09:16,679 be connected is electrically conductive. So it would be very difficult to 90 00:09:16,679 --> 00:09:22,110 circumvent the production process at this stage. And it also would be very difficult 91 00:09:22,110 --> 00:09:27,709 to contain because the PCB factory doesn't know which customers are going to receive 92 00:09:27,709 --> 00:09:34,470 those circuit boards. Super Micro also points out that during the assembly 93 00:09:34,470 --> 00:09:40,480 process when the parts are installed they have their employees on site the whole 94 00:09:40,480 --> 00:09:47,559 time. On my same holiday trip I also visited some PCB assembly companies and 95 00:09:47,559 --> 00:09:53,589 spoke with companies that are using doing contract manufacturing and they said that 96 00:09:53,589 --> 00:09:59,089 they also send their employees to the production line to observe the pick and 97 00:09:59,089 --> 00:10:05,600 place machines and the reflow and the rest of the surface mount assembly. Their big 98 00:10:05,600 --> 00:10:10,089 concern is that if they don't have someone there the parts that are fed in the pick 99 00:10:10,089 --> 00:10:17,660 in place will be replaced with either counterfeits or with salvaged parts. I 100 00:10:17,660 --> 00:10:23,459 visited the electronics market in ??????? bay where there are people desoldering 101 00:10:23,459 --> 00:10:29,190 e-waste and then sorting the parts into bins and selling these salvaged components 102 00:10:29,190 --> 00:10:34,589 by the kilo and for a few extra renminbi they'll put them on rails for you so that 103 00:10:34,589 --> 00:10:41,660 you can save a few pennies on your production process. The other concern that 104 00:10:41,660 --> 00:10:46,489 these companies have, is not just salvaged parts but straight up counterfeits. 105 00:10:46,489 --> 00:10:52,439 Especially for things that cost more than a few dollars each. The Arduino community 106 00:10:52,439 --> 00:10:59,139 was hit a few years ago with a bunch of counterfeit FTDI chips where the internal 107 00:10:59,139 --> 00:11:07,600 construction was entirely different. In this case it caused reliability issues but 108 00:11:07,600 --> 00:11:11,550 you can imagine from a security perspective this is really worrisome that 109 00:11:11,550 --> 00:11:15,709 parts that look identical might have completely different functionality inside 110 00:11:15,709 --> 00:11:25,379 of them. Super Micro also mentions that they X-ray their main boards to look for 111 00:11:25,379 --> 00:11:32,000 anomalies and I wasn't able to take any photos inside the factory there was doing 112 00:11:32,000 --> 00:11:38,230 x-rays. But in this Wikipedia photo we can clearly see active components like this 113 00:11:38,230 --> 00:11:45,670 SOIC chip are different from things like the SMD resistors and capacitors. So if an 114 00:11:45,670 --> 00:11:51,220 attacker were trying to subvert the supply chain by putting a disguise component it 115 00:11:51,220 --> 00:11:56,670 could be detected at this step. Another interesting thing in this photo are these 116 00:11:56,670 --> 00:12:02,680 inductors that are encased in dip packages. This is really common in a lot 117 00:12:02,680 --> 00:12:07,439 of Ethernet boards and occasionally people have thought they had some sort of 118 00:12:07,439 --> 00:12:13,589 hardware implant when they found inductors in their ethernet jacks but it's pretty 119 00:12:13,589 --> 00:12:19,799 it's fairly common and it shows it pretty clearly on the x-ray. Some other security 120 00:12:19,799 --> 00:12:26,069 researchers like Sophia D'Antoine did an extensive teardown of Super Micro boards 121 00:12:26,069 --> 00:12:33,439 including X-ray analysis and her group found a few oddities but nothing.. they 122 00:12:33,439 --> 00:12:37,529 didn't find anything malicious. There were no smoking guns. They just appeared to be 123 00:12:37,529 --> 00:12:43,190 sort of supply chain type things. You can read her blog post for more details about 124 00:12:43,190 --> 00:12:49,319 where they found things that shouldn't have been there. But turned out to be just 125 00:12:49,319 --> 00:13:00,879 actual signal condition components. So super micro in their ???? letter, they 126 00:13:00,879 --> 00:13:07,239 keep reenforcing that the manufacturing process that is the assembly process, it's 127 00:13:07,239 --> 00:13:11,179 during the manufacturing process and I agree with them. It would be very 128 00:13:11,179 --> 00:13:17,939 difficult to circumvent security in a reasonable way in that part of the 129 00:13:17,939 --> 00:13:23,189 process. But that's not the only place this could happen. We know that national 130 00:13:23,189 --> 00:13:30,309 security agencies intercept shipments of computer hardware and then have their 131 00:13:30,309 --> 00:13:37,249 tailored access operations open the computers, install hardware implants, 132 00:13:37,249 --> 00:13:43,670 reseal them and then have them continue on their way in shipment. The NSA even has a 133 00:13:43,670 --> 00:13:51,199 catalog of hardware implants like this JTAG implant Ethernet jacks with embedded 134 00:13:51,199 --> 00:13:57,009 computers in them as well as firmware specific ones that target servers SNM(?) 135 00:13:57,009 --> 00:14:05,490 and then some that can do data exfiltration via RF. So that's sort of 136 00:14:05,490 --> 00:14:09,481 tailored access operations is really ideal for this supply chain attack because it 137 00:14:09,481 --> 00:14:16,699 allows them to contain the exploit to a single customer. It allows them fairly 138 00:14:16,699 --> 00:14:21,180 good concealment as well as good cover that if it's discovered it's really hard 139 00:14:21,180 --> 00:14:25,769 to attribute where things went wrong. Now unlike if you find something inside your 140 00:14:25,769 --> 00:14:34,230 motherboard between the layers you know that had to have happened at the factory. 141 00:14:34,230 --> 00:14:47,040 So Super Micro also claim that this was technically implausible, that it was 142 00:14:47,040 --> 00:14:52,559 highly unlikely that unauthorized hardware would function properly because a third 143 00:14:52,559 --> 00:15:02,470 party with lack of complete knowledge of the design. I think that's inaccurate, 144 00:15:02,470 --> 00:15:07,639 both because we know the NSA does it and also because I have done it. 145 00:15:07,639 --> 00:15:10,319 *laughter* 146 00:15:10,319 --> 00:15:16,059 Really, all that you need to know is that these are common components. These flash 147 00:15:16,059 --> 00:15:20,310 chips show up on all the boards. You can search the internet for the data sheet and 148 00:15:20,310 --> 00:15:25,989 find exactly how it's wired into the rest of the system. And the only thing that we 149 00:15:25,989 --> 00:15:33,499 need to know to communicate to the BMC is the serial output pin from this component, 150 00:15:33,499 --> 00:15:43,429 so the BMC flash is connected over to the BMC CPU via the serial output and it goes 151 00:15:43,429 --> 00:15:51,589 through a small series resistor and that is where my implant goes in. Mine's a 152 00:15:51,589 --> 00:15:56,670 little bit larger than that resistor. It clicks onto the board and it has a small 153 00:15:56,670 --> 00:16:03,009 FPGA that hangs offside but it's completely plausible to fit it into 154 00:16:03,009 --> 00:16:12,139 something that small in fact a modern ARM M0 fits in the space of two transistors 155 00:16:12,139 --> 00:16:18,350 from a 65 002 from a few years ago. The Moore's Law means we can pack an amazing 156 00:16:18,350 --> 00:16:28,329 amount of CPU into a very very small amount of space. So on that 0 6 0 3 157 00:16:28,329 --> 00:16:36,100 resistor could fit around 100 cortex M0 it would be plenty powerful for this system. 158 00:16:36,100 --> 00:16:42,379 The problem is we only have those two pins so ordinarily on the spy flashing you need 159 00:16:42,379 --> 00:16:47,720 at least six pens but we don't have power and ground so we have to passively power 160 00:16:47,720 --> 00:16:53,059 this through the data signal that's passing through it. We don't have the chip 161 00:16:53,059 --> 00:16:59,959 select pin so we have to guess when this chip has been talked to. We don't have the 162 00:16:59,959 --> 00:17:04,980 data input pin so we don't know what addresses are being read or what commands 163 00:17:04,980 --> 00:17:11,190 are being sent. We have to reconstruct it from the data output pin and we also don't 164 00:17:11,190 --> 00:17:18,900 have a clock pin so we have to figure out how to synchronize to that clock. Lastly 165 00:17:18,900 --> 00:17:22,890 we don't have the ability to make arbitrary data changes. All we can do is 166 00:17:22,890 --> 00:17:29,060 disconnect the pin from the BMC so we can only turn 1 bits into 0 bits. We can't go 167 00:17:29,060 --> 00:17:35,300 the other way around. So with these limitations we can still do some pretty 168 00:17:35,300 --> 00:17:40,920 interesting things. Recovering the clock is actually pretty easy. We can look at 169 00:17:40,920 --> 00:17:49,670 the data stream and find the shortest bit transitions from 0 1 0 or 1 0 1 to 170 00:17:49,670 --> 00:17:55,060 estimate what the clock is which allows us to then reconstruct that data stream being 171 00:17:55,060 --> 00:18:00,870 sent to the BMC and if we look at the flash contents we can see that a lot of it 172 00:18:00,870 --> 00:18:07,570 is being fairly random noise but a lot of it is all white which in this case would 173 00:18:07,570 --> 00:18:15,110 mean that it's all one bits. So if we look at the way the flash is organized we can 174 00:18:15,110 --> 00:18:19,380 see there's the u-boot bootloader and that's executable. That's kind of 175 00:18:19,380 --> 00:18:25,230 difficult to make useful changes in, the kernel and the root file system are both 176 00:18:25,230 --> 00:18:33,040 compressed so that they look effectively like random noise but the nvram region is 177 00:18:33,040 --> 00:18:41,660 a jffs2 file system and this file system ??? 3 Megs, it's mostly empty and all that 178 00:18:41,660 --> 00:18:50,040 empty space is F F which is all ones. So this is plenty of ones for us to work on. 179 00:18:50,040 --> 00:18:55,380 Additionally it has fairly nice headers that we can we can match on. So when we 180 00:18:55,380 --> 00:19:00,570 see these magic bit masks we know when we've entered different parts of the file 181 00:19:00,570 --> 00:19:06,990 system. So given that we can now reconstruct the clock we can figure out 182 00:19:06,990 --> 00:19:13,310 where we are in the file system. This hardware implant can start to inject new 183 00:19:13,310 --> 00:19:20,320 data into what was the empty space. So this short file that we put in here is a 184 00:19:20,320 --> 00:19:28,020 small shell script and it is one of the network configuration scripts, so this is 185 00:19:28,020 --> 00:19:37,350 where I'm going to try a live demo and I hope this works. We're running in qemu 186 00:19:37,350 --> 00:19:45,660 since I didn't bring a Super Micro board and what we have on the left is the flash 187 00:19:45,660 --> 00:19:50,530 console excuse me the hardware implant console. And then on the right we have the 188 00:19:50,530 --> 00:19:57,353 serial console from the BMC so we can see it has loaded the kernel and in a second 189 00:19:57,353 --> 00:20:03,430 it's going to we should see a bunch of traffic, okay, so the implant is active. 190 00:20:03,430 --> 00:20:10,450 It has replaced the data when that nvram file system was mounted the BMC is now 191 00:20:10,450 --> 00:20:18,780 continuing on doing its set up. It's going to load a bunch of device drivers for that 192 00:20:18,780 --> 00:20:24,250 video. It pauses here for some reason that I haven't diagnosed because that's that's 193 00:20:24,250 --> 00:20:27,040 not my job. 194 00:20:27,040 --> 00:20:29,140 *laughter* 195 00:20:29,140 --> 00:20:33,020 And eventually it's going to configure the networks and it does that by running that 196 00:20:33,020 --> 00:20:43,010 shell script off of the nvram partition here it starts KVM stuff brings up some 197 00:20:43,010 --> 00:20:53,390 things. Allright. *applause* 198 00:20:53,390 --> 00:21:01,920 OK. So luckily we got to that point without having to fake the demo. In the 199 00:21:01,920 --> 00:21:07,820 hardware it's really flaky. My version works about one in eight times. But it 200 00:21:07,820 --> 00:21:12,041 doesn't typically cause a crash. So that's actually good for concealment because it 201 00:21:12,041 --> 00:21:17,850 becomes now much harder to determine which machines are affected. In qemu because 202 00:21:17,850 --> 00:21:21,640 it's emulating, it's a little more reliable but it's still it's only two out 203 00:21:21,640 --> 00:21:26,760 of three. If we let the BMC boot a little bit further it actually prints out this 204 00:21:26,760 --> 00:21:32,120 message. And if you hit enter it drops you to a shell with no password and you can 205 00:21:32,120 --> 00:21:38,170 then just run commands as root on the BMC and that's a lot easier than all this 206 00:21:38,170 --> 00:21:43,440 stuff with the SPI bus if you wanted to build a hardware implant against it. I 207 00:21:43,440 --> 00:21:48,540 don't know where the serial port is on the on the Super Micro but on a different tier 208 00:21:48,540 --> 00:21:54,030 1 server mainboard I was able to probe around the oscilloscope and locate the 209 00:21:54,030 --> 00:22:00,830 serial console for the BMC. Figure out it's 115 kbaud and it has the same code 210 00:22:00,830 --> 00:22:06,050 that you hit enter and you can run commands there. So that's a much easier 211 00:22:06,050 --> 00:22:11,990 way to do it. A big question a lot of people have is how do we actually detect 212 00:22:11,990 --> 00:22:18,100 this sort of flash implant. A lot of high assurance sites replace all of their roms 213 00:22:18,100 --> 00:22:22,760 with ones that they flash themselves but that doesn't get rid of the implant 214 00:22:22,760 --> 00:22:28,960 because it's outside of the ROM chip. Likewise reading the ROM chip doesn't show 215 00:22:28,960 --> 00:22:35,321 anything because it's not in the ROM itself it's it's outside of it. Even 216 00:22:35,321 --> 00:22:40,650 hooking up a logic analyzer to the bus and watching as the machine boots and seeing 217 00:22:40,650 --> 00:22:45,780 the data stream coming out of the flash won't actually reveal the implant because 218 00:22:45,780 --> 00:22:51,770 you'd have to put the logic probes on the PGA pads on the flat on the BMC itself. 219 00:22:51,770 --> 00:22:58,140 And that's a much harder task. Some people think "oh well we can see the weird 220 00:22:58,140 --> 00:23:03,150 network traffic when the BMC tries to exfiltrate the data" but that would be 221 00:23:03,150 --> 00:23:08,030 that's only one way for the BMC to affect things. There is a great talk a few years 222 00:23:08,030 --> 00:23:13,410 ago at DefCon from Intel ATR where they showed how something that can control the 223 00:23:13,410 --> 00:23:19,020 system firmware can backdoor hypervisors. And then they gave a use case where a 224 00:23:19,020 --> 00:23:26,180 unprivileged guest on a cloud system could read all of the rest of physical memory so 225 00:23:26,180 --> 00:23:34,760 it could see all of the other guests memory. So what do we do? The big problems 226 00:23:34,760 --> 00:23:39,560 is the BMC has way too many privileges. It's connected to pretty much everything 227 00:23:39,560 --> 00:23:46,650 in the system but the BMC is not our only concern. As @whitequark said, our PCs are 228 00:23:46,650 --> 00:23:52,300 just a bunch of embedded devices in a trench coat and they all have firmware. In 229 00:23:52,300 --> 00:23:56,680 fact pretty much everything on your system more complex than a resistor probably has 230 00:23:56,680 --> 00:24:01,270 firmware and if you have one of those Super Micro implants maybe even your 231 00:24:01,270 --> 00:24:08,500 resistors have firmware as well. I've found that the firmware and things like 232 00:24:08,500 --> 00:24:15,150 the power supplies can be used to gain code execution on the BMC. It's really 233 00:24:15,150 --> 00:24:20,750 interesting how tightly connected all of our systems are. And as Joe Fit's pointed 234 00:24:20,750 --> 00:24:26,700 out in his blackhat ???? talk, these are not multimillion dollar attacks these are 235 00:24:26,700 --> 00:24:33,500 five euro bits of hardware that we now have to really be worried about. I really 236 00:24:33,500 --> 00:24:38,480 like the guidelines that NIST has published that suggests that we think 237 00:24:38,480 --> 00:24:43,650 about our systems more in this holistic manner. Although the interpreting pretty 238 00:24:43,650 --> 00:24:50,290 much everything into the TPM is the trusted platform module for doing this 239 00:24:50,290 --> 00:24:55,580 attestation and I think we as a community need to do more to use the TPM. There 240 00:24:55,580 --> 00:25:01,060 actually a really good tool for securing our systems but they are also potentially 241 00:25:01,060 --> 00:25:08,030 subject to their own hardware implants. The NCC Group TPM genie is able to subvert 242 00:25:08,030 --> 00:25:14,600 the core root of trust by interposing on the TPM. So a lot of folks are proposing 243 00:25:14,600 --> 00:25:19,160 we should move to other trusted execution environments like SGX or Trustzone. And I 244 00:25:19,160 --> 00:25:24,960 think these have a lot of promise especially for trusted cloud computing. 245 00:25:24,960 --> 00:25:30,970 There also is a lot of innovation in the hardware roots of trust going on right now 246 00:25:30,970 --> 00:25:34,860 between the Google Titan, which initially was for their servers and is now showing 247 00:25:34,860 --> 00:25:39,740 up on all of their chrome books. The Microsoft Cerberus chip which again is the 248 00:25:39,740 --> 00:25:46,710 Azure system. They're actually publishing their firmware and the ASIC design so that 249 00:25:46,710 --> 00:25:49,880 people can have a little more faith in it and they hope it will become an open 250 00:25:49,880 --> 00:25:56,780 standard. And companies like Apple have also gone their own way. With the T2 and 251 00:25:56,780 --> 00:26:00,620 the T2's are really amazing chip for securing systems. But it does so at the 252 00:26:00,620 --> 00:26:06,790 expense of user freedom and that gets in the way of what I think the real way that 253 00:26:06,790 --> 00:26:11,130 we need to.. we need to solve this problem. We need to get rid of a lot of 254 00:26:11,130 --> 00:26:18,830 these secrets. Counter to what the Super Micro CEO said, having a secret 255 00:26:18,830 --> 00:26:22,770 motherboard design does not make you more secure. Things like the Open Compute 256 00:26:22,770 --> 00:26:27,140 hardware I think is a good vision for how we can move forward that when you buy an 257 00:26:27,140 --> 00:26:33,030 Open Compute server it comes with full schematics and gerber files. So that 258 00:26:33,030 --> 00:26:37,910 motivated customers can verify that the systems that they're buying are the ones 259 00:26:37,910 --> 00:26:42,140 that they think they that they're buying that all of the components are what they 260 00:26:42,140 --> 00:26:49,250 think they should be. I think the firmware also needs more openness. Ronald Minnich, 261 00:26:49,250 --> 00:26:56,150 Google is my co-lead on Linux boot project and we think that Linux in the firmware is 262 00:26:56,150 --> 00:27:03,821 a way forward to get a more secure more flexible and more resilient system. We're 263 00:27:03,821 --> 00:27:09,981 working with a spin off project called micro BMC that is using the Linux boot 264 00:27:09,981 --> 00:27:16,580 tools to build BMC firmware and this is opensource. It's reproducibly built it can 265 00:27:16,580 --> 00:27:22,740 work with roots of trust attestation. It's written in a memory safe language since 266 00:27:22,740 --> 00:27:27,740 it's a Google collaboration and go. And more importantly we've thrown away all of 267 00:27:27,740 --> 00:27:31,240 the legacy features that have been a source of a lot of security 268 00:27:31,240 --> 00:27:40,960 vulnerabilities in these systems. So did it happen? I don't know. Is it technically 269 00:27:40,960 --> 00:27:44,520 possible? I think so. I hope I've convinced all of you that this is 270 00:27:44,520 --> 00:27:50,770 definitely a technical possibility that we need to be concerned about and I hope that 271 00:27:50,770 --> 00:27:56,260 the way forward through hardware roots of trust with attestation and more 272 00:27:56,260 --> 00:28:01,400 importantly with open hardware so that we know that what the machines were running 273 00:28:01,400 --> 00:28:07,130 are running code that we know.. the code that we've built that we understand and 274 00:28:07,130 --> 00:28:13,080 that we can actually have a good chance of being able to take control back of them. 275 00:28:13,080 --> 00:28:18,300 If you're interested in more discussion on this and also on open firmware, there's an 276 00:28:18,300 --> 00:28:23,850 assembly here in this hall that has a bunch folks working on a core boot and 277 00:28:23,850 --> 00:28:29,110 Linux boot and a lot of these projects where you can help contribute and you can 278 00:28:29,110 --> 00:28:37,510 help also pressure vendors to make these this standard and a way forward for a more 279 00:28:37,510 --> 00:28:42,000 secure computing. So thank you all for coming. And I really enjoyed the chance to 280 00:28:42,000 --> 00:28:50,380 show off my modship of the state. 281 00:28:50,380 --> 00:28:56,030 *applause* 282 00:28:56,030 --> 00:29:02,600 Herald: Geat talk, thank you very much Trammel. We have 10 minutes for questions 283 00:29:02,600 --> 00:29:11,080 so please line up at the microphones if you have questions. And we also have a 284 00:29:11,080 --> 00:29:25,390 signal angel probably with questions from the internet. So any questions? Microphone 285 00:29:25,390 --> 00:29:29,870 number three? Mic 3: Yes, I was going to ask, what's 286 00:29:29,870 --> 00:29:35,650 your opinion on the Talos systems? The openPOWER based ones? 287 00:29:35,650 --> 00:29:41,830 Trammell: So the question is about the Talos power 9 based systems power 9 is a 288 00:29:41,830 --> 00:29:48,490 really interesting architecture. The.. it is using a open firmware very similar to 289 00:29:48,490 --> 00:29:55,250 Linux boot called Petty(??) boot that moves Linux into the bootloader. I'm a big 290 00:29:55,250 --> 00:29:58,860 fan. There's a lot of folks in the opensource community who are very excited 291 00:29:58,860 --> 00:30:07,540 about it. I'm hoping that there would be more power nine systems coming out. I'm 292 00:30:07,540 --> 00:30:13,130 also very excited about the brisque five systems. I think having open source CPUs 293 00:30:13,130 --> 00:30:19,310 use is a real way that we can have more assurance that our systems are what we 294 00:30:19,310 --> 00:30:22,800 think they are. Herald: Thank you, microphone number two 295 00:30:22,800 --> 00:30:27,100 please. Mic 2: Yes, thanks for the talk. I was 296 00:30:27,100 --> 00:30:32,810 wondering if you have just a scope probe over this serial, cause it's just a serial 297 00:30:32,810 --> 00:30:37,320 resistor which we're replacing. If you put just two scope probes on there and measure 298 00:30:37,320 --> 00:30:41,270 the voltage over it, in your situation would the voltage change there once in a 299 00:30:41,270 --> 00:30:42,400 while? Trammell: Yes, yes, yes. 300 00:30:42,400 --> 00:30:46,540 Mic 2: Well okay, in the normal case would it actually be quite consistent current. 301 00:30:46,540 --> 00:30:56,890 Or if you lowered the input impedance of the BMC chip who might already have fixed 302 00:30:56,890 --> 00:31:01,760 a part of the attack because the output sourcing current of your exploit is 303 00:31:01,760 --> 00:31:04,900 probably limited due to the limited supply you only can.. 304 00:31:04,900 --> 00:31:12,390 Herald: Your question please? Mic 2: Yes.. but.. do you see a way to get 305 00:31:12,390 --> 00:31:17,710 more power into your setup? Maybe using, well other power sources, other than the 306 00:31:17,710 --> 00:31:22,650 two pins, or maybe somewhere of.. Trammell: Well, so the question is about, 307 00:31:22,650 --> 00:31:28,420 would there be a way to do more arbitrary changes through redesigning the implant. 308 00:31:28,420 --> 00:31:34,190 One of the goals was to fit with only those two pins so that a single piece on 309 00:31:34,190 --> 00:31:38,900 the motherboard could be replaced. With a dual probe soldering iron and you can pop 310 00:31:38,900 --> 00:31:45,500 it out and stick a new one down in a matter of seconds. So, yes, if you have 311 00:31:45,500 --> 00:31:51,809 more pins where you can get more power from you can do much more interesting 312 00:31:51,809 --> 00:31:57,460 things. But that's.. would require a different set of changes to the 313 00:31:57,460 --> 00:32:02,480 motherboard. Herald: Thank you. Microphone 1 please. 314 00:32:02,480 --> 00:32:09,350 Mic 1: So, a lot of the -like- arguments that these implants were not feasible by a 315 00:32:09,350 --> 00:32:13,820 Super Micro where you also show the picture from the fab that you had to 316 00:32:13,820 --> 00:32:19,390 change the etching and the optical inspection and so on and so on. But how 317 00:32:19,390 --> 00:32:27,870 probable would you rate the fact that some acto just intercepted the manufacturing 318 00:32:27,870 --> 00:32:33,570 files and added that component already in the file because then all the optical 319 00:32:33,570 --> 00:32:38,810 inspection and that would all say well that matches what was sent to us. But that 320 00:32:38,810 --> 00:32:41,650 was not necessarily what Super Micro sent to the fab. 321 00:32:41,650 --> 00:32:44,900 Trammell: So the question is, could someone have modified all of the 322 00:32:44,900 --> 00:32:48,620 manufacturing files that went to the factory, and that's absolutely a 323 00:32:48,620 --> 00:32:54,520 possibility. But that's also very likely that that would be detected by Super Micro 324 00:32:54,520 --> 00:33:01,170 itself that in a lot of cases you don't necessarily want to trust the company that 325 00:33:01,170 --> 00:33:05,930 is making the product to also test it. And you probably want to have a separate 326 00:33:05,930 --> 00:33:11,059 company that does random spot checks to verify that the boards are actually being 327 00:33:11,059 --> 00:33:16,460 produced to the specification that you.. that you desire. So it's certainly 328 00:33:16,460 --> 00:33:24,050 possible and I really don't want to speculate as to the accuracy of that part 329 00:33:24,050 --> 00:33:31,030 of the story but yeah it would require quite a bit more changes. And also would 330 00:33:31,030 --> 00:33:34,679 be much more likely to be detected in the spot check. 331 00:33:34,679 --> 00:33:38,230 Herald: Great. Microphone number two please. 332 00:33:38,230 --> 00:33:44,510 Mic 2: Yes, for a lot of motherboards there are also quite a few components not 333 00:33:44,510 --> 00:33:53,750 populated some of which are on which you could consider sensitive myths. Wouldn't 334 00:33:53,750 --> 00:33:59,430 that make it. Yeah exactly. Wouldn't that make it very easy to do just pop something 335 00:33:59,430 --> 00:34:04,540 on there in parallel with one of the components and not have it be detected 336 00:34:04,540 --> 00:34:08,329 because it's like the board is modified. There is a component or you have no way of 337 00:34:08,329 --> 00:34:11,490 telling whether it had to be populated or not? 338 00:34:11,490 --> 00:34:18,599 Trammell: Super Micro puts a lot of extra pads on the board in this one particular 339 00:34:18,599 --> 00:34:28,700 one they have both 8 pin and 16 pin flash chip pads that are just in parallel 340 00:34:28,700 --> 00:34:32,989 together. So depending on which chip is cheaper that day of the week or who knows 341 00:34:32,989 --> 00:34:38,419 what, they will populate one or the other. So that's why in this particular photo 342 00:34:38,419 --> 00:34:47,950 having the position of that circle on the data output pin is very very interesting. 343 00:34:47,950 --> 00:34:56,659 Herald: Question answered? Okay. So one more question on microphone number two 344 00:34:56,659 --> 00:35:00,400 please? Mic 2: How far can signing of firmware be 345 00:35:00,400 --> 00:35:06,470 a solution to this problem? Trammell: Signing firmware solves a lot of 346 00:35:06,470 --> 00:35:13,401 the issues. It does however not all typically not all of the firmware are 347 00:35:13,401 --> 00:35:21,020 signed specifically is probably to be signed in in a modern BMC. The kernel and 348 00:35:21,020 --> 00:35:25,789 maybe the root file system might be signed. But the envy of RAM file system in 349 00:35:25,789 --> 00:35:32,589 this BMC is designed to be user modifiable so it can't be signed by the manufacturer, 350 00:35:32,589 --> 00:35:41,340 so this sort of attack would work against a signed BMC just as well. Also the "Hit 351 00:35:41,340 --> 00:35:49,509 enter to get a serial console" attack circumvents any signing. There are things 352 00:35:49,509 --> 00:35:56,140 on the host firmware on the x86 like boot card that do a really good job of making 353 00:35:56,140 --> 00:36:01,520 it harder to get code execution during the boot process. But there have been several 354 00:36:01,520 --> 00:36:07,720 CVEs where it has been implemented poorly. So even though signature's the firmware is 355 00:36:07,720 --> 00:36:13,800 signed, people have still managed to get code execution during that process. 356 00:36:13,800 --> 00:36:18,329 Herald: Great. Thank you Trammell Hudson again, a warm round of applause, thank you 357 00:36:18,329 --> 00:36:21,009 very much! 358 00:36:21,009 --> 00:36:24,009 *applause* 359 00:36:24,009 --> 00:36:25,529 *35c3 postrol music* 360 00:36:25,529 --> 00:36:52,000 Subtitles created by c3subtitles.de in the year 2021. Join, and help us!