G33K @ Work » Hacking

Easterhegg 2011 Talk

Andy — Mon, 25 Apr 2011 17:35:32 +0000

Dieses Jahr fand das Easterhegg wieder in Hamburg statt.

Die Slides zu dem von mir gehaltenen Talk/Workshop gibts nun hier für alle, die es interessiert

Solving Other People’s Problems

Andy — Mon, 31 Jan 2011 07:57:20 +0000

I needed a virtualized Mac OS. I want to know how a special EFI extension works which permanently changes the harddisk. I don’t want to do this on my real machine. I would have to reboot it several times and perhaps loose all my data on my HDD if something goes wrong while I’m experimenting which this extension. Also live debugging would be next to impossible.

Luckily, VMware and VirtualBox are able to virtualize Apple’s Mac OS X Server. His holy Steveness decided to allow virtualization only for the Server version of Mac OS. But there is help, so I “fixed” the DVD to be able to boot a Mac OS X 10.6.3 retail DVD in VMware.

Unfortunately VMware then told me that the CPU was halted by the guest operating system. This happens if a kernel issues a “hlt” statement on all currently active cores without interrupts enabled to wake up the CPU later. This is bad because the whole operating system is trapped in this state. But why did this happen with OS X?

The most usual causes for this are a) a programming error or b) a panic issued by some code in the kernel. A programming error can be excluded here, so the kernel must panic in a very early state. I googled the error message I got (“The CPU has been disabled by the guest operating system.”) and found this. Great. So the kernel seems to have problems with newer CPUs and I do have an i5 in my machine.

First I tried to circumvent this issue by several configuration changes in the VM configuration file and using different EFI Loaders but all those attempts failed, because it wasn’t some issue with the loaders. It was the kernel itself which panicked.

So I googled for a way to attach a debugger to the VM and I found one. VMware itself has a GDB Stub (see “GDB Server”). I configured the VM, ran it, attached the GDB to the VM, continued to execution and let the kernel panic. After that I hit CTRL+C in the GDB and the VM is halted in a state where I can modify it with the GDB completely:

andy@geekbook ~/Desktop % gdb mach_kernel
GNU gdb 6.3.50-20050815 (Apple version gdb-1472) (Wed Jul 21 10:53:12 UTC 2010)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...

> target remote localhost:8864
[New thread 1]
0x000000003fc40786 in ?? ()

> c
^C
Program received signal SIGINT, Interrupt.
0xffffff800022ddcf in Debugger ()

>

But what to do now? Well, we want to find out where the kernel panics and do something against it.
The VM already has panicked, so we can have a look at a stacktrace to determine which function caused the panic:

> bt
#0  0xffffff800022ddcf in Debugger ()
#1  0xffffff8000204ac3 in panic ()
#2  0xffffff80002cef74 in kernel_trap ()
#3  0xffffff80002e0e9a in return_from_trap ()
#4  0x0000000000002206 in ?? ()
#5  0xffffff80002d6bb1 in cnputc ()
#6  0xffffff800027cf30 in consdebug_putc ()
#7  0xffffff8000206577 in __doprnt ()
#8  0xffffff80002073b8 in kdb_printf ()
#9  0xffffff8000204b2b in panic ()
#10 0xffffff8000228834 in cpuid_set_info ()
#11 0xffffff80002c1c00 in cpuid_extfeatures ()
#12 0xffffff80002c4178 in vstart ()

What can we see here? This is the chain of functions the kernel called until it got into the state where it is now. Especially #9 is interesting. This is the panic() function which apparently lets the kernel panic, hence the name. It is called by #10 at address 0xffffff8000228834 which is the function cpuid_set_info.

This makes totally sense. It seems that the kernel tries to detect the CPU very early. Intel (and compatible) CPUs have a special assembler instruction to retrieve information about the CPU the code is running on. Now guess its name: “cpuid”. It seems that the kernel pedantically checks which CPU it is running on and if it is unknown, it panics.

Let’s fix this behaviour. What do we need for this? We need a good disassembler and our 1337 x86 Assembler skills.

We already got the first one. It’s the GDB:

> disassemble cpuid_set_info
Dump of assembler code for function cpuid_set_info:
0xffffff800022829e :	push   rbp
0xffffff800022829f :	mov    rbp,rsp
0xffffff80002282a2 :	push   r15
0xffffff80002282a4 :	push   r14
0xffffff80002282a6 :	push   r13
0xffffff80002282a8 :	push   r12
0xffffff80002282aa :	push   rbx
0xffffff80002282ab :	sub    rsp,0xa8
0xffffff80002282b2 :	mov    esi,0x198
0xffffff80002282b7 :	lea    rdi,[rip+0x49e6c2]        # 0xffffff80006c6980
0xffffff80002282be :	call   0xffffff80002c17f0 
0xffffff80002282c3 :	xor    eax,eax
0xffffff80002282c5 :	cpuid
0xffffff80002282c7 :	mov    DWORD PTR [rbp-0x40],eax
.......
0xffffff8000228ba1 :	leave
0xffffff8000228ba2 :	ret
0xffffff8000228ba3 :	mov    eax,0x6b5a4cd2
0xffffff8000228ba8 :	mov    DWORD PTR [rip+0x49df4e],eax        # 0xffffff80006c6afc
0xffffff8000228bae :	jmp    0xffffff8000228834 
End of assembler dump.

>

I shortened the dump, because it is too long and I will pick out the interesting things by hand and post them separately.

Puh, that’s much code. Where do we start? We have the address of the call to the panic function. The one from the stacktrace. Let’s have a look there:

0xffffff80002287d5 :	add    BYTE PTR [rax],al
0xffffff80002287d7 :	add    BYTE PTR [rax-0x50000000],dh
0xffffff80002287dd :	add    BYTE PTR [rax],al
0xffffff80002287df :	add    BYTE PTR [rax-0x50000000],dh
0xffffff80002287e5 :	add    BYTE PTR [rax],al
0xffffff80002287e7 :	add    BYTE PTR [rax-0x50000000],dh
0xffffff80002287ed :	add    BYTE PTR [rax],al
0xffffff80002287ef :	add    BYTE PTR [rdi],dh
0xffffff80002287f1 :	add    al,0x0
0xffffff80002287f3 :	add    BYTE PTR [rax-0x55ccc6d5],bh
0xffffff80002287f9 :	jmp    0xffffff8000228ba8 
0xffffff80002287fe :	mov    eax,0x73d67300
0xffffff8000228803 :	jmp    0xffffff8000228ba8 
0xffffff8000228808 :	mov    eax,0x426f69ef
0xffffff800022880d :	jmp    0xffffff8000228ba8 
0xffffff8000228812 :	mov    eax,0x78ea4fbc
0xffffff8000228817 :	jmp    0xffffff8000228ba8 
0xffffff800022881c :	mov    DWORD PTR [rip+0x49e2d6],0x0        # 0xffffff80006c6afc
0xffffff8000228826 :	lea    rdi,[rip+0x348073]        # 0xffffff80005708a0
0xffffff800022882d :	xor    eax,eax
0xffffff800022882f :	call   0xffffff8000204939

The last instruction is indeed a call to “panic”. But why is it called? There is no conditional branch instruction which decides whether to panic or not, but some weird jumps and obove them obviously incorrect code. Not necessarily incorrect, but semantically these instructions don’t make sense. We’ll leave them for now. Let’s see where other code could jump to the code that actually calls “panic”. A few instructions above, we find this:

0xffffff800022871f :	lea    rsi,[rip+0x49e25a]        # 0xffffff80006c6980
0xffffff8000228726 :	lea    rdi,[rip+0x348163]        # 0xffffff8000570890
0xffffff800022872d :	call   0xffffff8000224574 
0xffffff8000228732 :	test   eax,eax
0xffffff8000228734 :	jne    0xffffff8000228826 
0xffffff800022873a :	cmp    BYTE PTR [rip+0x49e28b],0x6        # 0xffffff80006c69cc
0xffffff8000228741 :	jne    0xffffff800022881c 
0xffffff8000228747 :	movzx  eax,BYTE PTR [rip+0x49e27f]        # 0xffffff80006c69cd
0xffffff800022874e :	sub    eax,0xd
0xffffff8000228751 :	cmp    al,0x21
0xffffff8000228753 :	ja     0xffffff800022881c 
0xffffff8000228759 :	movzx  eax,al
0xffffff800022875c :	lea    rdx,[rip+0x9]        # 0xffffff800022876c 
0xffffff8000228763 :	movsxd rax,DWORD PTR [rdx+rax*4]
0xffffff8000228767 :	add    rax,rdx
0xffffff800022876a :	jmp    rax

What does this code do? At the beginning it seems to load 2 values relative to the instruction pointer into registers rsi and rdi. Then it calls “strncmp”. strncmp usually takes two pointers to strings. So let’s have a look where these pointers actually point to with the GDB:

> print (char*)0xffffff80006c6980
$1 = 0xffffff80006c6980 "GenuineIntel"

> print (char*)0xffffff8000570890
$2 = 0xffffff8000570890 "GenuineIntel"

Both strings are the same. One string seems to be a constant and the other one is retrieved by a “cpuid” instruction. We could look further into this but it’s not interesting to us.
These strings are the same. strncmp will return 0 and the “jne” instruction after the “test” will not branch. This is not the place where we are panicking.
The next instruction then loads some value into the eax register and then some other value is subtracted and compared with 0×21. The following “ja” instruction (jump if above/greater than for unsigned values) jumps to “panic”. Perhaps this is where we panic? But I’m too lazy to read all this assembly and figure out what those values actually mean.
The solution is easy: I restarted the VM, set a breakpoint to the beginning of the code and single stepped until I hit the jump to the panic-calling code. It wasn’t the first compare and it also wasn’t the second one. It was this last “jmp rax”.

Huh? What’s happening here? Actually it’s quite easy. Let’s do it step by step.

The “lea rdx,[rip+0x9]” loads the address of the current instruction pointer + 9 into register rdx. GDB is nice to us and shows the resulting address as a comment next to the instruction: 0xffffff800022876c.
The following “movsxd rax,DWORD PTR [rdx+rax*4]” instruction looks weird. It loads a 32 bit value (DWORD) at address rdx+rax*4 into the register rax. The address in rax seems to be some kind of a table. A value in rax then is the offset inside this table. The *4 is done because one entry inside the table is 4 bytes long and the x86 addresses are bytewise.
After that rdx itself is added to rax and then a jmp occurs. And this jump really landed in the panic code. But why? And what’s this table address that is loaded into rdx at the beginning?

Remember the code I mentioned above which didn’t make sense? That’s where the address points to. This is just no code. It’s a table full of offsets. It looks like this if the don’t try to disassemble it but display these values as 4 byte unsigned values:

dd 88h
dd 92h
dd 9Ch
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0A6h
dd 0B0h
dd 0B0h
dd 437h
dd 0B0h
dd 0B0h
dd 0B0h
dd 437h
dd 437h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 0B0h
dd 437h

What value of this table was actually used? I again reset the VM, set a proper breakpoint and looked at the registers (GDB command “info reg”) while the address to jump to was calculated. It turned out that on my Core i5 machine offset 0×18 was used, which is a value of 0x0B0.

Adding this value to the beginning of the table (0xffffff800022876c + 0xB0 = 0xffffff800022881C) leads us directly to the code which calls panic. Cool. At least we now know what exactly happens. How do we fix this?

We just patch this table. We need to find the right value for our CPU or perhaps jump to a completely own stub which does some initialization work. This is what in the 10.6.6 kernel happens. The offsets are different because the linker put the symbols on different addresses but they also added one completely different and new value to the table for all the new CPUs.

I just tried value 0×437 and it works like a charm. I booted the vm again, set a breakpoint at “cpuid_set_info” and patched the table with a simple “set *0xFFFFFF80002287CC=0×00000437″ and continued the execution. The kernel boots, I installed the machine and now I need to do the same shit again to get the kernel on the HDD to boot. After that I can update the OS to 10.6.6 what should solve the issue.

But why the fuck did I do all this? The problem is that I needed the VM now. And not when I got a proper CD which runs on i5/i7 processors. I also didn’t have another Mac here to setup and update the VM.

It also seems that all retail DVDs are an older build which does not run on i5 and i7 CPUs. Well done, Apple… Hence the name of this article: “Fixing Other People’s Problems”. It should just not be my problem to patch a kernel to install an operating system in a virtualized environment. And all this, just because Apple decided to just “support” a hand full of CPUs.

Update: I just had a look at the XNU source code and found the c code I patched. Now it should be obvious why there was a branch table.

void
cpuid_set_info(void)
{
	i386_cpu_info_t		*info_p = &cpuid_cpu_info;

	bzero((void *)info_p, sizeof(cpuid_cpu_info));

	cpuid_set_generic_info(info_p);

	/* verify we are running on a supported CPU */
	if ((strncmp(CPUID_VID_INTEL, info_p->cpuid_vendor,
		     min(strlen(CPUID_STRING_UNKNOWN) + 1,
			 sizeof(info_p->cpuid_vendor)))) ||
	   (cpuid_set_cpufamily(info_p) == CPUFAMILY_UNKNOWN))
		panic("Unsupported CPU");

	info_p->cpuid_cpu_type = CPU_TYPE_X86;
	info_p->cpuid_cpu_subtype = CPU_SUBTYPE_X86_ARCH1;

	cpuid_set_cache_info(&cpuid_cpu_info);

	/*
	 * Find the number of enabled cores and threads
	 * (which determines whether SMT/Hyperthreading is active).
	 */
	switch (info_p->cpuid_cpufamily) {
	/*
	 * This should be the same as Nehalem but an A0 silicon bug returns
	 * invalid data in the top 12 bits. Hence, we use only bits [19..16]
	 * rather than [31..16] for core count - which actually can't exceed 8.
	 */
	case CPUFAMILY_INTEL_WESTMERE: {
		uint64_t msr = rdmsr64(MSR_CORE_THREAD_COUNT);
		info_p->core_count   = bitfield32((uint32_t)msr, 19, 16);
		info_p->thread_count = bitfield32((uint32_t)msr, 15,  0);
		break;
		}
	case CPUFAMILY_INTEL_NEHALEM: {
		uint64_t msr = rdmsr64(MSR_CORE_THREAD_COUNT);
		info_p->core_count   = bitfield32((uint32_t)msr, 31, 16);
		info_p->thread_count = bitfield32((uint32_t)msr, 15,  0);
		break;
		}
	}
	if (info_p->core_count == 0) {
		info_p->core_count   = info_p->cpuid_cores_per_package;
		info_p->thread_count = info_p->cpuid_logical_per_package;
	}

	cpuid_cpu_info.cpuid_model_string = ""; /* deprecated */
}

static uint32_t
cpuid_set_cpufamily(i386_cpu_info_t *info_p)
{
	uint32_t cpufamily = CPUFAMILY_UNKNOWN;

	switch (info_p->cpuid_family) {
	case 6:
		switch (info_p->cpuid_model) {
		case 13:
			cpufamily = CPUFAMILY_INTEL_6_13;
			break;
		case 14:
			cpufamily = CPUFAMILY_INTEL_YONAH;
			break;
		case 15:
			cpufamily = CPUFAMILY_INTEL_MEROM;
			break;
		case 23:
			cpufamily = CPUFAMILY_INTEL_PENRYN;
			break;
		case CPUID_MODEL_NEHALEM:
		case CPUID_MODEL_FIELDS:
		case CPUID_MODEL_DALES:
		case CPUID_MODEL_NEHALEM_EX:
			cpufamily = CPUFAMILY_INTEL_NEHALEM;
			break;
		case CPUID_MODEL_DALES_32NM:
		case CPUID_MODEL_WESTMERE:
		case CPUID_MODEL_WESTMERE_EX:
			cpufamily = CPUFAMILY_INTEL_WESTMERE;
			break;
		}
		break;
	}

	info_p->cpuid_cpufamily = cpufamily;
	return cpufamily;
}

At the beginning of “cpuid_set_info” we can see the strncmp of the vendor name followed by a call to “cpuid_set_cpufamily” where then the type of CPU is determined. The switch/case is the reason for the branch table.

OS Development Demo Code

Andy — Mon, 12 Jul 2010 08:36:14 +0000

A while ago, I wrote some demo code to demonstrate various things you need to do to write your own operating system.
In May 2010 I held a lecture and workshop about this code and the concepts at SigInt conference in Cologne, Germany.

I just thought that this code may be interesting to other people who did not attend this conference.

The demo code includes mainly x86 specific stuff and it is truly demo code, so don’t just copy it into another project. Even if it works at first sight, it it is highly likely to break sooner or later

The code contains samples for:

A small “Hello World” bootsector in realmode
A sample how to enable the A20 gate. This works at least in QEMU
Loading a barebone-kernel written in C by GRUB
Switching to protected mode and writing to the textmode framebuffer
Handling interrupts in protected mode using the old PIC
Configuring and using the old PIT (Timer)
Switching to longmode (64 Bit)
A multitasking demo how to do context switching

The first two samples are completely written in Assembly and are loaded directly by the BIOS from a floppy drive. All other samples are mainly written in C spiced with some Assembly and are loaded from a FAT formatted floppy disk by a GRUB bootloader.

For those of you who speak german, here are the slides of the conference:
Talk
Workshop

If you do anything with this demo code, I would appreciate to hear about your project or experiences. Also, If there are specific questions or problems about/with this code which are not easily resolvable by Googling and using your brain, feel free to ask in the comments.

Switch Hacking

Andy — Tue, 02 Feb 2010 12:18:33 +0000

Last week I stumbled upon tmbinc’s blog entry series “What’s inside”.
This one was especially interesting to me since I always wanted to buy a GBit switch (I never had one until today, really ).

I ordered this switch (a TP-Link TL-SG1008D) for just about 30€ including shipping and today I got my package. I opened it, checked if the switch works and voided the warranty.
After opening the device it was clear that I got some other hardware compared to tmbinc. I don’t have this monster heat sink on top of a BGA chip. Instead there are two smaller LQFP-Chips with one heat sink each.

A closer look at my and tmbinc’s images relieve that my switch is actually revision 1.1 and tmbinc’s is revision 2.0.

I tried to remove the heat sinks with pliers but I didn’t want to damage my new toy.
Also, I wanted to know how warm these chips get while using the device, so I took some ethernet cables, made some loops and inserted one cable into the ethernet jack into my laptop. As soon as Mac OS detected an active connection it broadcasted at least one package onto the line that was looping happily from one switch port to the next
I let this thing running for about 10 minutes and the chips didn’t get very warm. I hoped that the glue would get a bit less sticky due to the heat, but there was just no heat. Seems that the heat sinks did a good job there. So I tried the exact opposite and put this thing into my freezer for a few minutes. After it got cooled down a few degrees I took my pliers and applied some force and look: They came off.

As you can see there are two Realtek chips on this board. The right chip is an RTL8214. The Realtek Website says that this chip is just a quad-port gigabit ethernet transceiver.
The other chip is an RTL8368S. Here the Realtek Website tells us, that this chip is actually the switch controller for 8 ports, but it has only 4 transceivers, hence the second transceiver chip. On the pictures you can clearly see some bus between the two ICs.
And look at the features: Per port-ACLs on Layer 2, 3 and 4, VLAN, Spanning Tree, 802.1x, QoS and several statistic counters.
Unfortunately the datasheets for both chips don’t seem to be publicly available.
I absolutely hate all this NDA crap… Those fscking ethernet chips are no rocket science that needs to be protected in such a way. SRSLY!

On the upper side of the image above you can see a small IC with part number 24C08, which is an 8KBit I2C EEPROM. I haven’t looked at the contents yet, but I suspect that this EEPROM contains the configuration for the switch controller, but I doubt that the contents will be of any help without a datasheet

Some closeups of the chips:

Update: I found some datasheets for several Realtek Chips. There is no datasheet for the RTL8368, but for the RTL8369. It seems that the only difference between those two chips are that the latter doesn’t have any transceivers on-die, but only some SGMII ports for all 8 Ethernet ports. I think I’m going to dump the EEPROM and try to get some information out of it with the 8369 datasheet.