Remembering More Memory: XMS and a Real Hack

Last time we talked about how the original PC has a limit of 640 kB for your programs and 1 MB in total. But of course those restrictions chafed. People demanded more memory, and there were workarounds to provide it.

However, the workarounds were made to primarily work with the old 8088 CPU. Expanded memory (EMS) swapped pages of memory into page frames that lived above the 640 kB line (but below 1 MB). The system would work with newer CPUs, but those newer CPUs could already address more memory. That led to new standards, workarounds, and even a classic hack.

XMS

If you had an 80286 or above, you might be better off using extended memory (XMS). This took advantage of the fact that the CPU could address more memory. You didn’t need a special board to load 4MB of RAM into an 80286-based PC. You just couldn’t get to with MSDOS. In particular, the memory above 1 MB was — in theory — inaccessible to real-mode programs like MSDOS.

Well, that’s not strictly true in two cases. One, you’ll see in a minute. The other case is because of the overlapping memory segments on an 8088, or in real mode on later processors. Address FFFF:000F was the top of the 1 MB range.

PCs with more than 20 bits of address space ran into problems since some programs “knew” that memory access above that would wrap around. That is FFFF:0010, on an 8088, is the same as 0000:0000. They would block A20, the 21st address bit, by default. However, you could turn that block off in software, although exactly how that worked varied by the type of motherboard — yet another complication.

XMS allowed MSDOS programs to allocate and free blocks of memory that were above the 1 MB line and map them into that special area above FFFF:0010, the so-called high memory area (HMA).

Because of its transient nature, XMS wasn’t very useful for code, but it was a way to store data. If you weren’t using it, you could load some TSRs into the HMA to prevent taking memory from MSDOS.

Protected Mode Hacks

There is another way to access memory above the 1 MB line: protected mode. In protected mode, you still have a segment and an offset, but the segment is just an index into a table that tells you where the segment is and how big it is. The offset is just an offset into the segment. So by setting up the segment table, you can access any memory you like. You can even set up a segment that starts at zero and is as big as all the memory you can have.

You can use segments like that in a lot of different ways, but many modern operating systems do set them up very simply. All segments start at address 0 and then go up to the top of user memory. Modern processors, 80386s and up, have a page table mechanism that lets you do many things that segments were meant to do in a more efficient way.

However, MS-DOS can’t deal with any of that directly. There were many schemes that would switch to protected mode to deal with upper memory using EMS or XMS and then switch back to real mode.

Unfortunately, switching back to real mode was expensive because, typically, you had to set a bit in non-volatile memory and reboot the computer! On boot, the BIOS would notice that you weren’t really rebooting and put you back where you were in real mode. Quite a kludge!

There was a better way to run MSDOS in protected mode called Virtual86 mode. However, that was complex to manage and required many instructions to run in an emulated mode, which wasn’t great for performance. It did, however, avoid the real mode switch penalty as you tried to access other memory.

Unreal Mode

In true hacker fashion, several of us figured out something that later became known as Unreal Mode. In the CPU documentation, they caution you that before switching to real mode, you need to set all the segment tables to reflect what a segment in real mode looks like. Obviously, you have to think, “What if I don’t?”

Well, if you don’t, then your segments can be as big as you like. Turns out, apparently, some people knew about this even though it was undocumented and perhaps under a non-disclosure agreement. [Michal Necasek] has a great history about the people who independently discovered it, or at least, the ones who talked about it publicly.

The method was doomed, though, because of Windows. Windows ran in protected mode and did its own messing with the segment registers. If you wanted to play with that, you needed a different scheme, but that’s another story.

Modern Times

These days, we don’t even use video cards with a paltry 1 MB or even 100 MB of memory! Your PC can adroitly handle tremendous amounts of memory. I’m writing this on a machine with 64 GB of physical memory. Even my smallest laptop has 8 GB and at least one of the bigger ones has more.

Then there’s virtual memory, and if you have solid state disk drives, that’s probably faster than the old PC’s memory, even though today it is considered slow.

Modern memory systems almost don’t resemble these old systems even though we abstract them to pretend they do. Your processor really runs out of cache memory. The memory system probably manages several levels of cache. It fills the cache from the actual RAM and fills that from the paging device. Each program can have a totally different view of physical memory with its own idea of what physical memory is at any given address. It is a lot to keep track of.

Times change. EMS, XMS, and Unreal mode seemed perfectly normal in their day. It makes you wonder what things we take for granted today will be considered backward and antiquated in the coming decades.