|
Halaman 4 dari 4
Physical Address Extension - PAE Memory and Windows
Updated: February 9, 2005
On This Page
Introduction
PAE
is an Intel-provided memory address extension that enables support of
greater than 4 GB of physical memory for most 32-bit (IA-32) Intel
Pentium Pro and later platforms. This article provides information to
help device driver developers implement Windows drivers that support
PAE.
Microsoft supports Physical Address Extension (PAE) memory
in Microsoft Windows 2000, Windows XP, and Windows Server 2003
products:
|
Windows 2000 Advanced Server
|
8 GB of physical RAM
|
|
Windows 2000 Datacenter Server
|
32 GB of physical RAM
|
|
Windows XP (all versions)
|
4 GB of physical RAM*
|
|
Windows Server 2003 (and SP1), Standard Edition
|
4 GB of physical RAM*
|
|
Windows Server 2003, Enterprise Edition
|
32 GB of physical RAM
|
|
Windows Server 2003, Datacenter Edition
|
64 GB of physical RAM
|
|
Windows Server 2003 SP1, Enterprise Edition
|
64 GB of physical RAM
|
|
Windows Server 2003 SP1, Datacenter Edition
|
128 GB of physical RAM
|
* Total physical address space is limited to 4 GB on these versions of Windows.
PAE
is supported only on 32-bit versions of the Windows operating system.
64-bit versions of Windows do not support PAE. For information about
device driver and system requirements for 64-bit versions of Windows,
see 64-bit System Design.
Although
support for PAE memory is typically associated with support for more
than 4 GB of RAM, PAE can be enabled on Windows XP SP2, Windows Server
2003, and later 32-bit versions of Windows to support hardware enforced
Data Execution Prevention (DEP).
Operating System Support. The PAE kernel is not enabled by default for systems that can support more than 4 GB of RAM.
To boot the system and utilize PAE memory, the /PAE
switch must be added to the corresponding entry in the Boot.ini file.
If a problem should arise, Safe Mode may be used, which causes the
system to boot using the normal kernel (support for only 4 GB of RAM)
even if the /PAE switch is part of the Boot.ini file.
The
PAE mode kernel requires an Intel Architecture processor, Pentium Pro
or later, more than 4 GB of RAM, and Windows 2000, Windows XP, or
Windows Server 2003.
The PAE kernel can be enabled automatically without the /PAE switch present in the boot entry if the system has DEP enabled (/NOEXECUTE switch is present) or the system processor supports hardware-enforced DEP. Presence of the /NOEXECUTE switch on a system with a processor that supports hardware-enforced DEP implies the /PAE switch. If the system processor is capable of hardware-enforced DEP and the /NOEXECUTE switch is not present in the boot entry, Windows assumes /NOEXECUTE=optin by default and enables PAE mode. For more information, see the topic "Boot Options in a Boot.ini File" in the Windows DDK.
System Board Issues: DAC Capabilities for Buses
Various
chipsets are capable of supporting more than 4 GB of physical memory.
By using PAE, the Windows Datacenter and Advanced Server operating
systems can use this memory.
On a 64-bit platform, for optimal
performance, all PCI adapters (including 32-bit PCI adapters) must be
able to address the full physical address space. For 32-bit PCI
adapters, this means that they must be able to support the Dual Address
Cycle (DAC) command to permit them to transfer 64-bit addresses to the
adapter or device (that is, addresses above the 4 GB address space).
Adapters that cannot provide this support cannot directly access the
full address space on a 64-bit platform.
Unfortunately, Microsoft
is finding that not all PCI buses on a system board support DAC, which
is required for a 32-bit PCI adapter to address more than 4 GB of
memory. Furthermore, there is no way for a DAC-capable PCI device (or
its associated driver) to know that it is running on a non-DAC-capable
bus.
Given these issues in hardware, Microsoft must find the
optimal software workaround in the operating system from the standpoint
of customers, OEMs, and Microsoft. This section discusses several
possible software solutions that have been rejected due to various
inadequacies, and then discusses the selected solution.
Overriding Dma64BitAddress = Performance and Stability Problems
One software workaround would require the operating system to override the Dma64BitAddresses flag passed by the driver to HalGetAdapter:
if the driver passes TRUE and its device is on a bus that does not
support DAC, force this to FALSE. That causes the hardware abstraction
layer (HAL) to double-buffer transfers done through IoMapTransfer or GetScatterGatherList,
so the device never sees an address above 4 GB. For more information,
see the "Double-Buffer DMA Transfer" topic in the Windows DDK.
Unfortunately, HalGetAdapter
does not have the necessary information to determine the bus of the
caller's device. All that can be known is the contents of the
DEVICE_DESCRIPTION structure that the driver provides, where the only
relevant information is that the InterfaceType is PCI.
===================================================
typedef struct _DEVICE_DESCRIPTION {
ULONG Version;
BOOLEAN Master;
BOOLEAN ScatterGather;
BOOLEAN DemandMode;
BOOLEAN AutoInitialize;
BOOLEAN Dma32BitAddresses;
BOOLEAN IgnoreCount;
BOOLEAN Reserved1; // must be false
BOOLEAN Dma64BitAddresses;
ULONG BusNumber; // unused for WDM
ULONG DmaChannel;
INTERFACE_TYPE InterfaceType;
DMA_WIDTH DmaWidth;
DMA_SPEED DmaSpeed;
ULONG MaximumLength;
ULONG DmaPort;
} DEVICE_DESCRIPTION, *PDEVICE_DESCRIPTION;
=====================================================
In addition:
| • |
Double-buffering
has been shown in testing at Microsoft to have a negative performance
impact on I/O throughput and CPU utilization. This negative impact
increases as more memory is added beyond 4 GB.
|
| • |
The
delays associated with high-performance I/O and double-buffering might
cause timing issues for drivers and devices, which would negatively
impact system stability.
|
All of this is contrary to the goals of Windows Datacenter and Advanced Server to ensure increased scalability and reliability.
IoGetDmaAdapter = Not Used by All Drivers
Windows Driver Model (WDM) introduced the call, IoGetDmaAdapter(), which is similar to HalGetAdapter,
but also takes a pointer to the physical device object (PDO). This
allows the operating system to detect the caller's PCI bus and whether
it is a child device of a non-DAC bus. Then the PCI driver could
override the Dma64BitAddress field in DEVICE_DESCRIPTION so that the
HAL thinks the device can handle only 32-bit addresses.
The problem with this approach is that not all drivers use IoGetDmaAdapter. Many still use HalGetAdapter,
even though the recent DDKs specifically define this as an obsolete
call. Microsoft has no way of preventing third-party drivers from
calling HalGetAdapter; forcing drivers to use IoGetDmaAdapter by failing the call would render many otherwise capable drivers as no longer functional. Requiring all drivers to use IoGetDmaAdapter would create enormous test and performance issues.
Incorrect or No DMA Routines = No Possible Workaround
Regardless whether IoGetDmaAdapter or HalGetAdapter
is used, not all drivers use the DMA routines correctly. Some do not
use them at all because of the performance impact. It would be no
surprise to find 64-bit capable drivers that ignore the DMA routines
because they "know" they do not need such routines. In such cases,
there is no possible operating system workaround--all offending drivers
would have to be found and fixed.
Boot Device on Non-DAC Bus / All Non-DAC Buses = No Large Memory Support
Besides the many problems of non-DAC buses described above, there are two special cases:
| • |
Boot Device on Non-DAC Bus.
The first is the case where the boot device is on a non-DAC bus. Given
that a pagefile usually resides on the boot device and this is a
primary data path, then all pagefile I/O would be forced to be double
buffered, negatively impacting system performance and possibly leading
to system instability.
|
| • |
All Non-DAC Buses.
The second case is where all buses are non-DAC, in which case the user
has no option of moving DAC adapters, LME-capable adapters, or both to
DAC buses. The only solution in such a case is to limit memory support
to 4 GB, regardless of whether the processor, memory controller, or
system board physically support more than 4 GB of RAM.
|
Microsoft
does not expect any instances of the second case and few of the first,
but must take these possibilities into account in defining the overall
solution.
Selected Solution: Disable Memory Above 4 GB when Non-DAC Buses Exist
Because Microsoft has no reliable software workaround for this problem,
the only viable alternative is to disable configurations that do not
work and inform the administrator of the problem. Disabling all memory
above 4 GB if there are any non-DAC buses is one way to prevent
instability in this case. Microsoft thinks that this is the best
solution for customers, because it is the least likely to destabilize
the platform.
The following table summarizes this information.
|
DAC
|
DAC
|
Highest performance and stability.
|
|
DAC
|
Non-DAC
|
Double buffering required.
1 2
|
|
Non-DAC
|
DAC
|
Memory corruption possible due to non-compliance with PCI standard; Windows takes specific actions to avoid.3
|
|
Non-DAC
|
Non-DAC
|
Double buffering required.4
|
If
you, the IxV, have and test user-mode code [and this is almost
universally true] this is a test scenario you must cover. You must make
sure that the code you are testing can deal correctly with high virtual
addresses, especially above 2 GB. Windows should be tested with your
applications or utilities to ensure they work.
Usually,
VirtualAlloc returns virtual addresses in low -> high order. So,
unless your process allocates a lot of memory or it has a very
fragmented virtual address space, it will never get back very high
addresses. This is possibly hiding bugs related to high addresses.
There is a simple way to force allocations in top -> down order in
Windows Server 2003, Datacenter Edition and Enterprise Edition
operating systems and this can reveal important bugs.
You need to
set HKLM\System\CurrentControlSet\Control\Session Manager\Memory
Management\AllocationPreference REG_DWORD = 0x100000
All
applications, but especially those applications that are built with
LINKER_FLAGS=/LARGEADDRESSAWARE in the sources file, should be tested
with the /PAE, /NOLOWMEM and /3GB switches, and the registry change.
No
special hardware is required for MEM_TOP_DOWN testing. Any machine
running Windows Server 2003, Datacenter Edition or Enterprise Edition
operating systems supports this testing.
MEM_TOP_DOWN can also be used on Itanium-based systems. However, /3GB is an x86-specific feature.
The
/PAE, /NOLOWMEM and /3GB switches, and the registry change, can be used
at once. Note that /3GB will prevent access to physical memory beyond
16 GB because the kernel memory space is reduced with the /3GB switch,
and thus does not have enough room for the additional Page Table
Entries required when memory is larger than 16 GB.
Adapter and Driver Issues: LME and DAC Capable
All
physical memory is treated as general-purpose memory, so no new APIs
are needed to access I/O above the 4 GB physical memory address. Also,
direct I/O can be done for greater than 4 GB physical addresses--this
requires DAC-capable or 64-bit PCI devices. Devices and drivers that
can perform direct I/O beyond 4 GB are considered Large Memory Enabled
(LME).
Because Windows does not have a kernel PAE or LME API or
interface, the PAE-X86 kernel ensures that many items are identical to
the standard kernel, including:
| • |
Kernel memory space organization is unchanged.
|
| • |
PCI Base Address Registers [BAR] remains the same.
|
| • |
Registry flags work the same.
|
| • |
Non-paged pool size remains the same.
|
| • |
3GT feature is supported for up to 16 GB RAM.
|
| • |
IMAGE_FILE_LARGE_ADDRESS_AWARE continues to work.
|
| • |
"Well known" kernel addresses remain in the same locations.
|
However,
careful device driver development is still required. Hardware devices
should be DAC-capable or 64-bit capable with LME drivers; otherwise,
the device will function as "legacy" 32-bit and will be double
buffered, with lower relative performance.
Although
double-buffering can have a relatively small impact (single percentage
points) on 8 GB systems, this is enough to impact I/O intensive tasks
such as database activity. This is also dependent on a number of
factors beyond Microsoft's control, such as hardware design and device
driver optimizations like interrupt moderation and efficient use of the
PCI bus. As the amount of physical memory increases, so does the
negative performance impact in comparison to DAC/64-bit devices and LME
drivers.
General Guidelines for LME Drivers
The following guidelines aid in preventing LME driver failures:
| • |
Do
not use PVOID64. Using PVOID64 anywhere will return incorrect
information, because this call does not return valid information on the
Intel Architecture platform. Instead, use PHYSICAL_ADDRESS.
Note:
This does not apply for NDIS miniports. Also, miniports, such as USB
and others that are relatively low performance, need not be rewritten
to be LME, because the performance gain or loss is not significant.
These miniports, however, should correctly use the kernel interfaces
for Windows and not try to trick the operating system by use of
undocumented and unsupported shortcuts.
|
| • |
Do not call MmGetPhysicalAddress()
on a locked buffer, discard the high 32 bits, and then program the
adapter to DMA into the resultant address. This will certainly result
in corrupted memory, lost I/O, and system failure. If this call is
made, ensure that all address information returned is used and that the
driver correctly operates with that 64-bit address.
|
| • |
Do
not use PVOID when manipulating physical addresses. Because PVOID is
only 32 bits, address truncation will take place and memory corruption
will result.
|
| • |
Do
not use ULONG when manipulating physical addresses, because this has
exactly the same precautions and behavior as PVOID: system failure.
|
| • |
Do
not indicate support for scatter/gather in the DEVICE_DESCRIPTION when
not true in an attempt to avoid the buffering provided by HAL (the
"mapping registers").
|
| • |
If the driver cannot support 64-bit addresses, do not call IoMapTransfer(...) without having an AdapterControl(...) function (again, to avoid mapping registers), and do not supply zero as the value for MapRegisterBase. This will fail.
|
Other functions and calls might cause failures. Information is provided in the Windows DDK.
Guidelines for NDIS Miniports and SCSI Miniports
Guidelines for NDIS Miniports on PAE Systems
Network adapters that are 64-bit address capable should:
| • |
Use the NDIS scatter/gather DMA model.
Miniports need to call NdisMInitializeScatterGatherDma with Dma64Addresses = TRUE.
|
| • |
Use the NDIS deserialized miniport driver model.
Miniports should be deserialized for optimum performance on Windows Datacenter and Advanced Server operating systems.
|
See the entry for NdisMInitializeScatterGatherDma in the DDK documentation.
General Guidelines
The following should be noted for NDIS miniports:
| • |
Shared memory allocated using NdisMAllocateSharedMemory is guaranteed not to cross a 4 GB boundary.
|
| • |
NDIS_PER_PACKET_INFO_FROM_PACKET(ScatterGatherListPacketInfo) will never return NULL for miniports that support scatter/gather DMA.
|
| • |
The
physical address range indicated by SCATTER_GATHER_ELEMENT will not
cross a 4-GB boundary. If a virtual memory buffer does cross a 4-GB
boundary, it will be broken into two scatter/gather elements.
|
Guidelines for 32-Bit Address-Only Network Devices
The following guidelines are recommended for 32-bit address-only network devices:
| • |
Properly
written NDIS drivers will work as-is on PAE systems, but will have a
significant negative performance impact that grows as the amount of
installed RAM increases.
|
| • |
NdisMStartBufferPhysicalMapping will copy all fragments above the 4 GB address space to memory that is below the 4 GB mark.
|
Guidelines for 64-bit Address-Capable SCSI Miniports
(including all related adapters for SCSI 2)
The following guidelines are recommended for 64-bit address-capable SCSI miniports:
| • |
Miniports need to support scatter-gather DMA. They must not call any of the slave-mode DMA routines: ScsiPortFlushDma or ScsiPortIoMapTransfer.
|
| • |
Miniports should check the value of Dma64BitAddresses
in PORT_CONFIGURATION_INFORMATION to determine whether 64-bit physical
addresses are supported. If 64-bit physical addresses are supported,
the miniport should change its extension sizes to account for the
larger physical addresses (if necessary) and set the Dma64BitAddresses field to SCSI_DMA64_MINIPORT_SUPPORTED before calling ScsiPortGetUncachedExtension.
|
| • |
Miniports must not attempt to access data buffers using virtual addresses unless they have set the MapBuffers
bit in the PORT_CONFIGURATION_INFORMATION structure. The exceptions to
this rule are INQUIRY and REQUEST_SENSE operations that will always
have a valid virtual address.
|
| • |
Use SCSI_PHYSICAL_ADDRESS to access all physical addresses.
|
| • |
Uncached extensions and SRB extensions will not cross the 4 GB boundary.
|
| • |
No scatter/gather element will cross the 4 GB boundary.
|
Guidelines for Legacy SCSI Miniports
The following guidelines are recommended for legacy SCSI miniports:
| • |
Miniports need to support scatter/gather DMA. They must not call any of the slave-mode DMA routines: ScsiPortFlushDma or ScsiPortIoMapTransfer.
|
| • |
Miniports must not attempt to access data buffers using virtual addresses unless they have set the MapBuffers
bit in the PORT_CONFIGURATION_INFORMATION structure. The exceptions to
this rule are INQUIRY and REQUEST_SENSE operations that will always
have a valid virtual address.
|
| • |
Miniports
should not set the MapBuffers bit unless absolutely necessary, because
providing valid virtual addresses to a 32-bit driver on a LME system is
costly.
|
Creating and Testing LME Drivers
Use the following checkpoints to create and test LME drivers:
| • |
Do
not try to add more features into the driver when modifying it for a
Large Memory system. Instead, modify the driver as little as possible
to ensure LME capabilities.
|
| • |
Always use the most recent version of the Hardware Compatibility Test (HCT) provided at: http://www.microsoft.com/whdc/DevTools/HCTKit.mspx.
|
| • |
Test
both on systems with greater than 4 GB of physical memory and on
systems with less than 4 GB of system memory, using all the available
tools from Microsoft.
|
| • |
Test legacy drivers to ensure there is no corruption due to incorrect use of calls such as MmGetPhysicalAddress.
|
| • |
Test specifically for PAE systems. This requires a minimum installed RAM of 6 GB.
|
| • |
Test using the NOLOWMEM switch in the Boot.ini file:
| • |
Guarantees a 64-bit physical address above 4 GB.
|
| • |
Hides pages below 4 GB.
|
| • |
Fills pages below 4 GB with unique patterns.
Task Manager will display roughly total RAM minus ~4 GB.
|
|
| • |
Test using the /3GB switch simultaneously
| • |
Changes the kernel memory environment and exposes issues with drivers
|
| • |
Reduces the kernel memory space to 1 GB
|
|
| • |
Test using the MEM_TOP_DOWN registry setting
This forces all allocations for memory to be allocated from the top down, instead of the normal bottom up.
Set HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management\AllocationPreference REG_DWORD = 0x100000
|
| • |
Test using DevCtl, which is supplied with the HCT:
| • |
Deliberately attacks drivers through IOCTLs.
|
| • |
Tests exception handling and failure modes, and tests unexpected entry points into the driver.
|
| • |
Queries with buffers too small to contain returns.
|
| • |
Checks IOCTL/FSCTL for:
- Missing, small, or garbage buffers
- Data changing asynchronously
- Bad pointers
- Buffer mapping changing asynchronously.
|
| • |
Issues requests both synchronously and asynchronously to the device.
|
| • |
Tests IRP cancels, delays and nulls
|
| • |
Checks leak, pooltag, and lookaside information.
|
|
| • |
Test using Driver Verifier (provided with Windows):
| • |
All pool allocations are segregated to check for corruption.
|
| • |
Tests using extreme memory pressure--whenever IRQL or spinlock is acquired, try to page out the driver; catches fatal errors.
|
| • |
All
spinlock, IRQL and pool requests and releases are extensively error
checked: double releases of spinlocks, usage of uninitialized
variables, other forms of pool corruption, and so on.
|
|
Troubleshooting DAC Support and LME Drivers
The
following checkpoints can help OEMs, IHVs and customers determine
whether there are any issues relating to the system board, buses, or
adapters in supporting DAC and LME.
| • |
If
the driver fails at initialization, check with the system OEM to
determine whether all PCI buses present in the system support DAC.
|
| • |
If
a network adapter driver performs a bug check immediately upon a
network connection, determine whether all buses support DAC, again by
checking with the OEM.
|
| • |
If the PCI buses on the system are all DAC capable, check whether the hardware device is compliant with PCI 2.1.
|
| • |
If
the bus supports DAC and the device is PCI 2.1 compliant, check the
driver for assumptions being made about physical addresses.
|
Documentation
is provided in the Windows DDK about PAE memory support. A summary of
the information for customers is included in the following sections.
Hardware Requirements for PAE
The system must meet the following minimum requirements:
| • |
x86 Pentium Pro processor or later
|
| • |
More than 4 GB of RAM
|
| • |
450 NX or compatible chipset and support, or later
|
Enabling PAE
To enable PAE:
• Locate the Boot.ini file, which is typically in the root folder (for example, C:/) and remove its Read-Only and Hidden attributes.
• Open the Boot.ini file with a text editor, and then add the /PAE parameter to the ARC path, as shown in the following example:
=============================================================
multi(0)disk(0)rdisk(0)partition(2)
\WINNT="Windows ???? Datacenter Server" /PAE /basevideo /sos
=============================================================
• On the File menu, click Save.
• Restore the Read-Only attribute to the Boot.ini file.
Troubleshooting Specific Programs
Following are two examples of problems that might occur, with solutions that will rectify the problem.
Problem: The computer will not start after PAE is enabled.
Cause: Your hardware may not support PAE.
Solution: Start the system and run Safe Mode, which disables PAE. Then remove the /PAE parameter from the Boot.ini file.
To run Safe Mode:
|
1.
|
When you see the message "Please select the operating system to start," press F8.
|
|
2.
|
Use the arrow keys to highlight the appropriate Safe Mode option, and then press ENTER.
To use the arrow keys on the numeric keypad to select items, NUMLOCK must be off.
|
Problem: After PAE is enabled, the computer runs for a time and then displays a Stop error.
Cause: Your hardware may not support PAE.
Solution:
Contact your hardware vendor for a driver update. If your hardware or
driver is not capable of supporting PAE, disable PAE by removing the /PAE parameter in the Boot.ini file. If you must disable PAE but your system processor supports hardware-enforced DEP, add /NOPAE /NOEXECUTE=alwaysoff to your Boot.ini file. Note: This will disable the DEP feature on your computer.
Call to Action for LME and DAC Capable Cevices
| • |
Driver developers should follow the guidelines in this article to ensure that their devices are LME and DAC capable.
|
| • |
System
manufactures should ensure that only DAC-capable buses are included in
any system design intended to support Large Memory capabilities.
|
| • |
Requirements for Large Memory capabilities are defined in Microsoft Windows Logo Program System and Device Requirements.
|
 Top of page
|
Technical Update 
