Re: [MiNT] Virtual Memory

[Eero Tamminen <oak@helsinkinet.fi>]

Hi,

On lauantai 12 tammikuu 2013, Jo Even Skarstein wrote:
> As Adam has already explained, the big, big problem here is how to get
> this working properly with programs that use the AES and VDI. These
> absolutely requires that both the system (AES and VDI) and clients use
> the same memory space.

Because while the AES, VDI etc APIs send raw pointers, the current AESes
doesn't translate these address from the user-space process address-space
to kernel addresses, like MiNT GEMDOS (assumably) does.

So, either (Xa)AES needs to be updated to handle this translation,
or AES needs to use the same address space as the process.  If there
are multiple processes whose user-space memory buffers AES needs to
handle (e.g. to send messages), all these processes need same
address space as AES, it's not enough that current process and
AES have same address translation/mapping.


> Also, how does MiNT knows which page(s) can be
> swapped and which can't (e.g. interrupt- or traphandlers)?

All such handlers should be set on kernel side, so that it can handle
them properly.  On Linux the page structure has a bit for whether page
is swappable or not, MiNT should also have that.

Typically (all) memory used for DMA operations is something that needs to
be both non-swappable and continuous, latter because only most high end
(e.g. network) devices devices support scatter-gather.

(There could be a lot of (cached) pages used for file read/write operations
DMA transfer, and because these come & go and there can be a lot of them,
memory can get fragmented even on system with advanced virtual memory
managemnent.)

I'm not sure whether different flags would be needed on Atari machines
to mark whether page is swappable and whether it's currently (not)
movable, but does MiNT replace XBios functions and mark memory given
for DMA operations (DMAread()/DMAwrite()) as non-swappable?


Btw. On Linux kernel there are different functions for allocation memory
that is physically continuous (kmalloc()) and which isn't (vmalloc()).
Because vmalloc() is slower, in-kernel allocs that are smaller than
page size, are often done with kmalloc.


> It can be solved like Adam says by reserving a memory space (shared with
> the OS) for current applications, and put a new flag in the header for
> executables that are "VM-clean". But I worry about performance. Adapting
> the AES/VDI and clients to a VM model would require a whole new API.

Not a new API, but an AES that does the user-space <-> kernel translation
whenever addresses come from or go to client user space.

Another possibility is new GEMlib which would map all addresses given
to its bindings to system addresses.  Then the new API would be just
one for providing this mapping.

(User-space processes knowing system memory mappings is of course a security
issue, but I don't think our platform to be a target where one would need to
worry about this kind of attacks...)


> The RSC/OBJECT-tree would either have to be replaced with something new,
> or be accessed by the client through some API. Same with the shared
> AES/VDI structures. This will cost in terms of speed. I fear that AES/VDI
> applications would be restricted to "non-VM" also in the future.

If only command line applications i.e. ones using just MiNT APIs
that do the address translation at OS API interface, would be
demand paged & swapped, would that be of use?

Typical worst case memory hogs in a system are:
- Browser
- Complex document viewers (PDF) & editors (Word)
- Image manipulation programs with huge images
- Specific command line application that otherwise manipulate
  large data structures, e.g. g++ compiler can in some cases
  use GBs of RAM on Linux

On average usage, Browser is much worse than the others.

In the case of Browser, maybe it could have page content processing
and most of memory usage in a separate process that doesn't use AES/VDI.

Then there would be a separate GUI that keeps responsive because
content is rendered in a separate process, and doesn't need to use
so much (non-swappable) memory as it just shows shared memory buffer
from the renderer?


> Sven Karlsson implemented virtual memory in MiNT 1.14 (or maybe it was
> in an early 1.15? I really can't remember, it's 15 years ago or so) but
> it really didn't work that well. I suggest you study his implementation
> to see what went wrong.


	- Eero