Re: Dynamic Linking

[Wolfgang Lux <lux@heidelbg.ibm.com>]

Evan Langlois wrote:
> 
> OK, I've downloaded the source to dld, the GNU dynamic linker.  Its
> a true dynamic linker - it caan load and link object and library files
> at run-time, and can also unlink and unload the same.  It does it at
> run-time, not load time.  Its a neat library.
> 
> It has already beeen ported to the Atari ST, however, it seems some
> special calls are needed to utilize it.  Here is an idea I have to
> utilize this library with some extra benefits.
> 
>   1 - Convert header files so all functions call a trap, passing a
> structure to the trap function.  This structure contains the function
> name, the filename where the function can be found, a specific base
> register (which the called function can use to allocate data storage,
> which is moveable and specific to  the app).  
> 
>   2 - The trap calls the dld program to link code into the program.  Code
> is shared, and allocated from global shared memory, data areas are allocated
> and a pointer to them is saved in the passed structure - the data belongs
> to the application, and is free'd when the app exits, the code can be
> free'd any time, even when an application that uses it is still running.  The
> data segments may be moved to defrag memory (if needed).
> 
>   3 - When memory gets low, the libraries can be unloaded.  This allows all
> the features of dld, plus the programmer never needs to be concerned with
> when to load or unload a library, and never needs to call any init functions.
> In  fact, most code should compile "out of the box".  It's almost like a
> virtual memory system that makes best use of shared code and shared libs.
> Just think - only 1 copy of any function in memory at any once time, and
> no unused functions will ever take up memory.  
> 
> There would be a run-time penalty for this that using the regular dld would
> not have, but it doesn't look like dld can share libraries.  I was curious
> on everyone's comments and suggestions to this idea.  It looks doable, just
> encapsulate existing code into a shareable trap interface.
> 

Well here are just my 0.02$ (again :-). First of all let me say that I
suspect your proposal might be doable, but it will actually need a lot
of work. 

Ad 1: This looks nice at a first glance, but it completly hides all
difficulties behind the idea. Mainly: Where do you get the base
register from? And who should the called function allocate storage in
this area? Do you want to use some replacement of malloc? And nobody
has ever written code for moveable storage on the Atari ST (Apple has
provided something like that on the Macintosh; but before you can use
such storage, you'd have to lock the region and memory. Also it
requires you to use double indirection on every pointer!).

Ad 2: The main difficulty here is to get shared code. With plain dld
code is loaded into the data section of the executable, so it is by
definition not shared. While it certainly is possible to load the code
into some shared memory (and provide some per process data storage),
this will actually require position independent code to build (as
would be for a true shared library as well). But this isn't yet
supported for gcc on the Atari ST (even less with other compilers :-(
); the -fpic flag does work, but the code generated is not useable
(see below for details).

Ad 3: This would (in contrast to the points mentioned before) require
some kernel changes, so that either a process can be notified about
low storage, or you'd have to integrate the whole dld code into the
kernel.

To summarize: I do not think dld is the key for getting us shared
libraries. Keep in mind that while dynamic loading is required for
shared libraries (actually there is even less required than full
dynamic loading; you'd only have to do some linking and loading on
program startup), there are other important facts to get true shared
libraries beyond dynamic loading. The main point about shared
libraries (that your proposal --- and dld as well --- totally
neglects) is to provide shareable code. Only compiling with -mbaserel
is not enough to get shareable and dynamically loadable code. You have
to go through some pain to set up the base pointer for every loaded
module right (remember that the data section of each loaded module can
be loaded at different offset in different programs).

The usual (at least in the BSD world) approach is to compile so called
position independent code, which accesses all global variables
indirectly through the the global offset table. While the -fpic option
(which forces position independent code) is provided by gcc, it does
not produce usable code (at least in my eyes). The global offset table
must be allocated separately for each process, i.e. in the process'
data section, as it will contain pointers to the process' data. In
addition you need a separate global offset table for each loaded
module in the process (as you cannot fix the offsets into the table
for all different programs before linking is actually performed, but
this linking cannot be done dynamically, as the code otherwise would
have to patched). If you compile the code with the -fpic option, gcc
will add some code to the beginning of each (externally visible)
function, which saves the current pointer to the global offset table
and then reloads the global offset table. However this loading is done
from a constant offset relative to the text (sic!) section of the
program, which is actually not what we want. I do not see how this can
be fixed in such a way, that the pointer will be loaded from some
location in the data section of the process (which again must be
accessed through the global offset table!).

There would be an alternative to this scheme, by saving the current
global offset table pointer on every function call (actually only
those that are not linked statically in the current module) and setup
the new pointer to the global offset table before the function is
actually called. However, I do not enough of GCC RTL, to change this
behaviour; so if anybody would volunteer ... :-)

Finally, if we really get around the problem of compiling useable
position independent code on the ST, a solution, where the dynamic
libraries are linked in at load time or in the process startup code
looks more atractive to me than using traps to dynamically link the
code when it is used. This kind of dynamic loading is certainly doable
(and actually won't need so much work). You'd just have to take a
(close) look on who dynamic linking is implemented in e.g. NetBSD. The
main things to change are the lack of mmap and munmap of MiNT (which
definitely require VM) and may be little differences in the structure
of the global offset table.


Regards
Wolfgang

----
Wolfgang Lux
WZH Heidelberg, IBM Germany             Internet: lux@heidelbg.ibm.com
+49-6221-59-4546                        VNET:     LUX at HEIDELBG
+49-6221-59-3500 (fax)	                EARN:     LUX at DHDIBMIP