Technical blog

[deadwood]

PS. Yes, I know that a forum post is not a blog.  This is something I'm "starting small" to see if there is need and value for other from these kind of articles. They will cover all projects I work on (ABiv0, ABIv11 and AxRuntime). Please share your opinion!

----------
The next update on AxRuntime will allow creation of dual-run executables. This means the same executable can be started from Linux, either from desktop or from console, and from within "running" AxRuntime program. The second case is the case of using RunCommand() or System() APIs in your application or simply running commands and script from AROS-Shell. A command is nothing else than an executable "loaded" into shell, executed and unloaded.

Current version of AxRuntime, the 2.0 does not have the dual-run capability. It generates plain Linux ELF executables that can only be started from Linux side. On the other hand the libraries, like dos.library or intuition.library are actually ELF shared objects which are dynamically loaded uses dlopen() API. This already works great, for example AROS MPlayer compiled for AxRuntime is fully usable under Linux.



On AROS/Amiga side, loading of every binary follows the same path. Binary is loaded using LoadSeg(), control jumps to entry point function and once control returns, binary is unloaded. That's of course simplified description - in reality a few more things are happening. The AROS/Amiga entry point function has three standard arguments

Code: [Select]

(STRPTR argstr, LONG argsize, struct ExecBase *SysBase)

Knowing this, the first approach to dual-run was simple. Let's dlopen() a linux binary, execute code, and dlclose() it, mirroring AROS/Amiga behavior. The problems appeared already on first step. Since a couple of years it is no longer possible to dlopen() an ELF executable - this can only be done with ELF shared object. I didn't dive into understanding the reasons, but that's the situation right now. This meant that simplest approach failed.

Second approach then was the opposite - since ELF shared objects can be dynamically loaded to support AROS/Amiga needs, can a ELF shared object be made to be executed on Linux side? Good news is that it can! It is obviously not standard, but possible and supported. For example you can "run" Linux C library

Code: [Select]

$ /usr/lib/x86_64-linux-gnu/libc.so.6
and it will display its version and license information.

The process of achieving this is simple, once you get to understand it of course. First, the shared object needs to have a link to program interpreter placed in a specific section. This is achieved by following code:

Code: [Select]

const char dl_loader[] __attribute__((section(".interp"))) = "/lib64/ld-linux-x86-64.so.2";

Next, during building the linker needs to be informed what will be entry point that should be executed. There is a convenient linker option for this that is now embedded in axrt.specs file use during building of Ax executables. For reference, the standardized, Linux-side entry is called _axrt_start and can be viewed at https://github.com/deadw00d/AROS/blob/alt-runtime/arch/all-runtime/axrt/startup/startup.c#L127. Last thing, that is important for debugging is to add DEBUG symbols in .dynamic section of ELF shared object. GDB uses that symbols to communicated with program interpret - without it, resolving symbols doesn't work and you are left with debuggin assembler.

Last thing to cover is AROS/Amiga-side startup. As mentioned earlier this is a function with standardized arguments and in AxRuntim case it name is standardized to __startup_entry and can be seen at https://github.com/deadw00d/AROS/blob/alt-runtime/arch/all-runtime/axrt/startup/startup.c#L58. The name standardization is important for AxRuntime modules loader to locate it. Since the shared-object-executable can now be opened by dlopen(), dlsym() can be used to find __startup_entry function because it is exported from the ELF object.

And that's it. When executing from Linux, _axrt_start takes care of runtime initialization and eventually jumps to __startup_entry. When executing from already initialized runtime, control jumps directly to __startup_entry. Effect is for example AROS-Shell which can load and run commands!

[Amiwell]

It seems like a good idea to share the results of your work on a single thread, good work deadwood and thank you

[OlafS3]

so in theory you could use wanderer as desktop, start the shelll and execute a command, wanderer and shell from aros and command from linux?

[deadwood]

@OlafS3

I'm not sure I understand. Can you describe the scenario in more details? One thing to note is that AROS-Shell is not yet capable of launching Linux-side commands. I think that will also be possible with some tricks, but we are not yet there.

[deadwood]

It seems like a good idea to share the results of your work on a single thread, good work deadwood and thank you

Thanks

[OlafS3]

@OlafS3

I'm not sure I understand. Can you describe the scenario in more details? One thing to note is that AROS-Shell is not yet capable of launching Linux-side commands. I think that will also be possible with some tricks, but we are not yet there.

Yes that was the idea, that you can mix aros and linux like you can on aros 68k with aros and amiga. The advantage would be you could use linux components without needing sources and compile it in a special way.

[magorium]

Please share your opinion!

I love seeing those kind of posts !

As a software (not system) developer i am not familiar with all inner workings so everything that sheds a light onto things is a plus for me.

Please keep it up if you are able to.

[deadwood]

I love seeing those kind of posts !

As a software (not system) developer i am not familiar with all inner workings so everything that sheds a light onto things is a plus for me.

Please keep it up if you are able to.

Thanks for sharing.

[miker1264]

I love seeing those kind of posts !

As a software (not system) developer i am not familiar with all inner workings so everything that sheds a light onto things is a plus for me.

Please keep it up if you are able to.

Thanks for sharing.

It's good to see progress. 

Thanks deadwood.

[deadwood]

64-bit time
=========

While working on next version of AxRuntime I came to an interesting problem that started with a simple compilation problem with "struct timeval" but turned into a deeper issue.

What is "struct timeval"? It is a simple structure that is used to store time value, composed of two fields: one contains number of seconds from a certain point in time and the other contains number of microseconds in current second. Seems not so special, but what is special is that the name of the structure, timeval, is used at the same time in Amiga API and in C library API. In both cases however, the field names are slightly different. The good thing however was that in both cases the sizes of the fields were the same so it was possible to use the following trick:

Code: [Select]

__extension__ struct timeval
{
    union  /* Seconds passed. */
    {
        unsigned AROS_32BIT_TYPE tv_secs;   /* AROS field */
        unsigned AROS_32BIT_TYPE tv_sec;    /* POSIX field */
    };
    union /* Microseconds passed in the current second. */
    {
        unsigned AROS_32BIT_TYPE tv_micro; /* AROS field */
        signed   AROS_32BIT_TYPE tv_usec;  /* POSIX field */
    };
};

What it does, in short, is instruct a compiler to accept the first field name as either tv_secs or tv_sec and the second as tv_micro or tv_usec. Amiga-like source codes use first field names, source codes using C library functions use second field names. Everything works as expected.

Why the the topic then?

Because AxRuntime is a bit special. It does not have its own C library - it uses C library of Linux and it turned out that some time ago Linux migrated from 32 bit fields to 64 bit fields to solve the Y2038 problem. You can read more about it here and here. This however means two things:
1) Simple trick as above won't work anymore, because "POSIX" fields are now 64-bit wide, while "AROS" field is still expected to 32 bit
2) Amiga API also suffers from the same integer overflow problem (but further in time, at year 2106)

I was hang up on this issues for some time and eventually came to conclusion that the best way forward is to actually extend Amiga APIs to support 64-bit time. That will take some time however, but what can be done already today is to prepare for the transition, so that software compiled today will continue working after the transitions happens. This meant the following:

A) Reviewing all places where time is handled in AROS and deciding whether 32-bit size is enough. These can be places like delays or time offsets - essentially places that work with "relative" time. No change needs to happen there as 32 bit is enough to store these offsets. On the other hand places that require "absolute" time (like a valid date) will have to migrate to 64-bit time in future.
B) Draft a specification for 64-bit time that will describe final state and transition steps. You can find it at https://github.com/deadw00d/AROS/blob/master/rom/timer/64bit. When creating it I researched both Amiga OS 4 and MorphOS and neither of them migrated to 64-bit time yet, so there was nothing to base on. The specification follows however principles that were laid out when adding 64-bit filesystem support with regards to naming and backwards compatibility.

As for solving the immediate problem of AxRuntime (point 1. above), another trick was used, making the AROS-side struct look like it has 32-bit fields:

Code: [Select]

__extension__ struct timeval
{
    union
    {
        struct
        {
        unsigned AROS_32BIT_TYPE tv_secs;   /* AROS field */
        unsigned AROS_32BIT_TYPE tv_micro;  /* AROS field */
        };
        /* Seconds passed. */
        __time_t tv_sec;                 /* POSIX field */
    };
    /* Microseconds passed in the current second. */
    __suseconds_t tv_usec;                  /* POSIX field */
};

I know it feels hacky. It will be properly solved once 64-bit time is introduced in AROS. When this happens, struct timeval will be 64-bit wide on both sides, AROS and POSIX.