Multiple architecture problem and proposed solution

Introduction

We are seeing increasing cases of systems that have support for multiple binary targets. The FHS, LSB and other standards do not currently address several runtime issues related to this issue. Several partial solutions for various problems exist but are inconsistent and suboptimal.

This document seeks to record problem use cases, record current, solutions, propose a generic solution that covers as many of the use cases as possible, and detail the implications of such a solution.

Problem

There are several classes of this problem:

1. 32/64 Architectures

   Architectures with support for 32bit and 64bit versions of the same instruction set

   Examples:
   • i386 / x86_64
   • ppc / ppc64
   • s390 / s390x
   • sparc / sparc64
   • mips / mips64
   • sh / sh64
   • potential: ia64(lp32) / ia64
   • potential: hppa / hppa64
   
   Current practices:
   • Red Hat and SuSE on x86_64
     32bit libraries go in /lib
     64bit libraries go in /lib64
     mixed 32/64 in /bin /usr/bin
     
   
   Comments:
   • We have both the case of a mostly 32bit system with some 64bit libs/binaries and a mostly 64 bit system with some 32bit libs/binaries. Ideally they would be treated the same. So what goes in /lib? It may need to exist for legacy reasons, should it be a symlink to /lib32 or /lib64?
     
   • Today ia64 puts 64bit libs in /lib, what happens if ia64 lp32 is implemented? Would it go in /lib32?
   • ia64 32bit(ILP32) and hppa64 are not yet supported but there are rumors of people working on it.
   
   
2. Mixed instruction sets

   Architectures with support for running other instruction sets (either via hardware or software emulation)

   Examples:
   • i386 / ia64 (hardware emulation)
   • i386 / any (via qemu)
   • s390 / any (via Hercules)
   • ia64 / i386 (via ski)
   • potential: other emulation combinations(qemu supports quite a few)
   
   Current practices:
   • Debian 3.0 ia64
     
     • ia32-libs package, installs in /emul/ia32-linux
     • /lib/ld-linux.so.2 symlink to PI in /emul/ia32-linux
     • ia32 binaries in system root (by hand)
   • qemu advocates:
     using the system root for homogeneous
     /usr/local/qemu-arch/ for heterogeneous
     
   
   Comments:
   This could potentially be any target combination.
   
   
3. Mixed OS environment

   Running binaries from one OS on another via a compatibility layer

   Examples:
   • Linux on Other
     • Linux/i386 on FreeBSD/i386
     • Linux/i386 on Solaris/x86
   • Other on Linux
     • Linux on Other
       • Linux/i386 on FreeBSD/i386
       • Linux/i386 on Solaris/i386
       • Linux/ia64 on HP-UX/ia64
     • Other on Linux
       • Solaris/sparc on Linux/sparc
       • HP-UX/hppa on Linux/hppa
       • HP-UX/ia64 on Linux/ia64
       • Irix/mips on Linux/mips
       • osf/1/alpha on Linux/alpha
   • Non-FHS compliant on Linux and system level emulation
     • dosemu
     • wine
     • bochs
     • vmware
     • etc
   
   Current practices:
   • Linux on FreeBSD: /compat/linux
   • Solaris on Linux: /usr/gnemul/sunos/
   • Others usually use pseudoroots or chroots
   Comments:
   As Linux becomes more pervasive, we'll see more and more OSes support Linux runtimes.
   The solution should support the possibility of running any UNIX-like(insist on FHS compliant?) OS. The details will likely be implementation specific. If the implementation is chroot based, the solution should specify where the chroot lives. If the implementation is mixed in the system root, the solution should specify how.
   
   
4. Mixed endian

   Architectures that support running mixed endian binaries.

   Examples:
   • potential: arm, hppa, ppc(can switch on the fly, but linux syscalls are BE), ia64
   • mips / mipsel: not supported at runtime but potentially on the same system
   
   Comments:
   Not currently a large problem but documented here for completeness.
   
   
5. Mixed ABIs and instruction set extensions

   Architectures with more than one ABI.

   Examples:
   • i386 / i586 / i686 / MMX / SSE / etc.
   • AltiVec
   • sh3 vs sh4
   • ABI transitions
   
   
6. Applications that support multiple archs/OSs

   Applications that support multiple archs/subarchs/OSs usually for cross development or performance reasons

   Examples:
   • gcc
     
     • Libs/headers: /usr/lib/gcc-lib/arch-os/version
       example: /usr/lib/gcc-lib/i486-linux/3.3.3
       
     • Binaries: /usr/bin/arch-os-tool
       example: /usr/bin/i386-linux-g++
       
   • libstdc++
     
     • Headers: /usr/include/c++/version/arch-os/
       example: /usr/include/c++/3.2/i386-linux/
   • openssl
     
     • example: /usr/lib/i586/
       
   • octave
     
     • example: /usr/lib/octave/2.1.50/oct/i386-pc-linux-gnu/
       
   • emacs
     
     • example: /usr/lib/emacs/20.7/i386-debian-linux-gnu/
       
   • various interpreters/languages/tools: ccache, grub, rep, ruby, tau, wml, systemimager
   • applications that take advantage of instruction set extensions like MMX, SSE, AltiVEC, etc.

Proposal

• In order to support binaries for multiple targets on the same system, we need to be able to:
  • Runtime
    • install application binaries for multiple targets
    • ensure that those binaries' program interpreter is available without collision
    • support all those applications' library dependencies
    • therefore have multiple copies of the same library (for different targets) installed without collision, since binaries for multiple targets will be installed on the same system
  • Development
    • install multiple copies (for different targets) of development libraries and headers
• Reasonable migration plan for existing implementations
• Consistent solution across implementations for each above problem
• No/few arch specific solutions

• One general solution that solves as many of the above problems as reasonably possible
• Some of the above can be solved by chrooting, but a general one filesystem solution is desired as well
• Address the package namespace issue(installing packages for one arch on another without collision). It doesn't have to solve it, but should at least have a recommendation for runtime implementations.

• FHS changes:
  • Target specific libraries, both shared and static, for arch-os belong in prefix/lib/arch-os/
    
  • If present, (for migration from FHS 2.3)
    • /lib64 is symlinked (or bind mounted) with the desired proper /lib/arch-os directory
    • /lib32 is symlinked (or bind mounted) with the desired proper /lib/arch-os directory
  • Target specific header files for arch-os belong in prefix/include/arch-os/
    
  • Non-target specific libraries and header files remain in prefix/lib and prefix/include
• FHS Examples:
  • /usr/lib/i386-linux/
  • /usr/include/i386-linux/
  • /usr/lib/x86_64-linux/
  • /usr/local/lib/ppc-linux/
  • /usr/local/include/ppc-linux/
  • /usr/X11R6/lib/i386-linux/ (deprecated?)
  • /opt/foo/lib/sparc-solaris/
  • /opt/bar/include/sparc-solaris/
  
  
• Linux Program Interpreter changes:
  • The program interpreter will be /lib/arch-os/ld.so.version
  • If present, (for migration purposes) (current PI names are documented here)
    • /lib/ld-linux.so.2 is a symlink to the desire system default /lib/arch-os/ld.so.2
      Affects: i386-linux, sparc64-linux, sparc-linux, alpha-linux, arm-linux, sh-linux
    • /lib/ld.so.1 is a symlink to the desire system default /lib/arch-os/ld.so.1
      Affects: mips-linux, hppa-linux, cris-linux, ELF default case
    • /lib64/ld64.so.1 is a symlink to /lib/arch-os/ld.so.1
      Affects: powerpc64-linux, s390x-linux
    • /lib/ld-linux-ia64.so.2 is a symlink to the desire system default /lib/ia64-linux/ld.so.2
      Affects: ia64-linux
    • /lib64/ld-linux-x86-64.so.2 is a symlink to /lib/x86_64-linux/ld.so.2
      Affects: x86_64-linux
  • The program interpreter needs to be able to find the libraries installed in prefix/lib/arch-os/ directories by default.
  
  
• LSB Program Interpreter changes:
  • The LSB program interpreter will be /lib/arch-os/ld-lsb.so.version
  • If present, (for migration purposes, current PI names are documented here)
    • /lib/ld-lsb.so.1 is a symlink to /lib/arch-os/ld-lsb.so.1
      
    • /lib/ld-lsb-ppc32.so.1 is a symlink to /lib/powerpc-linux/ld.so.1
      
    • /lib64/ld-lsb-ppc64.so.1 is a symlink to /lib/powerpc64-linux/ld.so.1
      
    • /lib/ld-lsb-ia64.so.1 is a symlink to /lib/ia64-linux/ld-lsb.so.1
      
    • /lib64/ld-lsb-x86-64.so.1 is a symlink to /lib/x86_64-linux/ld-lsb.so.1
      
    • /lib/ld-lsb-s390.so.1 is a symlink to /lib/s390-linux/ld-lsb.so.1
      
    • /lib64/ld-lsb-s390x.so.1 is a symlink to /lib/s390x-linux/ld-lsb.so.1
      
  
  
• Compiler changes:
  • The compiler needs to be able to find the libraries installed in prefix/lib/arch-os/ directories by default
  • The compiler needs to be able to find the header files installed in prefix/include/arch-os/ directories by default
  
  
• Library changes :
  Software can continue to install libraries and header files in the existing prefix/lib/ and prefix/include/ directories. If multi-arch support is desired the following changes are needed,
  • Install libraries in the appropriate prefix/include/arch-os/ directories
  • Install header files in the appropriate prefix/include/arch-os/ directories
  • Any software that has hard-coded library paths needs to be changed to use the PI to locate it's libraries
  • Any development software that has hard-coded library and include paths needs to be changed to use the compiler to locate it's libraries and headers

Implementation Options

Runtimes implementing this proposal can put the libraries/headers/program interpreters wherever they like as long as they can be accessed as described in the proposal. This allows for a range of possible implementations and the ability for a gradual transition. Here are some potential options.

• In all cases, software can expect to be able to install
  • to the old paths
  • to the new paths
  • the needed runtime libraries and PIs for a new target in the correct locations and provide support for that target without needing any changes from the OS provider (an example is a target emulator)

• Option A: Simple, support one target allow for others (most distros today)
  • Libraries remain in prefix/lib and header files in prefix/include
  • Hard links for prefix/lib/arch-os and prefix/lib
  • Due to the above links the PI and LSB-PI may already be correct depending on the target. If not some symlinks may be required.
  • If needed a /lib64 symlink (or bind mount)
  • Hard links for prefix/include/arch-os and prefix/include
  • Because the libraries and headers haven't moved the PI and compiler will still continue to find them in their existing locations. So no changes to the PI and compilers search paths are needed.
  • No support for multi-arch yet in things like system package tools
  
  
• Option B: Normal one target support (where a large set of OS providers will likely move to)
  • Libraries for single target in appropriate prefix/lib/arch-os/ and header files in prefix/include/arch-os/
  • If needed a /lib64 symlink (or bind mount)
  • PI for single target moved into /lib/arch-os/
  • Required PI symlinks for that single target
  • PI, compiler, etc. search paths changed to look in the new locations (in addition to the existing locations)
  • No support for multi-arch yet in things like system package tools
  
  
• Option C: Full multi-arch support (where some OS providers will likely move to)
  • Same as option B, but with native support for multiple targets
  • System totally multi-arch aware including things like package tools etc.

Benefits

• Fixes the problems in a clean and consistent manner
• Meets our required and desired results described in the proposal
• Cleans up some inconsistent PI naming
• Has the added benefit of providing a way to do long term, gradual migration from one target to another. This has a few interesting implications,
  • Migration from one standard pervasive legacy target to a new not yet widely adopted target, by providing support for the legacy target during a transition period. (Maybe we can finally move away from i386 and Legacy UNIX OSes?)
  • Migration from one target ABI to another ABI on the same target. Something has hasn't been easy before this.

FAQ

1. Isn't this impossible?
   No, Debian has a small-scale implementation already.
2. Will all software need to switch to this proposal right away?
   No, they can continue to install to the existing paths indefinitely. The main reason for changing to the new model will be to enable that software to be used on a multi-arch system.
3. Will I be able to install the same binary for two different targets on the same system?
   No, not without changing the name to avoid collision. This proposal assumes that bin directories don't need to be differentiated since users won't want more than one version of a binary installed. If there is demand for that it could be addressed in a similar manner, but we don't want to deal with it until there is sufficient demand.
4. Linux's ldconfig will walk the whole prefix/lib/ tree finding libraries for various targets, won't it blow up?
   No, its smart enough to ignore stuff not for it's target.
5. What about software that creates subdirectories under prefix/lib/ (or include)
   These will need to be evaluated on a case by case basis depending on if the content they are providing under that subdirectory is target specific.
6. What about software with binary plug-ins like apache?
   You can't mix and match target plug-ins with one binary apache today. In the case where you could, that software would want to follow the proposal and structure their software accordingly to avoid filesystem collisions.
7. Why not prefix/arch-os/lib/ (and include/)?
   It would pollute the prefix directory. Can you imagine adding one entry for each target to the root and /usr directories? Better that they go under the prefix/lib/ (and include/) directories which already contain many files.