人生在世，看得透，又看得远者prevail everywhere.

gg.gg/13nbnz gg.gg/13p5tj gg.gg/13p9s5 gg.gg/13tljl gg.gg/13xudz gg.gg/13xy3p gg.gg/143tqc linux.die.net linux.it.net.cn ostechnix.com unix.com gg.gg/19yv96 man.linuxde.net gg.gg/148erg bit.ly/2vsM34J bit.ly/2EzoUDo bit.ly/2wCsZSI bit.ly/2v6jGJi bit.ly/2tW6eYT bit.ly/2u2MMtm bit.ly/2X6vadl bit.ly/2viLpHU bit.ly/2fzzbEU linuxprobe.com linuxtechi.com ttlsa.com systutorials.com ghacks.net linuxopsys.com reurl.cc/8W1x3X reurl.cc/NpzMWe reurl.cc/WrgYdx reurl.cc/Yv4Yvo reurl.cc/Lmy90K reurl.cc/Rr5aeG

Tuesday 27 August 2019

xgo - Go CGO cross compiler

Although Go strives to be a cross platform language, cross compilation from one platform to another is not as simple as it could be, as you need the Go sources bootstrapped to each platform and architecture.

The first step towards cross compiling was Dave Cheney's golang-crosscompile package, which automatically bootstrapped the necessary sources based on your existing Go installation. Although this was enough for a lot of cases, certain drawbacks became apparent where the official libraries used CGO internally: any dependency to third party platform code is unavailable, hence those parts don't cross compile nicely (native DNS resolution, system certificate access, etc).

A step forward in enabling cross compilation was Alan Shreve's gonative package, which instead of bootstrapping the different platforms based on the existing Go installation, downloaded the official pre-compiled binaries from the golang website and injected those into the local toolchain. Since the pre-built binaries already contained the necessary platform specific code, the few missing dependencies were resolved, and true cross compilation could commence... of pure Go code.

However, there was still one feature missing: cross compiling Go code that used CGO itself, which isn't trivial since you need access to OS specific headers and libraries. This becomes very annoying when you need access only to some trivial OS specific functionality (e.g. query the CPU load), but need to configure and maintain separate build environments to do it.

Enter xgo

My solution to the challenge of cross compiling Go code with embedded C/C++ snippets (i.e. CGO_ENABLED=1) is based on the concept of lightweight Linux containers. All the necessary Go tool-chains, C cross compilers and platform headers/libraries have been assembled into a single Docker container, which can then be called as if a single command to compile a Go package to various platforms and architectures.

Installation

Although you could build the container manually, it is available as an automatic trusted build from Docker's container registry (not insignificant in size):

docker pull karalabe/xgo-latest

To prevent having to remember a potentially complex Docker command every time, a lightweight Go wrapper was written on top of it.

go get github.com/karalabe/xgo

Usage

Simply specify the import path you want to build, and xgo will do the rest:

$ xgo github.com/project-iris/iris
...

$ ls -al
-rwxr-xr-x  1 root  root    9995000 Nov 24 16:44 iris-android-16-arm
-rwxr-xr-x  1 root  root    6776500 Nov 24 16:44 iris-darwin-10.6-386
-rwxr-xr-x  1 root  root    8755532 Nov 24 16:44 iris-darwin-10.6-amd64
-rwxr-xr-x  1 root  root    7114176 Nov 24 16:45 iris-ios-5.0-arm
-rwxr-xr-x  1 root  root   10135248 Nov 24 16:44 iris-linux-386
-rwxr-xr-x  1 root  root   12598472 Nov 24 16:44 iris-linux-amd64
-rwxr-xr-x  1 root  root   10040464 Nov 24 16:44 iris-linux-arm
-rwxr-xr-x  1 root  root    7516368 Nov 24 16:44 iris-windows-4.0-386.exe
-rwxr-xr-x  1 root  root    9549416 Nov 24 16:44 iris-windows-4.0-amd64.exe

If the path is not a canonical import path, but rather a local path (starts with a dot . or a dash /), xgo will use the local GOPATH contents for the cross compilation.

Build flags

A handful of flags can be passed to go build. The currently supported ones are

-v: prints the names of packages as they are compiled
-x: prints the build commands as compilation progresses
-race: enables data race detection (supported only on amd64, rest built without)
-tags='tag list': list of build tags to consider satisfied during the build
-ldflags='flag list': arguments to pass on each go tool link invocation
-buildmode=mode: binary type to produce by the compiler

Go releases

As newer versions of the language runtime, libraries and tools get released, these will get incorporated into xgo too as extensions layers to the base cross compilation image (only Go 1.3 and above will be supported).

You can select which Go release to work with through the -go command line flag to xgo and if the specific release was already integrated, it will automatically be retrieved and installed.

$ xgo -go 1.6.1 github.com/project-iris/iris

Additionally, a few wildcard release strings are also supported:

latest will use the latest Go release (this is the default)
1.6.x will use the latest point release of a specific Go version
1.6-develop will use the develop branch of a specific Go version
develop will use the develop branch of the entire Go repository

Output prefixing

xgo by default uses the name of the package being cross compiled as the output file prefix. This can be overridden with the -out flag.

$ xgo -out iris-v0.3.2 github.com/project-iris/iris
...

$ ls -al
-rwxr-xr-x  1 root  root   9995000 Nov 24 16:44 iris-v0.3.2-android-16-arm
-rwxr-xr-x  1 root  root   6776500 Nov 24 16:44 iris-v0.3.2-darwin-10.6-386
-rwxr-xr-x  1 root  root   8755532 Nov 24 16:44 iris-v0.3.2-darwin-10.6-amd64
-rwxr-xr-x  1 root  root   7114176 Nov 24 16:45 iris-v0.3.2-ios-5.0-arm
-rwxr-xr-x  1 root  root  10135248 Nov 24 16:44 iris-v0.3.2-linux-386
-rwxr-xr-x  1 root  root  12598472 Nov 24 16:44 iris-v0.3.2-linux-amd64
-rwxr-xr-x  1 root  root  10040464 Nov 24 16:44 iris-v0.3.2-linux-arm
-rwxr-xr-x  1 root  root   7516368 Nov 24 16:44 iris-v0.3.2-windows-4.0-386.exe
-rwxr-xr-x  1 root  root   9549416 Nov 24 16:44 iris-v0.3.2-windows-4.0-amd64.exe

Branch selection

Similarly to go get, xgo also uses the master branch of a repository during source code retrieval. To switch to a different branch before compilation pass the desired branch name through the --branch argument.

$ xgo --branch release-branch.go1.4 golang.org/x/tools/cmd/goimports
...

$ ls -al
-rwxr-xr-x  1 root  root   4171248 Nov 24 16:40 goimports-android-16-arm
-rwxr-xr-x  1 root  root   4139868 Nov 24 16:40 goimports-darwin-10.6-386
-rwxr-xr-x  1 root  root   5186720 Nov 24 16:40 goimports-darwin-10.6-amd64
-rwxr-xr-x  1 root  root   3202364 Nov 24 16:40 goimports-ios-5.0-arm
-rwxr-xr-x  1 root  root   4189456 Nov 24 16:40 goimports-linux-386
-rwxr-xr-x  1 root  root   5264136 Nov 24 16:40 goimports-linux-amd64
-rwxr-xr-x  1 root  root   4209416 Nov 24 16:40 goimports-linux-arm
-rwxr-xr-x  1 root  root   4348416 Nov 24 16:40 goimports-windows-4.0-386.exe
-rwxr-xr-x  1 root  root   5415424 Nov 24 16:40 goimports-windows-4.0-amd64.exe

Remote selection

Yet again similarly to go get, xgo uses the repository remote corresponding to the import path being built. To switch to a different remote while preserving the original import path, use the --remote argument.

$ xgo --remote github.com/golang/tools golang.org/x/tools/cmd/goimports
...

Package selection

If you used the above branch or remote selection machanisms, it may happen that the path you are trying to build is only present in the specific branch and not the default repository, causing Go to fail at locating it. To circumvent this, you may specify only the repository root for xgo, and use an additional --pkg parameter to select the exact package within, honoring any prior branch and remote selections.

$ xgo --pkg cmd/goimports golang.org/x/tools
...

$ ls -al
-rwxr-xr-x  1 root  root   4194956 Nov 24 16:38 goimports-android-16-arm
-rwxr-xr-x  1 root  root   4164448 Nov 24 16:38 goimports-darwin-10.6-386
-rwxr-xr-x  1 root  root   5223584 Nov 24 16:38 goimports-darwin-10.6-amd64
-rwxr-xr-x  1 root  root   3222848 Nov 24 16:39 goimports-ios-5.0-arm
-rwxr-xr-x  1 root  root   4217184 Nov 24 16:38 goimports-linux-386
-rwxr-xr-x  1 root  root   5295768 Nov 24 16:38 goimports-linux-amd64
-rwxr-xr-x  1 root  root   4233120 Nov 24 16:38 goimports-linux-arm
-rwxr-xr-x  1 root  root   4373504 Nov 24 16:38 goimports-windows-4.0-386.exe
-rwxr-xr-x  1 root  root   5450240 Nov 24 16:38 goimports-windows-4.0-amd64.exe

This argument may at some point be integrated into the import path itself, but for now it exists as an independent build parameter. Also, there is not possibility for now to build mulitple commands in one go.

Limit build targets

By default xgo will try and build the specified package to all platforms and architectures supported by the underlying Go runtime. If you wish to restrict the build to only a few target systems, use the comma separated --targets CLI argument:

--targets=linux/arm: builds only the ARMv5 Linux binaries (arm-6/arm-7 allowed)
--targets=windows/*,darwin/*: builds all Windows and OSX binaries
--targets=*/arm: builds ARM binaries for all platforms
--targets=*/*: builds all suppoted targets (default)

The supported targets are:

Platforms: android, darwin, ios, linux, windows
Achitectures: 386, amd64, arm-5, arm-6, arm-7, arm64, mips, mipsle, mips64, mips64le

Platform versions

By default xgo tries to cross compile to the lowest possible versions of every supported platform, in order to produce binaries that are portable among various versions of the same operating system. This however can lead to issues if a used dependency is only supported by more recent systems. As such, xgo supports the selection of specific platform versions by appending them to the OS target string.

--targets=ios-8.1/*: cross compile to iOS 8.1
--targets=android-16/*: cross compile to Android Jelly Bean
--targets=darwin-10.9/*: cross compile to Mac OS X Mavericks
--targets=windows-6.0/*: cross compile to Windows Vista

The supported platforms are:

All Android APIs up to Android Lollipop 5.0 (API level ids)
All Windows APIs up to Windows 8.1 limited by mingw-w64 (API level ids)
OSX APIs in the range of 10.6 - 10.11
All iOS APIs up to iOS 9.3

Mobile libraries

Apart from the usual runnable binaries, xgo also supports building library archives for Android (android/aar) and iOS (ios/framework). Opposed to gomobile however xgo does not derive library APIs from the Go code, so proper CGO C external methods must be defined within the package.

In the case of Android archives, all architectures will be bundled that are supported by the requested Android platform version. For iOS frameworks xgo will bundle armv7 and arm64 by default, and also the x86_64 simulator builds if the iPhoneSimulator.sdk was injected by the user:

Create a new docker image based on xgo: FROM karalabe/xgo-latest
Inject the simulator SDK: ADD iPhoneSimulator9.3.sdk.tar.xz /iPhoneSimulator9.3.sdk.tar.xz
Bootstrap the simulator SDK: $UPDATE_IOS /iPhoneSimulator9.3.sdk.tar.xz

CGO dependencies

The main differentiator of xgo versus other cross compilers is support for basic embedded C/C++ code and target-platform specific OS SDK availability. The current xgo release introduces an experimental CGO dependency cross compilation, enabling building Go programs that require external C/C++ libraries.

It is assumed that the dependent C/C++ library is configure/make based, was properly prepared for cross compilation and is available as a tarball download (.tar, .tar.gz or .tar.bz2). Further plans include extending this to cmake based projects, if need arises (please open an issue if it's important to you).

Such dependencies can be added via the --deps argument. They will be retrieved prior to starting the cross compilation and the packages cached to save bandwidth on subsequent calls.

A complex sample for such a scenario is building the Ethereum CLI node, which has the GNU Multiple Precision Arithmetic Library as it's dependency.

$ xgo --deps=https://gmplib.org/download/gmp/gmp-6.1.0.tar.bz2  \
    --targets=windows/* github.com/ethereum/go-ethereum/cmd/geth
...

$ ls -al
-rwxr-xr-x 1 root root 16315679 Nov 24 16:39 geth-windows-4.0-386.exe
-rwxr-xr-x 1 root root 19452036 Nov 24 16:38 geth-windows-4.0-amd64.exe

Some trivial arguments may be passed to the dependencies' configure script via --depsargs.

$ xgo --deps=https://gmplib.org/download/gmp/gmp-6.1.0.tar.bz2  \
    --targets=ios/* --depsargs=--disable-assembly               \
    github.com/ethereum/go-ethereum/cmd/geth
...

$ ls -al
-rwxr-xr-x 1 root root 14804160 Nov 24 16:32 geth-ios-5.0-arm

Note, that since xgo needs to cross compile the dependencies for each platform and architecture separately, build time can increase significantly.

from https://github.com/karalabe/xgo
-------------------

交叉编译Go程序

有几个Go的程序需要放在树莓派上跑，由于树莓派的性能太差、编译工具版本又低，想想还是在其他设备上编译好可执行文件直接拿来用方便，毕竟无需依赖的静态编译是Go的亮点嘛。

Go对交叉编译的支持是极好的：对于没有使用CGO的程序，只需要通过设置$CGO_ENABLED、$GOOS、$GOARCH参数即可轻松地实现跨平台编译；对于使用CGO的程序，大部分情况可以通过配置$CC参数使用自行准备的交叉编译工具进行编译。

I. 重点在这里

1、简单来说，没有用到 CGO 的 Golang 程序，直接用编译器自带的跨平台特性即可全平台编译;

2、用到了 CGO 需要发布到ARM平台的Golang程序，推荐自己配置ARM交叉编译环境;

3、用到了 CGO 需要发布到Win/Linux/Mac平台的Golang程序，推荐用使用go-ui-crossbuild 。

下面以macOS为例，简单记录一下过程中踩的坑吧。

不使用CGO的交叉编译

II. 一般做法

在macOS编译Linux和Windows的arm可执行程序：

    CGO_ENABLED=0 GOOS=linux GOARCH=arm go build main.go
    CGO_ENABLED=0 GOOS=windows GOARCH=arm go build main.go

其中：

$CGO_ENABLED：0表示关闭CGO
$GOOS：目标平台（编译后要运行的目标平台）的操作系统（darwin、freebsd、linux、windows）
$GOARCH：目标平台（编译后要运行的目标平台）的体系架构（386、amd64、arm）分别对应（32位、64位、ARM平台）的架构

树莓派属于ARM平台，对于编译给ARM使用的Go程序，需要根据实际情况配置$GOARM，这是用来控制CPU的浮点协处理器的参数。$GOARM默认是6，对于不支持VFP使用软件运算的老版本ARM平台要设置成5，支持VFPv1的设置成6，支持VFPv3的设置成7，详细说明如下：

GOARM=5: use software floating point; when CPU doesn't have VFP co-processor

GOARM=6: use VFPv1 only; default if cross compiling; usually ARM11 or better cores (VFPv2 or better is also supported)

GOARM=7: use VFPv3; usually Cortex-A cores

综上，给跑着linux的老版本树莓派编译Go程序的命令为：

    CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=5 go build main.go

III. 关于协处理器

协处理器是协助CPU完成其无法执行或执行效率、效果低下的处理工作而开发和应用的处理器。

这种CPU无法执行的专项工作有很多，比如设备间的信号传输、接入设备的管理等；执行效率、效果低下的有图形处理、声频处理等。为了进行这些专项处理，各种辅助处理器就诞生了。现在的计算机构架（x86、amd64）中，整数运算器与浮点运算器已经集成在一起，一般来讲因此浮点处理器内建于CPU中的协处理器，不算是辅助处理器，不过ARM架构因为低功耗的原因是个例外。

ARM架构中的VFP coprocessor是典型的协处理器，它提供浮点数基本运算（加、减、乘、除、开方、比较、取反）以及向量（vectors）功能，一般来讲，VFP可以同时支持最多8组单精度4组双精度浮点数的运算。

简单地说，使用协处理器可以提升某一方面的性能，可以粗暴地认为是一种硬件加速。

使用CGO的交叉编译

IV. 什么是CGO

C语言经过数十年发展，各个方面的开源代码、闭源库已经非常丰富。这对于一门现代编程语言而言，如何用好现成的C代码就显得极为重要，CGO就是一个令人惊异的技术，它允许Go与C的类库交互操作，让Go能够使用C语言积累的各种资源。

CGO作为Go中使用C语言代码的一种方式，在获得好处的同时就必定得有付出，这是亘古不变之理：由于CGO相当于使用了C语言，所以编译使用CGO部分的代码就必须按C语言路子来，而C语言原生的跨平台支持相对Go来说可是差远了，使得使用CGO后要编译跨平台的可执行文件就麻烦多了。

目前解决CGO跨平台编译问题的思路有：

用目标平台的工具链进行交叉编译
用原生代码重写CGO实现的功能

V. 用目标平台的工具链进行交叉编译

以树莓派的ARM平台为例，用ARM GNU Linux工具链，编译使用了CGO的Go项目。

macOS的文件系统默认是大小写不敏感的，而交叉编译工具链是基于大小写敏感的文件系统的，所以应当新建一个大小写敏感的磁盘镜像用于安放ARM GNU Linux工具链。打开Disk Utility，然后 File>New Image>Blank Image，然后在弹出窗口中设置名称为armx(名称自定)，大小大于512MB，格式为Mac OS Extended的映像。

下载或者按说明自行编译ARM GNU Linux工具链。

假设工具链包已经在~/Download目录中，解压缩工具链到到新建的磁盘镜像中：

    tar -zxv -C /Volumes/armx/ --strip-components 1 -f ~/Download/ARMx-2009q3-67.tar.bz2

现在，你就已经能够直接使用ARM工具链了，比如要编译main.c程序，可以在终端执行：

    /Volumes/Armx/bin/arm-none-linux-gnueabi-gcc main.c -o main

就可以获得一个名为main的可执行程序，这个程序在macOS是不能运行的，只能在ARM平台设备上的Linux系统中才能执行。

好了，现在开始编译用了CGO的Go程序给树莓派用吧：

    CGO_ENABLED=1 GOOS=linux GOARCH=arm CC=/Volumes/Armx/bin/arm-none-linux-gnueabi-gcc GOARM=5 go build -x main.go

其中$CC参数指定的是ARM工具链位置，如果不出意外的话，编译完成后就可以得到可以在树莓派上运行的Go程序了。

如果使用Beego提供的bee工具进行编译和打包，效果也是一样的的。

    CGO_ENABLED=1 GOOS=linux GOARCH=arm CC=/Volumes/Armx/bin/arm-none-linux-gnueabi-gcc GOARM=5 ~/go/bin/bee pack main.go

交叉编译过程中如果出现错误，多半是因为使用CGO部分的代码，在不同的平台存在不兼容的问题，需要根据错误提示逐项解决。

VI. 使用go-ui-crossbuild

go-ui-crossbuild是通过预制好的CGO跨平台编译环境，实现一键编译MacOS、Linux和Windows成品的项目，该项目的地址托管在Github上，使用起来也很简单，下载好docker镜像启动之后在go程序目录执行如下命令即可。

    docker run -v $GOPATH/project_name:/go/src/project_name magj/go-ui-crossbuild gouicrossbuild project_name ./cmd/gui

如果涉及到自身的包依赖，可以参考下列命令，将本地的src都映射到docker中去，编译成功之后会在项目文件夹下生成一个build文件夹，里面分别有MacOS用的app文件，Windows用的exe文件和Linux用的可执行文件。

    docker run -v /Users/holmesian/go/src/:/go/src/ magj/go-ui-crossbuild gouicrossbuild getDocument getDocument ./build

其中getDocument是项目名称，项目目录为/Users/holmesian/go/src/getDocument 。
-----------------------

相关帖子：https://briteming.blogspot.com/2016/12/travisgo.html

Total Pageviews