移植libibverbs到riscv架构
编译环境:OS: Ubuntu 20.10LTSx86-linux-gnu: gcc 10.3.0riscv-unknown-linux-gnu: gcc 9.2.01、获取libibverbs源码github链接:https://github.com/ImagineMiracle-wxn/libibverbsimaginemiracle:~$ git clone -b for-upstream
·
编译环境:OS: Ubuntu 20.10LTSx86-linux-gnu: gcc 10.3.0riscv-unknown-linux-gnu: gcc 9.2.0
1、获取libibverbs源码
github链接:https://github.com/ImagineMiracle-wxn/libibverbs
imaginemiracle:~$ git clone -b for-upstream git@github.com:ImagineMiracle-wxn/libibverbs.git
imaginemiracle:~$ cd libibverbs
2、交叉编译libibverbs
执行configure文件配置源码。
imaginemiracle:libibverbs$ ./configure --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/build/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu
配置输出如下:
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for riscv64-unknown-linux-gnu-strip... riscv64-unknown-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for riscv64-unknown-linux-gnu-gcc... riscv64-unknown-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... none needed
checking whether riscv64-unknown-linux-gnu-gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of riscv64-unknown-linux-gnu-gcc... gcc3
checking how to run the C preprocessor... riscv64-unknown-linux-gnu-gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether ln -s works... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... riscv64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by riscv64-unknown-linux-gnu-gcc... /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld
checking if the linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B
checking the name lister (/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to riscv64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld option to reload object files... -r
checking for riscv64-unknown-linux-gnu-objdump... riscv64-unknown-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for riscv64-unknown-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for riscv64-unknown-linux-gnu-ar... riscv64-unknown-linux-gnu-ar
checking for archiver @FILE support... @
checking for riscv64-unknown-linux-gnu-strip... (cached) riscv64-unknown-linux-gnu-strip
checking for riscv64-unknown-linux-gnu-ranlib... riscv64-unknown-linux-gnu-ranlib
checking command to parse /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B output from riscv64-unknown-linux-gnu-gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for riscv64-unknown-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if riscv64-unknown-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for riscv64-unknown-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if riscv64-unknown-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if riscv64-unknown-linux-gnu-gcc static flag -static works... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the riscv64-unknown-linux-gnu-gcc linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for riscv64-unknown-linux-gnu-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libnl-3.0... yes
checking for libnl-route-3.0... yes
checking for dlsym in -ldl... yes
checking for pthread_mutex_init in -lpthread... yes
checking for ANSI C header files... (cached) yes
checking valgrind/memcheck.h usability... no
checking valgrind/memcheck.h presence... no
checking for valgrind/memcheck.h... no
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking for an ANSI C-conforming const... yes
checking whether O_CLOEXEC is declared... yes
checking whether SOCK_CLOEXEC is declared... yes
checking for close on exec modifier for fopen()... e
checking whether ld accepts --version-script... yes
checking for .symver assembler support... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libibverbs.spec
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
成功生成Makefile文件，使用make命令编译源码。
imaginemiracle:libibverbs$ make
编译输出如下: 这里出现了错误。
make all-am
make[1]: Entering directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
CC examples/device_list.o
In file included from examples/device_list.c:43:
./include/infiniband/arch.h:135:2: warning: #warning No architecture specific defines found. Using generic implementation. [-Wcpp]
135 | #warning No architecture specific defines found. Using generic implementation.
| ^~~~~~~
CC src/libibverbs_la-cmd.lo
CC src/libibverbs_la-compat-1_0.lo
CC src/libibverbs_la-device.lo
In file included from src/device.c:48:
./include/infiniband/arch.h:135:2: warning: #warning No architecture specific defines found. Using generic implementation. [-Wcpp]
135 | #warning No architecture specific defines found. Using generic implementation.
| ^~~~~~~
CC src/libibverbs_la-init.lo
src/init.c: In function 'find_sysfs_devs':
src/init.c:111:9: warning: '%s' directive output may be truncated writing up to 255 bytes into a region of size between 0 and 255 [-Wformat-truncation=]
111 | "%s/%s", class_path, dent->d_name);
| ^~
src/init.c:110:3: note: 'snprintf' output between 2 and 512 bytes into a destination of size 256
110 | snprintf(sysfs_dev->sysfs_path, sizeof sysfs_dev->sysfs_path,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
111 | "%s/%s", class_path, dent->d_name);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/init.c:123:5: warning: '%s' directive output may be truncated writing up to 255 bytes into a region of size 64 [-Wformat-truncation=]
123 | "%s", dent->d_name);
| ^~
src/init.c:122:3: note: 'snprintf' output between 1 and 256 bytes into a destination of size 64
122 | snprintf(sysfs_dev->sysfs_name, sizeof sysfs_dev->sysfs_name,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
123 | "%s", dent->d_name);
| ~~~~~~~~~~~~~~~~~~~
CC src/libibverbs_la-marshall.lo
CC src/libibverbs_la-memory.lo
CC src/libibverbs_la-sysfs.lo
CC src/libibverbs_la-verbs.lo
src/verbs.c: In function 'ibv_find_gid_index':
src/verbs.c:600:57: warning: '%d' directive output may be truncated writing between 1 and 10 bytes into a region of size between 6 and 8 [-Wformat-truncation=]
600 | snprintf(name, sizeof(name), "ports/%d/gid_attrs/types/%d",
| ^~
src/verbs.c:600:31: note: directive argument in the range [0, 2147483647]
600 | snprintf(name, sizeof(name), "ports/%d/gid_attrs/types/%d",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/verbs.c:600:2: note: 'snprintf' output between 26 and 37 bytes into a destination of size 32
600 | snprintf(name, sizeof(name), "ports/%d/gid_attrs/types/%d",
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
601 | port_num, index);
| ~~~~~~~~~~~~~~~~
CC src/libibverbs_la-enum_strs.lo
CC src/libibverbs_la-neigh.lo
src/neigh.c: In function 'get_route_cb':
src/neigh.c:338:8: warning: 'sock_fd' may be used uninitialized in this function [-Wmaybe-uninitialized]
338 | err = try_send_to(sock_fd, buff, sizeof(buff), &addr_dst);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/neigh.c:303:6: note: 'sock_fd' was declared here
303 | int sock_fd;
| ^~~~~~~
CCLD src/libibverbs.la
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/../lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: cannot find -lnl-route-3
/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/../lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: cannot find -lnl-3
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:720: src/libibverbs.la] Error 1
make[1]: Leaving directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
make: *** [Makefile:554: all] Error 2
3、报错: cannot find -lnl-route-3
经查阅资料，了解到该问题是由于缺少libnl套件导致的错误。libnl套件下载链接：http://www.infradead.org/~tgr/libnl/
libnl 套件是一组库，为基于 netlink 协议的 Linux 内核接口提供 API。Netlink 是一种主要在内核和用户空间进程之间的 IPC 机制。它被设计为 ioctl 的更灵活的继承者，主要提供与网络相关的内核配置和监控接口。
该接口由这几个子库构成:
<1> libnl: 核心库实现了使用 netlink 协议所需的基础知识，例如套接字处理、消息构建和解析以及数据的发送和接收。这个库保持小而简约。该套件的其他库依赖于该库。
<2> libnl-route: NETLINK_ROUTE 系列配置接口的 API，包括网络接口、路由、地址、邻居和流量控制。
<3> libnl-genl: 通用 netlink 协议的 API，netlink 协议的扩展版本。
<4> libnl-nf: 基于 netlink 的 netfilter 配置和监控接口的 API（conntrack、日志、队列）
4、移植libnl套件
#创建安装目录
imaginemiracle:dependency$ mkdir install
imaginemiracle:dependency$ ls
libnl-3.2.25.tar.gz
#解压源码，进入源码目录
imaginemiracle:dependency$ tar -xzvf libnl-3.2.25.tar.gz
imaginemiracle:dependency$ cd libnl-3.2.25/
#配置libnl套件生成Makefile文件
imaginemiracle:libnl-3.2.25$ ./configure --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/dependency/install/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu
#编译并安装
imaginemiracle:libnl-3.2.25$ make; make install
安装成功。
imaginemiracle:install$ ls
etc include lib sbin share
imaginemiracle:install$ tree
.
├── etc
│ └── libnl
│ ├── classid
│ └── pktloc
├── include
│ └── libnl3
│ └── netlink
│ ├── addr.h
│ ├── attr.h
│ ├── cache-api.h
│ ├── cache.h
│ ├── cli
│ │ ├── addr.h
│ │ ├── class.h
│ │ ├── cls.h
│ │ ├── ct.h
│ │ ├── exp.h
│ │ ├── link.h
│ │ ├── neigh.h
│ │ ├── qdisc.h
│ │ ├── route.h
│ │ ├── rule.h
│ │ ├── tc.h
│ │ └── utils.h
│ ├── data.h
│ ├── errno.h
│ ├── fib_lookup
│ │ ├── lookup.h
│ │ └── request.h
│ ├── genl
│ │ ├── ctrl.h
│ │ ├── family.h
│ │ ├── genl.h
│ │ └── mngt.h
│ ├── handlers.h
│ ├── hash.h
│ ├── hashtable.h
│ ├── idiag
│ │ ├── idiagnl.h
│ │ ├── meminfo.h
│ │ ├── msg.h
│ │ ├── req.h
│ │ └── vegasinfo.h
│ ├── list.h
│ ├── msg.h
│ ├── netfilter
│ │ ├── ct.h
│ │ ├── exp.h
│ │ ├── log.h
│ │ ├── log_msg.h
│ │ ├── netfilter.h
│ │ ├── nfnl.h
│ │ ├── queue.h
│ │ └── queue_msg.h
│ ├── netlink-compat.h
│ ├── netlink.h
│ ├── netlink-kernel.h
│ ├── object-api.h
│ ├── object.h
│ ├── route
│ │ ├── act
│ │ │ └── mirred.h
│ │ ├── action.h
│ │ ├── addr.h
│ │ ├── class.h
│ │ ├── classifier.h
│ │ ├── cls
│ │ │ ├── basic.h
│ │ │ ├── cgroup.h
│ │ │ ├── ematch
│ │ │ │ ├── cmp.h
│ │ │ │ ├── meta.h
│ │ │ │ ├── nbyte.h
│ │ │ │ └── text.h
│ │ │ ├── ematch.h
│ │ │ ├── fw.h
│ │ │ ├── police.h
│ │ │ └── u32.h
│ │ ├── link
│ │ │ ├── api.h
│ │ │ ├── bonding.h
│ │ │ ├── bridge.h
│ │ │ ├── can.h
│ │ │ ├── inet.h
│ │ │ ├── info-api.h
│ │ │ ├── ip6tnl.h
│ │ │ ├── ipgre.h
│ │ │ ├── ipip.h
│ │ │ ├── ipvti.h
│ │ │ ├── macvlan.h
│ │ │ ├── sit.h
│ │ │ ├── veth.h
│ │ │ ├── vlan.h
│ │ │ └── vxlan.h
│ │ ├── link.h
│ │ ├── neighbour.h
│ │ ├── neightbl.h
│ │ ├── nexthop.h
│ │ ├── pktloc.h
│ │ ├── qdisc
│ │ │ ├── cbq.h
│ │ │ ├── dsmark.h
│ │ │ ├── fifo.h
│ │ │ ├── fq_codel.h
│ │ │ ├── htb.h
│ │ │ ├── netem.h
│ │ │ ├── plug.h
│ │ │ ├── prio.h
│ │ │ ├── red.h
│ │ │ ├── sfq.h
│ │ │ └── tbf.h
│ │ ├── qdisc.h
│ │ ├── route.h
│ │ ├── rtnl.h
│ │ ├── rule.h
│ │ ├── tc-api.h
│ │ └── tc.h
│ ├── socket.h
│ ├── types.h
│ ├── utils.h
│ └── version.h
├── lib
│ ├── libnl
│ │ └── cli
│ │ ├── cls
│ │ │ ├── basic.a
│ │ │ ├── basic.la
│ │ │ ├── basic.so
│ │ │ ├── cgroup.a
│ │ │ ├── cgroup.la
│ │ │ └── cgroup.so
│ │ └── qdisc
│ │ ├── bfifo.a
│ │ ├── bfifo.la
│ │ ├── bfifo.so
│ │ ├── blackhole.a
│ │ ├── blackhole.la
│ │ ├── blackhole.so
│ │ ├── fq_codel.a
│ │ ├── fq_codel.la
│ │ ├── fq_codel.so
│ │ ├── htb.a
│ │ ├── htb.la
│ │ ├── htb.so
│ │ ├── ingress.a
│ │ ├── ingress.la
│ │ ├── ingress.so
│ │ ├── pfifo.a
│ │ ├── pfifo.la
│ │ ├── pfifo.so
│ │ ├── plug.a
│ │ ├── plug.la
│ │ └── plug.so
│ ├── libnl-3.a
│ ├── libnl-3.la
│ ├── libnl-3.so -> libnl-3.so.200.20.0
│ ├── libnl-3.so.200 -> libnl-3.so.200.20.0
│ ├── libnl-3.so.200.20.0
│ ├── libnl-cli-3.a
│ ├── libnl-cli-3.la
│ ├── libnl-cli-3.so -> libnl-cli-3.so.200.20.0
│ ├── libnl-cli-3.so.200 -> libnl-cli-3.so.200.20.0
│ ├── libnl-cli-3.so.200.20.0
│ ├── libnl-genl-3.a
│ ├── libnl-genl-3.la
│ ├── libnl-genl-3.so -> libnl-genl-3.so.200.20.0
│ ├── libnl-genl-3.so.200 -> libnl-genl-3.so.200.20.0
│ ├── libnl-genl-3.so.200.20.0
│ ├── libnl-idiag-3.a
│ ├── libnl-idiag-3.la
│ ├── libnl-idiag-3.so -> libnl-idiag-3.so.200.20.0
│ ├── libnl-idiag-3.so.200 -> libnl-idiag-3.so.200.20.0
│ ├── libnl-idiag-3.so.200.20.0
│ ├── libnl-nf-3.a
│ ├── libnl-nf-3.la
│ ├── libnl-nf-3.so -> libnl-nf-3.so.200.20.0
│ ├── libnl-nf-3.so.200 -> libnl-nf-3.so.200.20.0
│ ├── libnl-nf-3.so.200.20.0
│ ├── libnl-route-3.a
│ ├── libnl-route-3.la
│ ├── libnl-route-3.so -> libnl-route-3.so.200.20.0
│ ├── libnl-route-3.so.200 -> libnl-route-3.so.200.20.0
│ ├── libnl-route-3.so.200.20.0
│ └── pkgconfig
│ ├── libnl-3.0.pc
│ ├── libnl-cli-3.0.pc
│ ├── libnl-genl-3.0.pc
│ ├── libnl-nf-3.0.pc
│ └── libnl-route-3.0.pc
├── sbin
│ ├── genl-ctrl-list
│ ├── nl-class-add
│ ├── nl-class-delete
│ ├── nl-classid-lookup
│ ├── nl-class-list
│ ├── nl-cls-add
│ ├── nl-cls-delete
│ ├── nl-cls-list
│ ├── nl-link-list
│ ├── nl-pktloc-lookup
│ ├── nl-qdisc-add
│ ├── nl-qdisc-delete
│ └── nl-qdisc-list
└── share
└── man
└── man8
├── genl-ctrl-list.8
├── nl-classid-lookup.8
├── nl-pktloc-lookup.8
├── nl-qdisc-add.8
├── nl-qdisc-delete.8
└── nl-qdisc-list.8
26 directories, 186 files
将安装目录下的文件迁移到riscv64-unknown-linux-gnu的对应目录下
imaginemiracle:~$ cp -rf libnl/install/etc/* riscv64-unknown-linux-gnu/sysroot/etc/
imaginemiracle:~$ cp -rf libnl/install/include/* riscv64-unknown-linux-gnu/include/
imaginemiracle:~$ cp -rf libnl/install/lib/* riscv64-unknown-linux-gnu/lib/
imaginemiracle:~$ cp -rf libnl/install/lib/* riscv64-unknown-linux-gnu/sysroot/lib/
imaginemiracle:~$ cp -rf libnl/install/sbin/* riscv64-unknown-linux-gnu/sysroot/sbin/
imaginemiracle:~$ cp -rf libnl/install/share/man/* riscv64-unknown-linux-gnu/sysroot/share/man/
5、再次编译libibverbs源码
重新配置、编译并安装。
imaginemiracle:libibverbs$ ./configure --prefix=/home/imaginemiracle/Ubuntu_Download/RDMA/build/ --build=x86_64-linux-gnu --host=riscv64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for riscv64-unknown-linux-gnu-strip... riscv64-unknown-linux-gnu-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for riscv64-unknown-linux-gnu-gcc... riscv64-unknown-linux-gnu-gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether riscv64-unknown-linux-gnu-gcc accepts -g... yes
checking for riscv64-unknown-linux-gnu-gcc option to accept ISO C89... none needed
checking whether riscv64-unknown-linux-gnu-gcc understands -c and -o together... yes
checking whether make supports the include directive... yes (GNU style)
checking dependency style of riscv64-unknown-linux-gnu-gcc... gcc3
checking how to run the C preprocessor... riscv64-unknown-linux-gnu-gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking minix/config.h usability... no
checking minix/config.h presence... no
checking for minix/config.h... no
checking whether it is safe to define __EXTENSIONS__... yes
checking whether ln -s works... yes
checking build system type... x86_64-pc-linux-gnu
checking host system type... riscv64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by riscv64-unknown-linux-gnu-gcc... /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld
checking if the linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B
checking the name lister (/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to riscv64-unknown-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld option to reload object files... -r
checking for riscv64-unknown-linux-gnu-objdump... riscv64-unknown-linux-gnu-objdump
checking how to recognize dependent libraries... pass_all
checking for riscv64-unknown-linux-gnu-dlltool... no
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for riscv64-unknown-linux-gnu-ar... riscv64-unknown-linux-gnu-ar
checking for archiver @FILE support... @
checking for riscv64-unknown-linux-gnu-strip... (cached) riscv64-unknown-linux-gnu-strip
checking for riscv64-unknown-linux-gnu-ranlib... riscv64-unknown-linux-gnu-ranlib
checking command to parse /home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin/riscv64-unknown-linux-gnu-nm -B output from riscv64-unknown-linux-gnu-gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for riscv64-unknown-linux-gnu-mt... no
checking for mt... mt
configure: WARNING: using cross tools not prefixed with host triplet
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if riscv64-unknown-linux-gnu-gcc supports -fno-rtti -fno-exceptions... no
checking for riscv64-unknown-linux-gnu-gcc option to produce PIC... -fPIC -DPIC
checking if riscv64-unknown-linux-gnu-gcc PIC flag -fPIC -DPIC works... yes
checking if riscv64-unknown-linux-gnu-gcc static flag -static works... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... yes
checking if riscv64-unknown-linux-gnu-gcc supports -c -o file.o... (cached) yes
checking whether the riscv64-unknown-linux-gnu-gcc linker (/media/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/riscv64-unknown-linux-gnu/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for riscv64-unknown-linux-gnu-pkg-config... no
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for libnl-3.0... yes
checking for libnl-route-3.0... yes
checking for dlsym in -ldl... yes
checking for pthread_mutex_init in -lpthread... yes
checking for ANSI C header files... (cached) yes
checking valgrind/memcheck.h usability... no
checking valgrind/memcheck.h presence... no
checking for valgrind/memcheck.h... no
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking sys/socket.h usability... yes
checking sys/socket.h presence... yes
checking for sys/socket.h... yes
checking for an ANSI C-conforming const... yes
checking whether O_CLOEXEC is declared... yes
checking whether SOCK_CLOEXEC is declared... yes
checking for close on exec modifier for fopen()... e
checking whether ld accepts --version-script... yes
checking for .symver assembler support... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libibverbs.spec
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool command
执行make编译源码。
imaginemiracle:libibverbs$ make
make all-am
make[1]: Entering directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
CCLD src/libibverbs.la
CCLD examples/ibv_devices
CC examples/devinfo.o
In file included from examples/devinfo.c:50:
./include/infiniband/arch.h:135:2: warning: #warning No architecture specific defines found. Using generic implementation. [-Wcpp]
135 | #warning No architecture specific defines found. Using generic implementation.
| ^~~~~~~
CCLD examples/ibv_devinfo
CC examples/asyncwatch.o
CCLD examples/ibv_asyncwatch
CC examples/rc_pingpong.o
CC examples/pingpong.o
CCLD examples/ibv_rc_pingpong
CC examples/uc_pingpong.o
CCLD examples/ibv_uc_pingpong
CC examples/ud_pingpong.o
CCLD examples/ibv_ud_pingpong
CC examples/srq_pingpong.o
CCLD examples/ibv_srq_pingpong
CC examples/xsrq_pingpong.o
CCLD examples/ibv_xsrq_pingpong
CC examples/cc_pingpong.o
examples/cc_pingpong.c: In function 'parse_command_line_args':
examples/cc_pingpong.c:775:4: warning: 'strncpy' specified bound 128 equals destination size [-Wstringop-truncation]
775 | strncpy(app_params->ib_devname, optarg, sizeof(app_params->ib_devname));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CCLD examples/ibv_cc_pingpong
make[1]: Leaving directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
执行make install安装libibverbs
imaginemiracle:libibverbs$ make install
make[1]: Entering directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
/usr/bin/mkdir -p '/home/imaginemiracle/Ubuntu_Download/RDMA/build/lib'
/bin/bash ./libtool --mode=install /usr/bin/install -c src/libibverbs.la '/home/imaginemiracle/Ubuntu_Download/RDMA/build/lib'
libtool: install: /usr/bin/install -c src/.libs/libibverbs.so.1.0.0 /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib/libibverbs.so.1.0.0
libtool: install: (cd /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib && { ln -s -f libibverbs.so.1.0.0 libibverbs.so.1 || { rm -f libibverbs.so.1 && ln -s libibverbs.so.1.0.0 libibverbs.so.1; }; })
libtool: install: (cd /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib && { ln -s -f libibverbs.so.1.0.0 libibverbs.so || { rm -f libibverbs.so && ln -s libibverbs.so.1.0.0 libibverbs.so; }; })
libtool: install: /usr/bin/install -c src/.libs/libibverbs.lai /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib/libibverbs.la
libtool: install: /usr/bin/install -c src/.libs/libibverbs.a /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib/libibverbs.a
libtool: install: chmod 644 /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib/libibverbs.a
libtool: install: riscv64-unknown-linux-gnu-ranlib /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib/libibverbs.a
libtool: finish: PATH="/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/riscv-gcc/tools/riscv64-linux-multilib/bin:/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/freedom-u-sdk/toolchain_bk/bin:/home/imaginemiracle/Disk_D/Linux_Workspace/riscv-project/rvlinux-u_IM/toolchain/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin" ldconfig -n /home/imaginemiracle/Ubuntu_Download/RDMA/build/lib
----------------------------------------------------------------------
Libraries have been installed in:
/home/imaginemiracle/Ubuntu_Download/RDMA/build/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the 'LD_RUN_PATH' environment variable
during linking
- use the '-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to '/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
/usr/bin/mkdir -p '/home/imaginemiracle/Ubuntu_Download/RDMA/build/bin'
/bin/bash ./libtool --mode=install /usr/bin/install -c examples/ibv_devices examples/ibv_devinfo examples/ibv_asyncwatch examples/ibv_rc_pingpong examples/ibv_uc_pingpong examples/ibv_ud_pingpong examples/ibv_srq_pingpong examples/ibv_xsrq_pingpong examples/ibv_cc_pingpong '/home/imaginemiracle/Ubuntu_Download/RDMA/build/bin'
libtool: install: /usr/bin/install -c examples/.libs/ibv_devices /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_devices
libtool: install: /usr/bin/install -c examples/.libs/ibv_devinfo /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_devinfo
libtool: install: /usr/bin/install -c examples/.libs/ibv_asyncwatch /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_asyncwatch
libtool: install: /usr/bin/install -c examples/.libs/ibv_rc_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_rc_pingpong
libtool: install: /usr/bin/install -c examples/.libs/ibv_uc_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_uc_pingpong
libtool: install: /usr/bin/install -c examples/.libs/ibv_ud_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_ud_pingpong
libtool: install: /usr/bin/install -c examples/.libs/ibv_srq_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_srq_pingpong
libtool: install: /usr/bin/install -c examples/.libs/ibv_xsrq_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_xsrq_pingpong
libtool: install: /usr/bin/install -c examples/.libs/ibv_cc_pingpong /home/imaginemiracle/Ubuntu_Download/RDMA/build/bin/ibv_cc_pingpong
/usr/bin/mkdir -p '/home/imaginemiracle/Ubuntu_Download/RDMA/build/include/infiniband'
/usr/bin/install -c -m 644 include/infiniband/arch.h include/infiniband/driver.h include/infiniband/kern-abi.h include/infiniband/opcode.h include/infiniband/verbs.h include/infiniband/sa-kern-abi.h include/infiniband/sa.h include/infiniband/marshall.h include/infiniband/compiler.h '/home/imaginemiracle/Ubuntu_Download/RDMA/build/include/infiniband'
/usr/bin/mkdir -p '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man1'
/usr/bin/install -c -m 644 man/ibv_asyncwatch.1 man/ibv_devices.1 man/ibv_devinfo.1 man/ibv_rc_pingpong.1 man/ibv_uc_pingpong.1 man/ibv_ud_pingpong.1 man/ibv_srq_pingpong.1 man/ibv_cc_pingpong.1 '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man1'
/usr/bin/mkdir -p '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man3'
/usr/bin/install -c -m 644 man/ibv_alloc_pd.3 man/ibv_attach_mcast.3 man/ibv_create_ah.3 man/ibv_create_ah_from_wc.3 man/ibv_create_comp_channel.3 man/ibv_create_cq.3 man/ibv_create_qp.3 man/ibv_create_srq.3 man/ibv_event_type_str.3 man/ibv_fork_init.3 man/ibv_get_async_event.3 man/ibv_get_cq_event.3 man/ibv_get_device_guid.3 man/ibv_get_device_list.3 man/ibv_get_device_name.3 man/ibv_modify_qp.3 man/ibv_modify_srq.3 man/ibv_open_device.3 man/ibv_poll_cq.3 man/ibv_post_recv.3 man/ibv_post_send.3 man/ibv_post_srq_recv.3 man/ibv_query_device.3 man/ibv_query_gid.3 man/ibv_query_pkey.3 man/ibv_query_port.3 man/ibv_query_qp.3 man/ibv_query_srq.3 man/ibv_rate_to_mult.3 man/ibv_reg_mr.3 man/ibv_req_notify_cq.3 man/ibv_resize_cq.3 man/ibv_rate_to_mbps.3 man/ibv_create_qp_ex.3 man/ibv_create_srq_ex.3 man/ibv_open_xrcd.3 man/ibv_get_srq_num.3 man/ibv_open_qp.3 man/ibv_query_device_ex.3 man/ibv_create_flow.3 '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man3'
/usr/bin/install -c -m 644 man/ibv_alloc_mw.3 man/ibv_bind_mw.3 man/ibv_inc_rkey.3 man/ibv_poll_cq_ex.3 man/ibv_create_cq_ex.3 man/ibv_query_rt_values_ex.3 '/home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man3'
make install-data-hook
make[2]: Entering directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
cd /home/imaginemiracle/Ubuntu_Download/RDMA/build/share/man/man3 && \
rm -f ibv_ack_async_event.3 && \
rm -f ibv_ack_cq_events.3 && \
rm -f ibv_close_device.3 && \
rm -f ibv_dealloc_pd.3 && \
rm -f ibv_dereg_mr.3 && \
rm -f ibv_destroy_ah.3 && \
rm -f ibv_destroy_comp_channel.3 && \
rm -f ibv_destroy_cq.3 && \
rm -f ibv_destroy_qp.3 && \
rm -f ibv_destroy_srq.3 && \
rm -f ibv_detach_mcast.3 && \
rm -f ibv_free_device_list.3 && \
rm -f ibv_init_ah_from_wc.3 && \
rm -f mult_to_ibv_rate.3 && \
rm -f ibv_node_type_str.3 && \
rm -f ibv_port_state_str.3 && \
rm -f mbps_to_ibv_rate.3 && \
rm -f ibv_close_xrcd.3 && \
rm -f ibv_dealloc_mw.3 && \
rm -f ibv_destroy_flow.3 && \
ln -s ibv_get_async_event.3 ibv_ack_async_event.3 && \
ln -s ibv_get_cq_event.3 ibv_ack_cq_events.3 && \
ln -s ibv_open_device.3 ibv_close_device.3 && \
ln -s ibv_alloc_pd.3 ibv_dealloc_pd.3 && \
ln -s ibv_reg_mr.3 ibv_dereg_mr.3 && \
ln -s ibv_create_ah.3 ibv_destroy_ah.3 && \
ln -s ibv_create_comp_channel.3 ibv_destroy_comp_channel.3 && \
ln -s ibv_create_cq.3 ibv_destroy_cq.3 && \
ln -s ibv_create_qp.3 ibv_destroy_qp.3 && \
ln -s ibv_create_srq.3 ibv_destroy_srq.3 && \
ln -s ibv_attach_mcast.3 ibv_detach_mcast.3 && \
ln -s ibv_get_device_list.3 ibv_free_device_list.3 && \
ln -s ibv_create_ah_from_wc.3 ibv_init_ah_from_wc.3 && \
ln -s ibv_rate_to_mult.3 mult_to_ibv_rate.3 && \
ln -s ibv_event_type_str.3 ibv_node_type_str.3 && \
ln -s ibv_event_type_str.3 ibv_port_state_str.3 && \
ln -s ibv_rate_to_mbps.3 mbps_to_ibv_rate.3 && \
ln -s ibv_open_xrcd.3 ibv_close_xrcd.3 && \
ln -s ibv_alloc_mw.3 ibv_dealloc_mw.3 && \
ln -s ibv_create_flow.3 ibv_destroy_flow.3
make[2]: Leaving directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
make[1]: Leaving directory '/media/imaginemiracle/Disk_D/Download/Ubuntu_Download/RDMA/libibverbs'
安装成功。
imaginemiracle:libibverbs-riscv_install$ ls
bin include lib share
imaginemiracle:libibverbs-riscv_install$ tree
.
├── bin
│ ├── ibv_asyncwatch
│ ├── ibv_cc_pingpong
│ ├── ibv_devices
│ ├── ibv_devinfo
│ ├── ibv_rc_pingpong
│ ├── ibv_srq_pingpong
│ ├── ibv_uc_pingpong
│ ├── ibv_ud_pingpong
│ └── ibv_xsrq_pingpong
├── include
│ └── infiniband
│ ├── arch.h
│ ├── compiler.h
│ ├── driver.h
│ ├── kern-abi.h
│ ├── marshall.h
│ ├── opcode.h
│ ├── sa.h
│ ├── sa-kern-abi.h
│ └── verbs.h
├── lib
│ ├── libibverbs.a
│ ├── libibverbs.la
│ ├── libibverbs.so -> libibverbs.so.1.0.0
│ ├── libibverbs.so.1 -> libibverbs.so.1.0.0
│ └── libibverbs.so.1.0.0
└── share
└── man
├── man1
│ ├── ibv_asyncwatch.1
│ ├── ibv_cc_pingpong.1
│ ├── ibv_devices.1
│ ├── ibv_devinfo.1
│ ├── ibv_rc_pingpong.1
│ ├── ibv_srq_pingpong.1
│ ├── ibv_uc_pingpong.1
│ └── ibv_ud_pingpong.1
└── man3
├── ibv_ack_async_event.3 -> ibv_get_async_event.3
├── ibv_ack_cq_events.3 -> ibv_get_cq_event.3
├── ibv_alloc_mw.3
├── ibv_alloc_pd.3
├── ibv_attach_mcast.3
├── ibv_bind_mw.3
├── ibv_close_device.3 -> ibv_open_device.3
├── ibv_close_xrcd.3 -> ibv_open_xrcd.3
├── ibv_create_ah.3
├── ibv_create_ah_from_wc.3
├── ibv_create_comp_channel.3
├── ibv_create_cq.3
├── ibv_create_cq_ex.3
├── ibv_create_flow.3
├── ibv_create_qp.3
├── ibv_create_qp_ex.3
├── ibv_create_srq.3
├── ibv_create_srq_ex.3
├── ibv_dealloc_mw.3 -> ibv_alloc_mw.3
├── ibv_dealloc_pd.3 -> ibv_alloc_pd.3
├── ibv_dereg_mr.3 -> ibv_reg_mr.3
├── ibv_destroy_ah.3 -> ibv_create_ah.3
├── ibv_destroy_comp_channel.3 -> ibv_create_comp_channel.3
├── ibv_destroy_cq.3 -> ibv_create_cq.3
├── ibv_destroy_flow.3 -> ibv_create_flow.3
├── ibv_destroy_qp.3 -> ibv_create_qp.3
├── ibv_destroy_srq.3 -> ibv_create_srq.3
├── ibv_detach_mcast.3 -> ibv_attach_mcast.3
├── ibv_event_type_str.3
├── ibv_fork_init.3
├── ibv_free_device_list.3 -> ibv_get_device_list.3
├── ibv_get_async_event.3
├── ibv_get_cq_event.3
├── ibv_get_device_guid.3
├── ibv_get_device_list.3
├── ibv_get_device_name.3
├── ibv_get_srq_num.3
├── ibv_inc_rkey.3
├── ibv_init_ah_from_wc.3 -> ibv_create_ah_from_wc.3
├── ibv_modify_qp.3
├── ibv_modify_srq.3
├── ibv_node_type_str.3 -> ibv_event_type_str.3
├── ibv_open_device.3
├── ibv_open_qp.3
├── ibv_open_xrcd.3
├── ibv_poll_cq.3
├── ibv_poll_cq_ex.3
├── ibv_port_state_str.3 -> ibv_event_type_str.3
├── ibv_post_recv.3
├── ibv_post_send.3
├── ibv_post_srq_recv.3
├── ibv_query_device.3
├── ibv_query_device_ex.3
├── ibv_query_gid.3
├── ibv_query_pkey.3
├── ibv_query_port.3
├── ibv_query_qp.3
├── ibv_query_rt_values_ex.3
├── ibv_query_srq.3
├── ibv_rate_to_mbps.3
├── ibv_rate_to_mult.3
├── ibv_reg_mr.3
├── ibv_req_notify_cq.3
├── ibv_resize_cq.3
├── mbps_to_ibv_rate.3 -> ibv_rate_to_mbps.3
└── mult_to_ibv_rate.3 -> ibv_rate_to_mult.3
8 directories, 97 files
6、迁移libibverbs库的相关文件
安装完成后，将目标目录的文件拷贝到交叉编译工具链的相应目录中即可。(注: 若使用libibverbs时，出现缺少<infiniband/driver.h>文件时，则需要将libibverbs源码下的include/infiniband/driver.h拷贝到交叉编译工具链的相应目录中即可)
imaginemiracle:~$ cp -rf libibverbs-riscv_install/include/* riscv64-unknown-linux-gnu/include/
imaginemiracle:~$ cp -rf libibverbs-riscv_install/lib/* riscv64-unknown-linux-gnu/lib
imaginemiracle:~$ cp -rf libibverbs-riscv_install/lib/* riscv64-unknown-linux-gnu/sysroot/lib/
GitCode 天启AI是一款由 GitCode 团队打造的智能助手，基于先进的LLM（大语言模型）与多智能体 Agent 技术构建，致力于为用户提供高效、智能、多模态的创作与开发支持。它不仅支持自然语言对话，还具备处理文件、生成 PPT、撰写分析报告、开发 Web 应用等多项能力，真正做到“一句话，让 Al帮你完成复杂任务”。
更多推荐
1
0- 0
已为社区贡献3条内容
所有评论(0)