Блог

Я пытаюсь скомпилировать пакет duckdb (0.3.1 из CRAN) из исходного tar-файла в Windows 10, используя набор инструментов R Tools gt;4.0 (из оболочки MSYS2 terminal x64 BASH).

Компиляция необходима, потому что наш брандмауэр блокирует загрузку готовых двоичных файлов из CRAN.

Я сначала проверил посылку с:

 R CMD check -l . --no-stop-on-test-error --install-args="--no-test-load" duckdb_0.3.1-1.tar.gz  

Это --no-test-load потому, что у нас есть белый список DLL, поэтому мне нужно сначала создать библиотеки DLL, а затем внести их в белый список, прежде чем им будет разрешено загружаться.

Это дает ожидаемый результат: пакет может быть скомпилирован (как для архитектуры i386, так и для архитектуры x64) и установлен, но при check попытке загрузить пакет возникают ошибки:

 * using log directory 'P:/tmp/duckdb.Rcheck' * using R version 4.0.2 (2020-06-22) * using platform: x86_64-w64-mingw32 (64-bit) * using session charset: ISO8859-1 * using option '--no-stop-on-test-error' * checking for file 'duckdb/DESCRIPTION' ... OK * this is package 'duckdb' version '0.3.1-1' * package encoding: UTF-8 * checking package namespace information ... OK * checking package dependencies ... OK * checking if this is a source package ... OK * checking if there is a namespace ... OK * checking for executable files ... OK * checking for hidden files and directories ... OK * checking for portable file names ... OK * checking whether package 'duckdb' can be installed ... OK * checking installed package size ... NOTE  installed size is 32.4Mb  sub-directories of 1Mb or more:  libs 32.1Mb * checking package directory ... OK * checking DESCRIPTION meta-information ... OK * checking top-level files ... OK * checking for left-over files ... OK * checking index information ... OK * checking package subdirectories ... OK * checking R files for non-ASCII characters ... OK * checking R files for syntax errors ... OK * loading checks for arch 'i386' ** checking whether the package can be loaded ... ERROR Loading this package had a fatal error status code 1 Loading log: Loading required package: DBI Error: package or namespace load failed for 'duckdb' in inDL(x, as.logical(local), as.logical(now), ...):  unable to load shared object 'P:/tmp/duckdb/libs/i386/duckdb.dll':  LoadLibrary failure: Access is denied. Execution halted ** DONE  Status: 1 ERROR, 1 NOTE See  'P:/tmp/duckdb.Rcheck/00check.log' for details.  

В журнале установки есть несколько предупреждений, связанных со сравнением подписанных и неподписанных значений, но ничего фатального. Вот конец i386 раздела журнала установки, демонстрирующего отсутствие ошибок при ld создании окончательного duckdb.dll :

 lt;secret-pathgt;/Rtools40/mingw32/bin/g   -shared -s -static-libgcc -o duckdb.dll tmp.def rapi.o database.o connection.o statement.o register.o scan.o utils.o altrep.o types.o duckdb/extension/parquet/column_reader.o duckdb/extension/parquet/parquet-extension.o duckdb/extension/parquet/parquet_metadata.o duckdb/extension/parquet/parquet_reader.o duckdb/extension/parquet/parquet_statistics.o duckdb/extension/parquet/parquet_timestamp.o duckdb/extension/parquet/parquet_writer.o duckdb/amalgamation-1.o duckdb/amalgamation-2.o duckdb/amalgamation-3.o duckdb/amalgamation-4.o duckdb/amalgamation-5.o duckdb/amalgamation-6.o duckdb/amalgamation-7.o duckdb/third_party/fmt/format.o duckdb/third_party/hyperloglog/hyperloglog.o duckdb/third_party/hyperloglog/sds.o duckdb/third_party/libpg_query/pg_functions.o duckdb/third_party/libpg_query/postgres_parser.o duckdb/third_party/libpg_query/src_backend_nodes_list.o duckdb/third_party/libpg_query/src_backend_nodes_makefuncs.o duckdb/third_party/libpg_query/src_backend_nodes_value.o duckdb/third_party/libpg_query/src_backend_parser_gram.o duckdb/third_party/libpg_query/src_backend_parser_parser.o duckdb/third_party/libpg_query/src_backend_parser_scan.o duckdb/third_party/libpg_query/src_backend_parser_scansup.o duckdb/third_party/libpg_query/src_common_keywords.o duckdb/third_party/miniz/miniz.o duckdb/third_party/parquet/parquet_constants.o duckdb/third_party/parquet/parquet_types.o duckdb/third_party/re2/re2/bitstate.o duckdb/third_party/re2/re2/compile.o duckdb/third_party/re2/re2/dfa.o duckdb/third_party/re2/re2/filtered_re2.o duckdb/third_party/re2/re2/mimics_pcre.o duckdb/third_party/re2/re2/nfa.o duckdb/third_party/re2/re2/onepass.o duckdb/third_party/re2/re2/parse.o duckdb/third_party/re2/re2/perl_groups.o duckdb/third_party/re2/re2/prefilter.o duckdb/third_party/re2/re2/prefilter_tree.o duckdb/third_party/re2/re2/prog.o duckdb/third_party/re2/re2/re2.o duckdb/third_party/re2/re2/regexp.o duckdb/third_party/re2/re2/set.o duckdb/third_party/re2/re2/simplify.o duckdb/third_party/re2/re2/stringpiece.o duckdb/third_party/re2/re2/tostring.o duckdb/third_party/re2/re2/unicode_casefold.o duckdb/third_party/re2/re2/unicode_groups.o duckdb/third_party/re2/util/rune.o duckdb/third_party/re2/util/strutil.o duckdb/third_party/snappy/snappy-sinksource.o duckdb/third_party/snappy/snappy.o duckdb/third_party/thrift/thrift/protocol/TProtocol.o duckdb/third_party/thrift/thrift/transport/TBufferTransports.o duckdb/third_party/thrift/thrift/transport/TTransportException.o duckdb/third_party/utf8proc/utf8proc.o duckdb/third_party/utf8proc/utf8proc_wrapper.o duckdb/third_party/zstd/common/entropy_common.o duckdb/third_party/zstd/common/error_private.o duckdb/third_party/zstd/common/fse_decompress.o duckdb/third_party/zstd/common/xxhash.o duckdb/third_party/zstd/common/zstd_common.o duckdb/third_party/zstd/compress/fse_compress.o duckdb/third_party/zstd/compress/hist.o duckdb/third_party/zstd/compress/huf_compress.o duckdb/third_party/zstd/compress/zstd_compress.o duckdb/third_party/zstd/compress/zstd_compress_literals.o duckdb/third_party/zstd/compress/zstd_compress_sequences.o duckdb/third_party/zstd/compress/zstd_compress_superblock.o duckdb/third_party/zstd/compress/zstd_double_fast.o duckdb/third_party/zstd/compress/zstd_fast.o duckdb/third_party/zstd/compress/zstd_lazy.o duckdb/third_party/zstd/compress/zstd_ldm.o duckdb/third_party/zstd/compress/zstd_opt.o duckdb/third_party/zstd/decompress/huf_decompress.o duckdb/third_party/zstd/decompress/zstd_ddict.o duckdb/third_party/zstd/decompress/zstd_decompress.o duckdb/third_party/zstd/decompress/zstd_decompress_block.o duckdb/amalgamation-8.o -LC:lt;secret-pathgt;/R-40~1.2/bin/i386 -lR There were 50 or more warnings (use warnings() to see the first 50)  installing to P:/tmp/00LOCK-duckdb/00new/duckdb/libs/i386    *** arch - x64 lt;snip similar log for x64 architecture...gt;  

At the end of this I have a fully-compiled and installed package in the «duckdb» directory — I just can’t load it into my R session due to DLL whitelisting. So far so good.

Then I try to install the package using R CMD INSTALL --no-test-load -l . duckdb_0.3.1-1.tar.gz (and variations). Unfortunately the build fails. The log seems identical up to the point where it tries to link the .o files into the duckdb.dll , at which point:

 lt;secret-pathgt;/Rtools40/mingw32/bin/g   -shared -s -static-libgcc -o duckdb.dll tmp.def rapi.o database.o connection.o statement.o register.o scan.o utils.o altrep.o types.o duckdb/extension/parquet/column_reader.o duckdb/extension/parquet/parquet-extension.o duckdb/extension/parquet/parquet_metadata.o duckdb/extension/parquet/parquet_reader.o duckdb/extension/parquet/parquet_statistics.o duckdb/extension/parquet/parquet_timestamp.o duckdb/extension/parquet/parquet_writer.o duckdb/amalgamation-1.o duckdb/amalgamation-2.o duckdb/amalgamation-3.o duckdb/amalgamation-4.o duckdb/amalgamation-5.o duckdb/amalgamation-6.o duckdb/amalgamation-7.o duckdb/third_party/fmt/format.o duckdb/third_party/hyperloglog/hyperloglog.o duckdb/third_party/hyperloglog/sds.o duckdb/third_party/libpg_query/pg_functions.o duckdb/third_party/libpg_query/postgres_parser.o duckdb/third_party/libpg_query/src_backend_nodes_list.o duckdb/third_party/libpg_query/src_backend_nodes_makefuncs.o duckdb/third_party/libpg_query/src_backend_nodes_value.o duckdb/third_party/libpg_query/src_backend_parser_gram.o duckdb/third_party/libpg_query/src_backend_parser_parser.o duckdb/third_party/libpg_query/src_backend_parser_scan.o duckdb/third_party/libpg_query/src_backend_parser_scansup.o duckdb/third_party/libpg_query/src_common_keywords.o duckdb/third_party/miniz/miniz.o duckdb/third_party/parquet/parquet_constants.o duckdb/third_party/parquet/parquet_types.o duckdb/third_party/re2/re2/bitstate.o duckdb/third_party/re2/re2/compile.o duckdb/third_party/re2/re2/dfa.o duckdb/third_party/re2/re2/filtered_re2.o duckdb/third_party/re2/re2/mimics_pcre.o duckdb/third_party/re2/re2/nfa.o duckdb/third_party/re2/re2/onepass.o duckdb/third_party/re2/re2/parse.o duckdb/third_party/re2/re2/perl_groups.o duckdb/third_party/re2/re2/prefilter.o duckdb/third_party/re2/re2/prefilter_tree.o duckdb/third_party/re2/re2/prog.o duckdb/third_party/re2/re2/re2.o duckdb/third_party/re2/re2/regexp.o duckdb/third_party/re2/re2/set.o duckdb/third_party/re2/re2/simplify.o duckdb/third_party/re2/re2/stringpiece.o duckdb/third_party/re2/re2/tostring.o duckdb/third_party/re2/re2/unicode_casefold.o duckdb/third_party/re2/re2/unicode_groups.o duckdb/third_party/re2/util/rune.o duckdb/third_party/re2/util/strutil.o duckdb/third_party/snappy/snappy-sinksource.o duckdb/third_party/snappy/snappy.o duckdb/third_party/thrift/thrift/protocol/TProtocol.o duckdb/third_party/thrift/thrift/transport/TBufferTransports.o duckdb/third_party/thrift/thrift/transport/TTransportException.o duckdb/third_party/utf8proc/utf8proc.o duckdb/third_party/utf8proc/utf8proc_wrapper.o duckdb/third_party/zstd/common/entropy_common.o duckdb/third_party/zstd/common/error_private.o duckdb/third_party/zstd/common/fse_decompress.o duckdb/third_party/zstd/common/xxhash.o duckdb/third_party/zstd/common/zstd_common.o duckdb/third_party/zstd/compress/fse_compress.o duckdb/third_party/zstd/compress/hist.o duckdb/third_party/zstd/compress/huf_compress.o duckdb/third_party/zstd/compress/zstd_compress.o duckdb/third_party/zstd/compress/zstd_compress_literals.o duckdb/third_party/zstd/compress/zstd_compress_sequences.o duckdb/third_party/zstd/compress/zstd_compress_superblock.o duckdb/third_party/zstd/compress/zstd_double_fast.o duckdb/third_party/zstd/compress/zstd_fast.o duckdb/third_party/zstd/compress/zstd_lazy.o duckdb/third_party/zstd/compress/zstd_ldm.o duckdb/third_party/zstd/compress/zstd_opt.o duckdb/third_party/zstd/decompress/huf_decompress.o duckdb/third_party/zstd/decompress/zstd_ddict.o duckdb/third_party/zstd/decompress/zstd_decompress.o duckdb/third_party/zstd/decompress/zstd_decompress_block.o duckdb/amalgamation-8.o -LC:lt;secret-pathgt;/R-4.0.2/bin/i386 -lR lt;secret-pathgt;Rtools40mingw32i686-w64-mingw32binld.exe: cannot open output file duckdb.dll: No such file or directory collect2.exe: error: ld returned 1 exit status no DLL was created ERROR: compilation failed for package 'duckdb' * removing 'P:/tmp/duckdb'  

Note I have obfuscated the full path to executables but I have had no problems building and installing packages without compiled code so I know that R Tools generally works in this environment. I have diffed the install logs of the check and INSTALL runs but I can’t seem to find anything that seems significant. The only real difference is that in the check logs, a shortened path using the tilde seems to be passed to build utilities e.g. the argument -LC:lt;secret-path-with-tildesgt;/R-40~1.2/bin/i386 -lR passed to ld by check whereas in the INSTALL run the expanded path is used. These paths are somewhat long (about 110 chars) and I wonder whether that is somehow causing ld to choke — but there are no embedded spaces. I doubt this is the problem but mention it for completeness.

It would be very helpful to know the exact parameters with with R CMD check invokes INSTALL so that I can narrow things down. I could not find this documented in the «Writing R Extensions» manual or the «Tools» section of the «R Internals» manual. I’ve also had a quick search through the ld manual but no mention of long paths. In case it’s not obvious, I have no experience using compilers (beyond the automagical stuff that happens on my Debian box at home). My next strategy is to go source-code diving but I am hoping to avoid that if possible.

Вероятно, также стоит упомянуть , что наша библиотека пакетов сайта добавлена в мою .Rprofile (которая ничего не делает, кроме настройки .libPaths() ), и я знаю check , что она должна работать --vanilla , поэтому я попытался также запустить, например R --vanilla CMD INSTALL ... , но с тем же результатом. Я также задавался вопросом , может ли быть что-то другое в каталоге поэтапной установки, поэтому я попытался перейти --no-staged-install в INSTALL него, снова с тем же результатом, и в этот момент я решил, что было бы лучше обратиться за помощью в понимании проблемы, а не продолжать методом проб и ошибок.