Блог

Главная - Вопросы по программированию - Асинхронные вызовы MPI делают память безадресной в соответствии с запросом проверки valgrind клиента. Что может вызвать это?

Асинхронные вызовы MPI делают память безадресной в соответствии с запросом проверки valgrind клиента. Что может вызвать это?

#c #memory #mpi #valgrind #openmpi

#c #память #mpi #valgrind #openmpi

Вопрос:

Я тестирую некоторый MPI-код с valgrind, который известен некоторыми ложными срабатываниями при неинициализированном чтении, но приведенный ниже код показывает эту ошибку, которую я никогда не видел. Это происходит с openmpi, в mpich код valgrind-чистый.

 [... some errors that I believe to be false positives]
==8229== Unaddressable byte(s) found during client check request
==8229==    at 0x4016F2: main (unaddressable.c:45)
==8229==  Address 0x7909ce0 is 0 bytes inside a block of size 400 alloc'd
==8229==    at 0x483980B: malloc (vg_replace_malloc.c:307)
==8229==    by 0x401460: main (unaddressable.c:20)

Я понимаю, что ошибки безадресной памяти возникают при доступе к массивам кучи за пределами границ, но здесь valgrind подтверждает, что адрес находится внутри выделенного блока, который еще не был освобожден. Какие события, кроме освобождения, могут вообще сделать память безадресной?

Вот мой код, который я считаю правильным, он был скомпилирован и запущен mpicc -Wall -Wextra -Wpedantic -g unaddressable.c amp;amp; mpiexec -n 4 valgrind --quiet --track-origins=yes ./a.out .

 #include <mpi.h>
#include <valgrind/memcheck.h>
#include <assert.h>
#include <stdlib.h>


int main()
{
    MPI_Init(NULL, NULL);


    int world_rank; MPI_Comm_rank(MPI_COMM_WORLD, amp; world_rank);
    int world_size; MPI_Comm_size(MPI_COMM_WORLD, amp;world_size);

    int left = (world_rank-1 world_size)%world_size;
    int right = (world_rank 1 world_size)%world_size;

    int size = 100;
    int * recv_from_right = malloc(sizeof(int)*size);
    int * recv_from_left = malloc(sizeof(int)*size);
    int * send_to_left = malloc(sizeof(int)*size);
    int * send_to_right = malloc(sizeof(int)*size);
    assert(recv_from_right amp;amp; recv_from_left amp;amp; send_to_left amp;amp; send_to_right);

    for (int i=0; i<size; i  )
    {
        send_to_left[i] = i;
        send_to_right[i] = size-i;
    }


    MPI_Request reqs[4]; MPI_Request * req = amp;reqs[0];

    MPI_Isend(send_to_left, size, MPI_INT, left, 0, MPI_COMM_WORLD, req  );
    MPI_Irecv(recv_from_right, size, MPI_INT, right, 0, MPI_COMM_WORLD, req  );

    MPI_Isend(send_to_right, size, MPI_INT, right, 1, MPI_COMM_WORLD, req  );
    MPI_Irecv(recv_from_left, size, MPI_INT, left, 1, MPI_COMM_WORLD, req  );

    MPI_Waitall(4, reqs, MPI_STATUS_IGNORE);

    // these client check requests produce error "Unaddressable byte(s) found [...] 0 bytes inside a block"
    VALGRIND_CHECK_MEM_IS_DEFINED(recv_from_left, size*sizeof(int));
    VALGRIND_CHECK_MEM_IS_DEFINED(recv_from_right, size*sizeof(int));


    free(send_to_left); free(send_to_right); free(recv_from_left); free(recv_from_right);
    MPI_Finalize();
}

Я использую

  • gcc 9.3.1
  • valgrind-3.16.0

для mpi:

  • mpich 3.3.2 (valgrind-clean)
  • Откройте MPI 4.0.2 (для ошибки)

Я установил Openmpi через dnf, ниже приведен мой вывод для ompi_info

                  Package: Open MPI
                          mockbuild@buildhw-02.phx2.fedoraproject.org
                          Distribution
                Open MPI: 4.0.2
  Open MPI repo revision: v4.0.2
   Open MPI release date: Oct 07, 2019
                Open RTE: 4.0.2
  Open RTE repo revision: v4.0.2
   Open RTE release date: Oct 07, 2019
                    OPAL: 4.0.2
      OPAL repo revision: v4.0.2
       OPAL release date: Oct 07, 2019
                 MPI API: 3.1.0
            Ident string: 4.0.2
                  Prefix: /usr/lib64/openmpi
 Configured architecture: x86_64-unknown-linux-gnu
          Configure host: buildhw-02.phx2.fedoraproject.org
           Configured by: mockbuild
           Configured on: Mon Oct  7 23:28:49 UTC 2019
          Configure host: buildhw-02.phx2.fedoraproject.org
  Configure command line: '--prefix=/usr/lib64/openmpi'
                          '--mandir=/usr/share/man/openmpi-x86_64'
                          '--includedir=/usr/include/openmpi-x86_64'
                          '--sysconfdir=/etc/openmpi-x86_64'
                          '--disable-silent-rules' '--enable-builtin-atomics'
                          '--enable-mpi-cxx' '--enable-mpi-java'
                          '--enable-mpi1-compatibility' '--with-sge'
                          '--with-valgrind' '--enable-memchecker'
                          '--with-hwloc=/usr' '--with-libevent=external'
                          '--with-pmix=external' 'CC=gcc' 'CXX=g  '
                          'LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now
                          -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
                          'CFLAGS= -O2 -g -pipe -Wall -Werror=format-security
                          -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
                          -fexceptions -fstack-protector-strong
                          -grecord-gcc-switches
                          -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
                          -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
                          -mtune=generic -fasynchronous-unwind-tables
                          -fstack-clash-protection -fcf-protection'
                          'CXXFLAGS= -O2 -g -pipe -Wall
                          -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
                          -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
                          -fstack-protector-strong -grecord-gcc-switches
                          -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
                          -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
                          -mtune=generic -fasynchronous-unwind-tables
                          -fstack-clash-protection -fcf-protection'
                          'FC=gfortran' 'FCFLAGS= -O2 -g -pipe -Wall
                          -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
                          -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions
                          -fstack-protector-strong -grecord-gcc-switches
                          -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
                          -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
                          -mtune=generic -fasynchronous-unwind-tables
                          -fstack-clash-protection -fcf-protection'
                Built by: mockbuild
                Built on: Mon Oct  7 23:34:02 UTC 2019
              Built host: buildhw-02.phx2.fedoraproject.org
              C bindings: yes
            C   bindings: yes
             Fort mpif.h: yes (all)
            Fort use mpi: yes (full: ignore TKR)
       Fort use mpi size: deprecated-ompi-info-value
        Fort use mpi_f08: yes
 Fort mpi_f08 compliance: The mpi_f08 module is available, but due to
                          limitations in the gfortran compiler and/or Open
                          MPI, does not support the following: array
                          subsections, direct passthru (where possible) to
                          underlying Open MPI's C functionality
  Fort mpi_f08 subarrays: no
           Java bindings: yes
  Wrapper compiler rpath: runpath
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
      C compiler version: 9.2.1
            C   compiler: g  
   C   compiler absolute: /usr/bin/g  
           Fort compiler: gfortran
       Fort compiler abs: /usr/bin/gfortran
         Fort ignore TKR: yes (!GCC$ ATTRIBUTES NO_ARG_CHECK ::)
   Fort 08 assumed shape: yes
      Fort optional args: yes
          Fort INTERFACE: yes
    Fort ISO_FORTRAN_ENV: yes
       Fort STORAGE_SIZE: yes
      Fort BIND(C) (all): yes
      Fort ISO_C_BINDING: yes
 Fort SUBROUTINE BIND(C): yes
       Fort TYPE,BIND(C): yes
 Fort T,BIND(C,name="a"): yes
            Fort PRIVATE: yes
          Fort PROTECTED: yes
           Fort ABSTRACT: yes
       Fort ASYNCHRONOUS: yes
          Fort PROCEDURE: yes
         Fort USE...ONLY: yes
           Fort C_FUNLOC: yes
 Fort f08 using wrappers: yes
         Fort MPI_SIZEOF: yes
             C profiling: yes
           C   profiling: yes
   Fort mpif.h profiling: yes
  Fort use mpi profiling: yes
   Fort use mpi_f08 prof: yes
          C   exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: yes
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
              dl support: yes
   Heterogeneous support: no
 mpirun default --prefix: no
       MPI_WTIME support: native
     Symbol vis. support: yes
   Host topology support: yes
            IPv6 support: no
      MPI1 compatibility: yes
          MPI extensions: affinity, cuda, pcollreq
   FT Checkpoint support: no (checkpoint thread: no)
   C/R Enabled Debugging: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA allocator: bucket (MCA v2.1.0, API v2.0.0, Component v4.0.2)
           MCA allocator: basic (MCA v2.1.0, API v2.0.0, Component v4.0.2)
           MCA backtrace: execinfo (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA btl: tcp (MCA v2.1.0, API v3.1.0, Component v4.0.2)
                 MCA btl: openib (MCA v2.1.0, API v3.1.0, Component v4.0.2)
                 MCA btl: vader (MCA v2.1.0, API v3.1.0, Component v4.0.2)
                 MCA btl: uct (MCA v2.1.0, API v3.1.0, Component v4.0.2)
                 MCA btl: usnic (MCA v2.1.0, API v3.1.0, Component v4.0.2)
                 MCA btl: self (MCA v2.1.0, API v3.1.0, Component v4.0.2)
            MCA compress: gzip (MCA v2.1.0, API v2.0.0, Component v4.0.2)
            MCA compress: bzip (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA crs: none (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA dl: dlopen (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA event: external (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA hwloc: external (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA if: linux_ipv6 (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                  MCA if: posix_ipv4 (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
         MCA installdirs: env (MCA v2.1.0, API v2.0.0, Component v4.0.2)
         MCA installdirs: config (MCA v2.1.0, API v2.0.0, Component v4.0.2)
          MCA memchecker: valgrind (MCA v2.1.0, API v2.0.0, Component v4.0.2)
              MCA memory: patcher (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA mpool: hugepage (MCA v2.1.0, API v3.0.0, Component v4.0.2)
             MCA patcher: overwrite (MCA v2.1.0, API v1.0.0, Component
                          v4.0.2)
                MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA pmix: ext3x (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA pstat: linux (MCA v2.1.0, API v2.0.0, Component v4.0.2)
              MCA rcache: grdma (MCA v2.1.0, API v3.3.0, Component v4.0.2)
           MCA reachable: weighted (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA shmem: mmap (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA shmem: sysv (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA shmem: posix (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA timer: linux (MCA v2.1.0, API v2.0.0, Component v4.0.2)
              MCA errmgr: default_hnp (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
              MCA errmgr: default_tool (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
              MCA errmgr: default_app (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
              MCA errmgr: default_orted (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
                 MCA ess: pmi (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA ess: singleton (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
                 MCA ess: env (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA ess: hnp (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA ess: tool (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA ess: slurm (MCA v2.1.0, API v3.0.0, Component v4.0.2)
               MCA filem: raw (MCA v2.1.0, API v2.0.0, Component v4.0.2)
             MCA grpcomm: direct (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA iof: tool (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA iof: orted (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA iof: hnp (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA odls: default (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA odls: pspawn (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA oob: tcp (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA plm: rsh (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA plm: isolated (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA plm: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA ras: gridengine (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                MCA regx: naive (MCA v2.1.0, API v1.0.0, Component v4.0.2)
                MCA regx: reverse (MCA v2.1.0, API v1.0.0, Component v4.0.2)
                MCA regx: fwd (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA rmaps: rank_file (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA rmaps: seq (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA rmaps: ppr (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA rmaps: round_robin (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA rmaps: resilient (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA rmaps: mindist (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA rml: oob (MCA v2.1.0, API v3.0.0, Component v4.0.2)
              MCA routed: direct (MCA v2.1.0, API v3.0.0, Component v4.0.2)
              MCA routed: radix (MCA v2.1.0, API v3.0.0, Component v4.0.2)
              MCA routed: binomial (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA rtc: hwloc (MCA v2.1.0, API v1.0.0, Component v4.0.2)
              MCA schizo: slurm (MCA v2.1.0, API v1.0.0, Component v4.0.2)
              MCA schizo: orte (MCA v2.1.0, API v1.0.0, Component v4.0.2)
              MCA schizo: flux (MCA v2.1.0, API v1.0.0, Component v4.0.2)
              MCA schizo: ompi (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA state: orted (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA state: hnp (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA state: app (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA state: tool (MCA v2.1.0, API v1.0.0, Component v4.0.2)
               MCA state: novm (MCA v2.1.0, API v1.0.0, Component v4.0.2)
                 MCA bml: r2 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: basic (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: libnbc (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                MCA coll: inter (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: tuned (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: self (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: sm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA coll: sync (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA fbtl: posix (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                MCA fbtl: pvfs2 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA fcoll: dynamic_gen2 (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA fcoll: two_phase (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA fcoll: dynamic (MCA v2.1.0, API v2.0.0, Component v4.0.2)
               MCA fcoll: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
               MCA fcoll: vulcan (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA fs: pvfs2 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA fs: ufs (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                  MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA mtl: psm2 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA mtl: ofi (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA mtl: psm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA osc: sm (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA osc: monitoring (MCA v2.1.0, API v3.0.0, Component
                          v4.0.2)
                 MCA osc: rdma (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA osc: pt2pt (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA osc: ucx (MCA v2.1.0, API v3.0.0, Component v4.0.2)
                 MCA pml: v (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA pml: ucx (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA pml: ob1 (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA pml: cm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
                 MCA pml: monitoring (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                 MCA rte: orte (MCA v2.1.0, API v2.0.0, Component v4.0.2)
            MCA sharedfp: lockedfile (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
            MCA sharedfp: sm (MCA v2.1.0, API v2.0.0, Component v4.0.2)
            MCA sharedfp: individual (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)
                MCA topo: treematch (MCA v2.1.0, API v2.2.0, Component
                          v4.0.2)
                MCA topo: basic (MCA v2.1.0, API v2.2.0, Component v4.0.2)
           MCA vprotocol: pessimist (MCA v2.1.0, API v2.0.0, Component
                          v4.0.2)

Комментарии:

1. как вы установили Open MPI? если вы создали его из исходных текстов, какова ваша configure командная строка?

2. @GillesGouaillardet Я добавил информацию, предоставленную командой ompi_info, которая включает в себя командную строку configure

3. поскольку Open MPI был настроен с --with-valgrind --enable-memchecker помощью , он вызывает valgrind API, например, чтобы пометить память как неадресируемую между MPI_Irecv() и MPI_Wait() . Здесь может быть ошибка. Я бы посоветовал вам вручную перестроить открытый MPI без этих опций.

4. Вероятно, это не источник проблемы, но — для корректности — последний аргумент MPI_Waitall должен быть MPI_STATUSES_IGNORE (обратите внимание на форму множественного числа). В Open MPI его значение такое же, как MPI_STATUS_IGNORE , но стандарт говорит, что это не обязательно так.