(Paper #153)
Although InfiniBand Architecture is relatively new in the high performance computing area, it offers many features which help us to improve the performance of communication subsystems. One of these features is Remote Direct Memory Access (RDMA) operations. In this paper, we propose a new design of MPI over InfiniBand which brings the benefit of RDMA to not only large messages, but also small and control messages. We also achieve scalability by exploiting application communication pattern and combining send/receive operations with RDMA operations. Our RDMA-based MPI implementation currently delivers a latency of 7.5 microseconds for small messages and a peak bandwidth of 857 million bytes (817 mega bytes) per second. Performance evaluation at the MPI level shows that for small messages, our RDMA based design can reduce the latency by 22%, increase the bandwidth by over 70%, and reduce the host overhead by 15%. For large messages, we improve performance by reducing the time for transferring control messages. We have also shown that our new design is beneficial to MPI collective communication and NAS Parallel Benchmarks.
Keywords:
Architecture
Middleware and APIs