Direct I/O capabilities in the Reconfigurable Application-Specific Computing (RASC) library (rasclib) allows the driver to send data directly from the user space buffers to the RASC coprocessor, without an intervening kernel copy. The main advantage of direct I/O is improved performance.

The direct I/O driver requires that all user buffers be aligned on a 128 byte (128B) cacheline boundary. Furthermore, the buffers must be an even multiple of a 128B cacheline in size. The buffers must be mapped onto contiguous (or as contiguous as possible) physical memory. The physical memory that implements the buffer must have no more than 64 instances of memory discontinuity. All send and receive operations must be direct or all operations must be indirect. Mixed I/O mode is not supported.

The rasclib_open call has a new argument, as follows:

int rasclib_cop_open(char * as_id, unsigned flags)

or

int rasclib_algorithm_open(char * al_id, unsigned flags)

The flags argument can be passed either the RASCLIB_BUFFERED_IO or RASCLIB_DIRECT_IO flag.

If rasclib_open call is passed RASCLIB_BUFFERED_IO, everything works, as previously, with the kernel space copy of the user buffer to a kernel direct memory access (DMA) buffer.

If rasclib_open call is passed RASCLIB_DIRECT_IO, the driver performs direct I/O, enforcing the above restrictions on all send /receive operations associated with this device or algorithm.

In addition, rasclib now supports a memory allocation utility that allocates memory that satisfies the driver restrictions. It does this by allocating memory from the hugepages.

In order to use these routines, you must have your system administrator enable the hugetlbfs configuration option. The following commands can be run very shortly after boot (or as part of the init process):

echo 16 > /proc/sys/vm/nr_hugepages
echo 1 > /proc/sys/fs/shm-use-hugepages
echo 1 > /proc/sys/fs/mmap-use-hugepages
mount none -t hugetlbfs /mnt/hugetlb
chmod 0777 /mnt/hugetlb

The first line sets the number of hugepages. The value of 16 is an example. In a production environment, you need to change that value to suit your system. The last two lines mount the hugetlbfs filesystem and enable general access.

In the example above, it is mounted at /mnt/hugetlb. You can mount it anywhere rasclib can find it. If it is not mounted, the allocation fails.

For more information on hugepages, see /usr/src/linux-2.6.x.x/Documentation/vm/hugetlbpage.txt.

The rasclib function calls for hugepages are, as follows:

void * rasclib_huge_alloc(long size)
void rasclib_huge_free(void *p)
void rasclib_huge_clear()

The rasclib_huge_alloc() function allocates size bytes and returns a void * (pointer) to the resulting pages. The size argument is rounded up to the next 128B cacheline. It returns a number less than zero (which can be passed to rasclib_perror()) on failure.

The rasclib_huge_free() function frees the pointer argument it is passed. The space is returned to the internal free list, it is not unmapped. It is coalesced, if possible.

The rasclib_huge_clear() function unmaps all hugepages that have been allocated, whether they have been free'ed or not. Use this routine only when you are finished using hugepages.

For more information on rasclib, see Chapter 4, “RASC Abstraction Layer”.