List Info

Thread: Using UNIX:UNIX-MMAP to read large text files under CMUCL?




Using UNIX:UNIX-MMAP to read large text files under CMUCL?
user name
2007-04-05 11:34:51
I'm familiar with how to use mmap() in C for this (reading
large text
files), but I'm not sure how to go about it under CMUCL.

I found this example in comp.lang.lisp, but since it just
returns the
pointer (the output of the mmap call as an integer), how do
I convert
that pointer into some kind of lisp object/structure that I
can use to
read the results, either with (read-sequence) or
(read-line)?

    (defun mmap (file offset length &optional write-p)
      (let* ((ofs32 (if (zerop (logand offset #x80000000))
                      offset
                      (- offset #x100000000))) ; Fake up a
"(signed-byte 32)"
             (mode (logior #+linux unix::O_SYNC     ; Not on
BSD (*sigh*)
                           (if write-p unix:O_RDWR
unix:O_RDONLY)))
             (fd (unix:unix-open file mode 0)))
        (if (not fd)
          (error "Can't open ~s: ~s" file
(unix:get-unix-error-msg))
          (unwind-protect
              (let ((mode (logior unix:PROT_READ
                                  (if write-p
unix:PROT_WRITE 0))))
                (system:sap-int
                  (unix:unix-mmap nil length mode
unix:MAP_SHARED fd ofs32)))
            (unix:unix-close fd)))))

Also, when using CMUCL's mmap call, am I forced to open the
file with
(unix:unix-open) or can I use (with-open-file) as well?

When I tried the latter, the call to (unix:unix-mmap)
returned a null value:

CL-USER> (with-open-file (file-stream
"/path/to/very_large_file.txt" :direction
:input)
           (let ((mmap-ptr (unix:unix-mmap nil (file-length
file-stream) unix:PROT_READ unix:MAP_PRIVATE
(system:fd-stream-fd
file-stream) 0)))
             mmap-ptr))
NIL


Re: Using UNIX:UNIX-MMAP to read large text files under CMUCL?
country flaguser name
United States
2007-04-07 09:46:47
  |Date: Sat, 07 Apr 2007 10:42:01 -0400
  |From: Raymond Toy <raymond.toyericsson.com>
  |
  |Madhu wrote:
  |> Try the following definition of unix-mmap (for
unix-glibc2.lisp, which
  |> includes linux)
  |
  |Thanks for pointing out the issue.  I guess it was lost
during the
  |corruption and I didn't see it.  (It makes me wonder what
else was lost
  |during that month or so.)
  |
  |I'll check in something very similar to this by just
copying the version
  |from unix.lisp.
  |

Note there is a small difference in the return type of the
first
argument between what i posted, and what was in unix.lisp
(which had
been checked in in sept): `int' can be negative (and i
noticed this
caused problems sometimes on linux.)

Perhaps you can check the commit log mail during that time
period. I
have not rsynced my cvs since sept, still hoping to
bootstrap a
double-double lisp from sources someday 

--
Madhu




Re: Using UNIX:UNIX-MMAP to read large text files under CMUCL?
country flaguser name
United States
2007-04-07 09:42:01
Madhu wrote:
> Try the following definition of unix-mmap (for
unix-glibc2.lisp, which
> includes linux)

Thanks for pointing out the issue.  I guess it was lost
during the
corruption and I didn't see it.  (It makes me wonder what
else was lost
during that month or so.)

I'll check in something very similar to this by just copying
the version
from unix.lisp.

I'll probably also create an official patch for 19d.

Ray


[1-3]

about | contact  Other archives ( Real Estate discussion Medical topics )