How to write a decent process filter?

I'm building a program that communicates with Emacs, and one of the challenges I'm facing is writing Emacs's process filter function. Its input string is a series of s-expressions to be evaluated. Here is a sample:

(gimme-append-to-buffer "25 - William Christie dir, Les Arts Florissants - Scene 2. Prelude - Les Arts Florissants\n")
(gimme-append-to-buffer "26 - William Christie dir, Les Arts Florissants - Cybele: 'Je Veux Joindre' - Les Arts Florissants\n")
(gimme-append-to-buffer "27 - William Christie dir, Les Arts Florissants - Scene 3. Cybele: 'Tu T'Etonnes, Melisse' - Les Arts Florissants\n")
(gimme-append-to-buffer "28 - William Christie dir, Les Arts Florissants - Cybele: 'Que Les Plus Doux Zephyrs'. Scene 4. - Les Arts Florissants\n")
(gimme-appen开发者_运维技巧d-to-buffer "29 - William Christie dir, Les Arts Florissants - Entree Des Nations - Les Arts Florissants\n")
(gimme-append-to-buffer "30 - William Christie dir, Les Arts Florissants - Entree Des Zephyrs - Les Arts Florissants\n")
(gimme-append-to-buffer "31 - William Christie dir, Les Arts Florissants - Choeur Des Nations' 'Que Devant Vous' - Les Arts Florissants\n")
(gimme-append-to-buffer "32 - William Christie dir, Les Arts Florissants - Atys: 'Indigne Que Je Suis' - Les Arts Florissants\n")
(gimme-append-to-buffer "33 - William Christie dir, Les Arts Florissants - Reprise Du Choeur Des Nations : 'Que Devant Nous' - Les Arts Florissants\n")
(gimme-append-to-buffer "34 - William Christie dir, Les Arts Flor*emphasized text*issants - Reprise De L'Air Des Zephyrs - Les Arts Florissants\n")

The first problem that I've faced is that the string is somehow not fully formed when the function is so called, so writing something like (mapcar 'eval (format "(%s)" input-string)) won't work.

To deal with this first problem, I was using a loop. The full function I wrote is:

 (defun eval-all-sexps (s)
   (loop for x = (ignore-errors (read-from-string s))
          then (ignore-errors (read-from-string (substring s position)))
          while x
          summing (or (cdr x) 0) into position
          doing (eval (car x))))

Now the second problem that showed up is that the function is called twice with a somewhat large input, first with valid but partial content, then with what looks like pieces of the remaining data.

To solve this problem, I'm considering using a junk variable to hold up what remains from a loop and then concatenating it to the input of the next call, but I was wondering if you guys have any other suggestions on how to deal with such a problem more elegantly.

Thanks!

That's what I do in a shell mode I use.
I look for a special end-of-record string in the process output, to know when the output is complete. Until then I concat the chunks.

(defun my-shell-exec-filter (process result)
  (let ((end-of-result (string-match my-shell-end-of-record-string result)))
    (if (and end-of-result
             (numberp end-of-result)
             (> end-of-result 0))
        (setq my-shell-reply 
              (concat my-shell-reply
                      (substring (substring result 0 end-of-result) 0 -1)))

      (progn
        (setq my-shell-reply (concat my-shell-reply result))
        (accept-process-output process my-shell-exec-timeout 5)))))

This isn't quite right because the end-of-record string can be split across multiple invocations of that filter fn. So to be correct, it ought to concat first, then check for a match. But anyway you get the idea.

Also it recurses, and because of that, for very very long input, it can incur a stack overflow exception. So it definitely is not as clean as I would like.

The thing is kicked off with

      (setq my-shell-reply nil) 
      (set-process-filter proc my-shell-exec-filter)
      (process-send-string proc "whatever...") 
      (if (not (accept-process-output proc my-shell-exec-timeout 100))
          (error "unexpected response."))

      ;; examine my-shell-reply here ... 

My final solution was:

(defun eval-all-sexps (s)
  (let ((s (concat gimme-filter-remainder s)))
    (setq gimme-filter-remainder
          (loop for x = (ignore-errors (read-from-string s))
                then (ignore-errors (read-from-string (substring s position)))
                while x
                summing (or (cdr x) 0) into position
                doing (eval (car x))
                finally (return (substring s position))))))