How to detect eof in ml-lex

While writing a code in ml-lex we need to write to write the eof function val eof = f开发者_StackOverflow社区n () => EOF; is this a necessary part to write also if i want my lexer to stop at the detection of an eof then what should i add to the given function. Thanks.

The User’s Guide to ML-Lex and ML-Yacc by Roger Price is great for learning ml-lex and ml-yacc.

The eof function is mandatory in the user declarations part of your lex definition together with the lexresult type as:

The function eof is called by the lexer when the end of the input stream is reached.

Where your eof function can either throw an exception if that is appropriate for your application or the EOF token. In any way it have to return something of type lexresult. There is an example in chapter 7.1.2 of the user guide which prints a string if EOF was in the middle of a block comment.

I use a somewhat "simpler" eof function

structure T = Tokens
structure C = SourceData.Comments

fun eof data =
if C.depth data = 0 then
    T.EOF (~1, ~1)
else
  fail (C.start data) "Unclosed comment"

where the C structure is a "special" comment handling structure that counts number of opening and closing comments. If the current depth is 0 then it returns the EOF token, where (~1, ~1) are used indicate the left and right position. As I don't use this position information for EOF i just set it to (~1, ~1).

Normally you would then set the %eop (end of parse) to use the EOF token in the yacc file, to indicate that what ever start symbol that is used, it may be followed by the EOF token. Also remember to add EOF to %noshift. Se section 9.4.5 for %eop and %noshift.

Obviously you have to define EOF in %term declaration of your yacc file aswel.

Hope this helps, else take a look at an MLB parser or an SML parser written in ml-lex and ml-yacc. The MLB parser is the simplest and thus might be easier to understand.