token_get_all whitespaces behaviour
I don't know if someone can help me, but i'll ask anyway. I'm creating a function like the php token_get_all
written in javascript. This function should "tokenize" a given php code, but i have some problems with whitespaces.
Executing the token_get_all
function in php i see that only some whitespaces are considered tokens, the other ones are ignored.
Can someone explain me how this function behaves wit开发者_JAVA技巧h whitespaces? Have you ever found some documentation about it?
UPDATE
<?php
if ($var == 0)
{
?>
- Beetween php and if: ignored
- Beetween if and (: tokenized
- Beetween $var and =: tokenized
- Beetween = and 0: tokenized
- Beetween ) and {: tokenized
- Beetween { and ?>: tokenized
Actually, it is never ignored. Zend lexer always returns whitespace, for highlighting/indenting purposes.
"<?php if" (one space) is two tokens: "<?php " -- note the space -- and "if")
"<?php if" (two spaces) is three tokens: "<?php ", T_WHITESPACE + "if"
example:
$t = token_get_all("<?php echo 1;?>");
echo token_name($t[1][0]); // T_ECHO
$t = token_get_all("<?php echo 1;?>");
echo token_name($t[1][0]); // T_WHITESPACE
I've found the solution. Generally whitespaces are ignored after the php open tags: <?php
, <?
but not <?=
UPDATE
It has taken 2 hours, but i've understood the behaviour:).
<?php
and <?
get also the following space char or new line char (preceeded by \r or not). The rest of the whitespaces are parsed in other tokens but grouped if they follow the first whitespace. Let me explain better with your examples:
<?php echo "test"?>
Tokens: "<?php
","echo
"....
<?php echo "test"?>
Tokens: "<?php
"," (remaining whitespaces)","echo
"...
Another example with new lines:
<?php
echo "test"
?>
Tokens: "<?php
\n","echo
"....
<?php
echo "test"
?>
Tokens: "<?php
\n","\n\n(remaining new lines)","echo
"....
I've tested it all the day so i'm sure that it behaves like this.
精彩评论