Why is this StringTokenizer not tokenizing properly following the second time?

I want to parse the following using StringTokenizer for every string matching agent>. I tried it using the code like this. Where I am going wrong?

StringTokenizer stringtokenizer=new StringTokenize(hari,"agent>");

while(stringtokenizer.hasMoreTokens())
{
    String token = stringtokenizer.nextToken();
    System.out.println("output ="+token);
}

It is tokenizing properly only for two times and then it is responding irregularly... I mean for first time

output =ACT-USER::USER_1:AU01::******;


   SIMULATOR 09-11-17 09:37:01
M  AU01 COMPLD
;

Second time:

output =RTRV-HDR:::RH01;


   SIMULATOR 09-11-17 09:37:01
M  RH01 COMPLD
;

Third time:

output =RTRV-EQPT::ALL:RE01;


   SIMULATOR 09-11-17 09:37:02
M  RE01 DENY
   SROF
   /*S
output =us, R
output =qu
output =s
output =d Op
output =r
output =io
output = F
output =il
output =d*/
;


ACT-USER::USER_1:AU01::******;


   SIMULATOR 开发者_运维知识库09-11-17 09:37:01
M  AU01 COMPLD
;
agent>RTRV-HDR:::RH01;


   SIMULATOR 09-11-17 09:37:01
M  RH01 COMPLD
;
agent>RTRV-EQPT::ALL:RE01;


   SIMULATOR 09-11-17 09:37:02
M  RE01 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>RTRV-ALM-EQPT::ALL:RA01;


   SIMULATOR 09-11-17 09:37:03
M  RA01 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>RTRV-ALM-EQPT::ALL:RA02::MJ,;


   SIMULATOR 09-11-17 09:37:03
M  RA02 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>RTRV-ALM-EQPT::ALL:RA03::MN,;


   SIMULATOR 09-11-17 09:37:04
M  RA03 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>RTRV-ALM-EQPT::ALL:RA04::MN,T-LASERCURR-1-HIGH;


   SIMULATOR 09-11-17 09:37:04
M  RA04 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>RTRV-PM-EQPT::ALL:RP01;


   SIMULATOR 09-11-17 09:37:05
M  RP01 DENY
   SROF
   /*Status, Requested Operation Failed*/
;
agent>INIT-SYS::SLOT-1-1-2:IS01::1;


   SIMULATOR 09-11-17 09:37:05
M  IS01 COMPLD
;
agent>CANC-USER::USER_1:CU01;


   SIMULATOR 09-11-17 09:37:06
M  CU01 DENY
   IIAC
   /*Invalid Access Identifier*/
;
agent>

Please let me know where i am going wrong.

You didn't read the API. StringTokenizer does not accept a string as a single delimiter. It accepts a string where each character in the string is treated as a one character delimiter.

Try using String.split(...);

Try this,

int idx=0; int tokenCount;
String words[]=new String [500];
String message=//String to be tokenize;
StringTokenizer st=new StringTokenizer(message);
tokenCount=st.countTokens();
System.out.println("Number of tokens = " + tokenCount);
while (st.hasMoreTokens()) // is there stuff to get?
    {words[idx]=st.nextToken(); idx++;}
for (idx=0;idx<tokenCount; idx++)
    { if(words[idx].equals("agent>");//do something

}