problem with deserialization xml to objects - unwanted split by special chars

I try to deserialize xml to objects, and i met a problem with encoding of various items in xml tree.

XML example:

<?xml version="1.0" encoding="utf-8"?>
<results>
  <FlightTravel>
    <QuantityOfPassengers>6</QuantityOfPassengers>
    <Id>N5GWXM</Id>
    <InsuranceId>330992</InsuranceId>
    <TotalTime>3h 00m</TotalTime>
    <TransactionPrice>540.00</TransactionPrice>
    <AdditionalPrice>0</AdditionalPrice>
    <InsurancePrice>226.56</InsurancePrice>
    <TotalPrice>9561.31</TotalPrice>
    <CompanyName>XXXXX</CompanyName>
    <TaxID>111-11-11-111</TaxID>
    <InvoiceStreet>Jagiellońska</InvoiceStreet>
    <InvoiceHouseNo>8</InvoiceHouseNo>
    <InvoiceZipCode>Jagiellońska</InvoiceZipCode>
    <InvoiceCityName>Warszawa</InvoiceCityName>
    <PayerStreet>Jagiellońska</PayerStreet>
    <PayerHouseNo>8</PayerHouseNo>
    <PayerZipCode>11-111</PayerZipCode>
    <PayerCityName>Warszawa</PayerCityName>
    <PayerEmail>no-reply@xxxx.pl</PayerEmail>
    <PayerPhone>123123123</PayerPhone>
    <Segments>
      <Segment0>
        <DepartureAirport>WAW</DepartureAirport>
        <DepartureDate>śr. 06 lip</DepartureDate>
        <DepartureTime>07:50</DepartureTime>
        <ArrivalAirport>VIE</ArrivalAirport>
        <ArrivalDate>śr. 06 lip</ArrivalDate>
        <ArrivalTime>09:15</ArrivalTime>
      </Segment0>
      <Segment1>
        <DepartureAirport开发者_运维技巧>VIE</DepartureAirport>
        <DepartureDate>śr. 06 lip</DepartureDate>
        <DepartureTime>10:00</DepartureTime>
        <ArrivalAirport>SZG</ArrivalAirport>
        <ArrivalDate>śr. 06 lip</ArrivalDate>
        <ArrivalTime>10:50</ArrivalTime>
      </Segment1>
    </Segments>
  </FlightTravel>
</results>

XML Deserialization function in python:

# -*- coding: utf-8 -*-

from lxml import etree
import codecs

class TitleTarget(object):
    def __init__(self):
        self.text = []
    def start(self, tag, attrib):
        self.is_title = True #if tag == 'Title' else False
    def end(self, tag):
        pass
    def data(self, data):
        if self.is_title:
            self.text.append(data)
    def close(self):
        return self.text

parser = etree.XMLParser(target = TitleTarget())

infile = 'Flights.xml'
results = etree.parse(infile, parser)

out = open('wynik.txt', 'w')
out.write('\n'.join(results))
out.close()

Output:

['6', 'N5GWXM', '330992', '3h 00m', '540.00 ', '0', '226.56', '9561.31', 'XXXXX', '111-11-11-111', 'Jagiello', 'ń', 'ska', '8', 'Jagiello', 'ń', 'ska', 'Warszawa', 'Jagiello', 'ń', 'ska', '8', '11-111', 'Warszawa', 'no-reply@xxxx.pl', '123123123', 'WAW', 'ś', 'r. 06 lip', '07:50', 'VIE', 'ś', 'r. 06 lip', '09:15', 'VIE', 'ś', 'r. 06 lip', '10:00', 'SZG', 'ś', 'r. 06 lip', '10:50']

In item 'Jagiellońska' is special char 'ń'. When parser appending data to array then char 'ń' is some king of split character and my question is why this is happening? Rest of items are appending to array correctly. In item 'śr 06.lip' is exactly the same situation.

The problem is that the data method of your target class may be called more than once per element. This may happen if the feeder crosses a block boundary, for example. Looks like it can also happen when it hits a non-ASCII character. This is ancient legend. I can't find where this is documented. However if you change your target class to something like the following, it will work. I have tested it on your data.

class TitleTarget(object):
    def __init__(self):
        self.text = []
    def start(self, tag, attrib):
        self.is_title = True #if tag == 'Title' else False
        if self.is_title:
            self.text.append(u'')
    def end(self, tag):
        pass
    def data(self, data):
        if self.is_title:
            self.text[-1] += data
    def close(self):
        return self.text

To get a better grasp of what your output is like, do print repr(results) after the parse call. You should now see such pieces of unsplit text as

u'Jagiello\u0144ska\n    '
u'\u015br. 06 lip\n        '