Memory buildup when parsing XML into a Core Data store using NSXMLParser

I have a problem with an app that takes an XML feed, parses it, and stores the results into Core Data.

The problem only occurs on the very first run of the app when there is nothing in the store and the whole feed is parsed and stored. The problem is simply that memory allocations build up and up until, on 50% of attempts it crashes the app, usually at around 10Mb. The objects allocated seem to be CFData(store) objects and I can't seem to find any way to force a release of them. If you can get it to run once and successfully complete the parsing and save to core data then every subsequent launch is fine, memory usage never exceeds 2.5Mb

Here's the general approach I have before we get into code: Get the feed into an NSData object. Use NSFileManager to store it as a file. Create a URL from the file path and give it to the parseXMLFile: method. Delegate is self. On reaching parser:didStartElement:namespaceURI:qualifiedName:attributes: I create a dictionary to catch data from tags I need. The parser:foundCharacters: method saves the contents of the tag to a string The parser:didEndElement:... method then determines if the tag is something we need. If so it adds it to the dictionary if not it ignores it. Once it reaches the end of an item it calls a method to add it to the core data store.

From looking at sample code and other peoples postings here it seems there's nothing in the general approach thats wrong.

The code is below, it comes from a larger context of a view controller but I omitted anything not directly related.

In terms of things I have tried: The feed is XML, no option to convert to JSON, sorry. It's not the images being found and stored in the shared documents area.

Clues as to what it might be: This is the entry I get from Instruments on the largest most numerous things allocated (256k per item):

Object Address Category Creation Time Live Size Responsible Library Responsible Caller

1 0xd710000 CFData (store) 16024774912 • 262144 CFNetwork URLConnectionClient::clientDidReceiveData(_CFData const*, URLConnectionClient::ClientConnectionEventQueue*)

And here's the code, edited for anonymity of the content ;)

-(void)parserDidStartDocument:(NSXMLParser *)feedParser { }

-(void)parserDidEndDocument:(NSXMLParser *)feedParser { }

-(void)parser:(NSXMLParser *)feedParser didStartElement:(NSString *)elementName namespaceURI:(NSString *)namespaceURI qualifiedName:(NSString *)qualifiedName attributes:(NSDictionary *)attributeDict
{
    eachElement = elementName;
    if ( [eachElement isEqualToString:@"item" ] )
    {
        //create a dictionary to store each item from the XML feed
        self.currentItem = [[[NSMutableDictionary alloc] init] autorelease];
    }
}


-(void)parser:(NSXMLParser *)feedParser foundCharacters:(NSString *)feedString 
{
    //Make sure that tag content doesn't line break unnecessarily
    if (self.currentString == nil) 
    {
        self.currentString = [[[NSMutableString alloc] init]autorelease];
    }
    [self.currentString appendString:feedString];

}

-(void)parser:(NSXMLParser *)feedParser didEndElement:(NSString *)elementName namespaceURI:(NSString *)namespaceURI qualifiedName:(NSString *)qName 
{
    eachElement = elementName;
    NSString *tempString = [self.currentString stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];

if ( [eachElement isEqualToString:@"title"] ) 
    {
        //skip the intro title
        if (![tempString isEqualToString:@"Andy Panda UK"])
        {
            //add item to di citonary output to console
            [self.currentItem setValue:tempString forKey:eachElement];
        }
    }
    if ( [eachElement isEqualToString:@"link"] ) 
    {
        //skip the intro link
        if (![tempString isEqualToString:@"http://andypanda.co.uk/comic"])
        {
            //add item to dicitonary output to console
            [self.currentItem setValue:tempString forKey:eachElement];
        }
    }
    if ( [eachElement isEqualToString:@"pubDate"] ) 
    {
        //add item to dicitonary output to console
        [self.currentItem setValue:tempString forKey:eachElement];
    }
    if ( [eachElement isEqualToString:@"description"] ) 
    {
        if ([tempString length] > 150)
        {
            //trims the string to just give us the link to the image file
            NSRange firstPart = [tempString rangeOfString:@"src"];
            NSString *firstString = [tempString substringFromIndex:firstPart.location+5];
            NSString *secondString;
            NSString *separatorString = @"\"";
            NSScanner *aScanner = [NSScanner scannerWithString:firstString];
            [aScanner scanUpToString:separatorString intoString:&secondString];

            //trims the string further to give us just the credits
            NSRange secondPart = [firstString rangeOfString:@"title="];
            NSString *thirdString = [firstString substringFromIndex:secondPart.location+7];
            thirdString = [thirdString substringToIndex:[thirdString length] - 12];
            NSString *fourthString= [secondString substringFromIndex:35];

            //add the items to the dictionary and output to console
            [self.currentItem setValue:secondString forKey:@"imageURL"];
            [self.currentItem setValue:thirdString forKey:@"credits"];
            [self.currentItem setValue:fourthString forKey:@"imagePreviewURL"];
            //safety sake set unneeded objects to nil before scope rules kick in to release them
            firstString = nil;
            secondString = nil;
            thirdString = nil;
        }
        tempString = nil;
    }


    //close the feed and release all the little objects! Fly be free!
    if ( [eachElement isEqualToString:@"item" ] )
    {
        //get the date sorted
        NSDateFormatter *formatter = [[NSDateFormatter alloc] init];
        formatter.dateFormat = @"EEE, dd MMM yyyy HH:mm:ss ZZZZ";
        NSDate *pubDate = [formatter dateFromString:[currentItem valueForKey:@"pubDate"]];
        [formatter release];
        NSArray *fetchedArray = [[NSArray alloc] initWithArray:[fetchedResultsController fetchedObjects]];
        //build the string to make the image
        NSString *previewURL = [@"http://andypanda.co.uk/comic/comics-rss/" stringByAppendingString:[self.currentItem valueForKey:@"imagePreviewURL"]];

        if ([fetchedArray count] == 0)
        {
            [self sendToCoreDataStoreWithDate:pubDate andPreview:previewURL];
        } 
        else 
        {
            int i, matches = 0;
            for (i = 0; i < [fetchedArray count]; i++)
            {
                if ([[self.currentItem valueForKey:@"title"] isEqualToString:[[fetchedArray objectAtIndex:i] valueForKey:@"stripTitle"]])
                {
                    matches++;
                }
            }

            if (matches == 0)
            {
                [self sendToCoreDataStoreWithDate:pubDate andPreview:previewURL];
            }
        }
        previewURL = nil;
        [previewURL release];
        [fetchedArray release];
    }
    self.currentString = nil;
}

- (void)sendToCoreDataStoreWithDate:(NSDate*)pubDate andPreview:(NSString*)previewURL {
    NSManagedObjectContext *context = [fetchedResultsController managedObjectContext];
    NSEntityDescription *entity = [[fetchedResultsController fetchRequest] entity];
    newManagedObject = [NSEntityDescription insertNewObjectForEntityForName:[entity name] inManagedObjectContext:context];
    [newManagedObject setValue:[NSDate date] forKey:@"timeStamp"];
    [newManagedObject setValue:[[currentItem valueForKey:@"title"] description] forKey:@"stripTitle"];
    [newManagedObject setValue:[[currentItem valueForKey:@"credits"] description] forKey:@"stripCredits"];
    [newManagedObject setValue:[[currentItem valueForKey:@"imageURL"] description] forKey:@"stripImgURL"];
    [newManagedObject setValue:[[currentItem valueForKey:@"link"] description] forKey:@"stripURL"];
    [newManagedObject setValue:pubDate forKey:@"stripPubDate"];
    [newManagedObject setValue:@"NO" forKey:@"stripBookmark"];
    //**THE NEW SYSTEM**
    NSString *destinationPath = [(AndyPadAppDelegate *)[[UIApplication sharedApplication] delegate] applicationDocumentsDirectory];
    NSString *guidPreview = [[NSProcessInfo processInfo] globallyUniqueString];
    NSString *guidImage = [[NSProcessInfo processInfo] globallyUniqueString];
    NSString *dpPreview = [destinationPath stringByAppendingPathComponent:guidPreview];
    NSString *dpImage = [destinationPath stringByAppendingPathComponent:guidImage];
    NSData *previewD = [NSData dataWithContentsOfURL:[NSURL URLWithString:previewURL]];
    NSData *imageD = [NSData dataWithContentsOfURL:[NSURL URLWithString:[currentItem valueForKey:@"imageURL"]]];
    //NSError *error = nil;
    [[NSFileManager defaultManager] createFileAtPath:dpPreview contents:previewD  attributes:nil];
    [[NSFileManager defaultManager] createFileAtPath:dpImage contents:imageD attributes:nil];
    //TODO: BETTER ERROR HANDLING WHEN COMPLETED APP
    [newManagedObject setValue:dpPreview forKey:@"stripLocalPreviewPath"];
    [newManagedObject setValue:dpImage forKey:@"stripLocalImagePath"];  
    [newManagedObject release];
    before++;
    [self.currentItem removeAllObjects];
    self.currentItem = nil;
    [self.currentItem release];
    [xmlParserPool drain];
    self.xmlParserPool = [[NSAutoreleasePool alloc] init];

}    

[EDIT] @Rog I tried playing with NSOperation but no luck so far, the same me,ory build up with the same items, all about 256k each. Here's the code:

- (void)sendToCoreDataStoreWithDate:(NSDate*)pubDate andPreview:(NSString*)previewURL
{
    NSManagedObjectContext *context = [fetchedResultsController managedObjectContext];
    NSEntityDescription *entity = [[fetchedResultsController fetchRequest] entity];

    newManagedObject = [NSEntityDescription insertNewObjectForEntityForName:[entity name] inManagedObjectContext:context];


    [newManagedObject setValue:[NSDate date] forKey:@"timeStamp"];
    [newManagedObject setValue:[[currentItem valueForKey:@"title"] description] forKey:@"stripTitle"];
    [newManagedObject setValue:[[currentItem valueForKey:@"credits"] description] forKey:@"stripCredits"];
    [newManagedObject setValue:[[currentItem valueForKey:@"imageURL"] description] forKey:@"stripImgURL"];
    [newManagedObject setValue:[[currentItem valueForKey:@"link"] description] forKey:@"stripURL"];
    [newManagedObject setValue:pubDate forKey:@"stripPubDate"];
    [newManagedObject setValue:@"NO" forKey:@"stripBookmark"];

NSString *destinationPath = [(AndyPadAppDelegate *)[[UIApplication sharedApplication] delegate] applicationDocumentsDirectory];
NSString *guidPreview = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *guidImage = [[NSProcessInfo processInfo] globallyUniqueString];
NSString *dpPreview = [destinationPath stringByAppendingPathComponent:guidPreview];
NSString *dpImage = [destinationPath stringByAppendingPathComponent:guidImage];

//Create an array and send the contents off to be dispatched to an operation queue
NSArray *previewArray = [NSArray arrayWithObjects:dpPreview, previewURL, nil];
NSOperation *prevOp = [self taskWithData:previewArray];
[prevOp start];

//Create an array and send the contents off to be dispatched to an operation queue
NSArray *imageArray = [NSArray arrayWithObjects:dpImage, [currentItem valueForKey:@"imageURL"], nil];
NSOperation *imagOp = [self taskWithData:imageArray];
[imagOp start];

[newManagedObject setValue:dpPreview forKey:@"stripLocalPreviewPath"];
[newManagedObject setValue:dpImage forKey:@"stripLocalImagePath"];  
//[newManagedObject release]; **recommended by stackoverflow answer
before++;
[self.currentItem removeAllObjects];
self.currentItem = nil;
[self.currentItem release];
[xmlParserPool drain];
self.xmlParserPool = [[NSAutoreleasePool alloc] init];


}

- (NSOperation*)taskWithData:(id)data
{
    NSInvocationOperation* theOp = [[[NSInvocationOperation alloc] initWithTarget:self
                                                                         selector:@selector(threadedImageDownload:) 
                                                                           object:data] autorelease];
    return theOp;
}

- (void)threadedImageDownload:(id)data 
{
    NSURL *urlFromArray1 = [NSURL URLWithString:[data objectAtIndex:1]];
    NSString *stringFromArray0 = [NSString stringWithString:[data objectAtIndex:0]];
    NSData *imageFile = [NSData dataWithContentsOfURL:urlFromArray1];
    [[NSFileManager defaultManager] createFileAtPath:stringFromArray0 
                                            contents:imageFile 
                                    开发者_如何学C      attributes:nil];
    //-=====0jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjmn **OLEG**
}

You can disable the caching :

NSURLCache *sharedCache = [[NSURLCache alloc] initWithMemoryCapacity:0 diskCapacity:0 diskPath:nil];
[NSURLCache setSharedURLCache:sharedCache];
[sharedCache release];

Or clear it :

[[NSURLCache sharedURLCache] removeAllCachedResponses];

This should fix your problem.

Take a look at AQXMLParser. There is a example app that shows how to integrate it with Core Data on a background thread. It's a streaming parser so memory usage is minimal.

Set up a second MOC and fetched results controller for the UI and refetch when the parsing is complete to get the new data. (one option is to register for change notifications when the background MOC saves to merge between contexts)

For images, there are a number of third party categories on UIImageVew that support asynchronous downloads and placeholder images. So you would parse the image URL, store it in Core Data, then set the URL on the image view when that item is viewed. This way the user doesn't need to wait for all the images to be downloaded.

Your problem is here:

[newManagedObject release];

Get rid of it, and the crashes will be gone (you don't release NSManagedObjects, Coredata will handle that for you).

Cheers,

Rog