.Net WCF RIA Services: To page or not to page? and how much data to send down the pipe

So I was having a debate with my co worker about WCF RIA Services, and exactly how much data we send to the client in a given request. Here is the setup:

We are creating a reporting application in Silverlight and grabbing data from a WCF RIA Service to populate various grids and charts. We have two differing ideas as to how to implement the service. There are a few caveats which are important for making this decision.

Caveat 1: A "Report" could be any number of tables/charts, but most likely will consist of only a few each.

Caveat 2: The full size of a data table could be thousands of rows.

Caveat 3: Some tables will contain 50+ columns (not our idea...). The users would need the ability to show hide columns, because honestly, who could assimilate that much data all at once?

We have two different ideas:

1. Create a Data Contract for each table/chart and a separate GetTableX()/GetChartX() service method for each. We would use a Domain Service for this so filtering/sorting/paging could be handled on the server, and only a certain number of objects would be returned.

   Because a "Report" could contain n tables/charts, this would mean sending n requests each time a Report is loaded, and an additional request for each page on a table.

   The service would also be somewhat verbose, 15 reports (which will inevitably grow), with perhaps 3-5 methods per report could mean 45 to 75 methods.

   Because users can hide certain columns, and some will be hidden by default, there could be a fair amount of data sent to the client that isn't event viewed. As far as I know there isn't a way to limit this with a DomainService.

2. Create a Report object for each repo开发者_开发问答rt, with a collection of grids/charts, along with the data for each chart/table, filtered on criteria universal to each report. Serialize and pass the report object back to the client, letting Silverlight handle paging/sorting/filtering.

   This would mean significantly fewer request to the server, but the message payload would also be greater.

   I'm not sure, as I haven't investigated enough, but we would probably have to dig deep into WCF RIA Services to get this to work, or we may be better off scraping RIA and doing our own thing on top of WCF. However, this would give us the flexibility to choose which columns need to be sent to the browser in the hide/show columns scenario, which would lower the overall payload going across the line.

tl;dr = Is it better to send all your data across the wire once, or to page it?

1. It seems the answer strongly depends on how much data, but how do you guys make that decision?

2. How much data is to much to send in a request? When does paging become more efficient? or at least superficially more efficient to the user?**

3. From a business perspective, when is it better to just use the built in tools rather than rolling your own. Suppose your own would be more efficient, how do you weigh the value of that efficiency against the extra work?

In our app we sometimes make RIA calls that would returns as much as a 10Mb of data, but worked in the simplest sense, but has really come to bite us in the butt when we deployed because there are all kinds of limits in the whole Web Server <-> HTTP <--> browser stack that have caused issues. I'm talking about timeouts, buffer sizes, max request sizes, etc. Incidentally, it's not a simple matter to increase the timeouts for RIA calls.

To ease these pains, we started fetching large calls in RIA in smaller chunks. We'd have one server side query that when executed returns all of the primary keys of the desired rows (eg. a 10000 guids). Then on the client, we use RIA to fetch the objects in say chunks of 500 at a time, by the primary key.

The biggest gain this has given us is in the user experience, because we can display a progress bar for how much of the data is loaded. Chunking is also useful because it keeps the server a little more responsive. Loading 10mb of data in a single call occupies the server exclusively, but loading in several chunks allows resources to be shared a bit more smoothly.

Hope that helps.