Avoiding GC thrashing with WSE 3.0 MTOM service

For historical reasons, I have some WSE 3.0 web services that I cannot upgrade to WCF on the server side yet (it is also a substantial amount of work to do so).

These web services are being used for file transfers from client to server, using MTOM encoding. This can also not be changed in the short term, for reasons of compatibility. Secondly, they are being called from both Java and .NET, and therefore need to be cross-platform, hence MTOM.

How it works is that an "upload" WebMethod is called by the client, sending up a chunk of data at a time, since files being transferred could potentially be gigabytes in size.

However, due to not being able to control parts of the开发者_StackOverflow stack before the WebMethod is invoked, I cannot control the memory usage patterns of the web service.

The problem I am running into is for file sizes from 50MB or so onwards, performance is absolutely killed because of GC, since it appears that WSE 3.0 buffers each chunk received from the client in a new byte[] array, and by the time we've done 50MB we're spending 20-30% of time doing GC.

I've played with various chunk sizes, from 16k to 2MB, with no real great difference in results.

Smaller chunks are killed by the latency involved with round-tripping, and larger chunks just postpone the slowdown until GC kicks in.

Any bright ideas on cutting down on the garbage created by WSE? Can I plug into the pipeline somehow and jury-rig something that has access to the client's request stream and streams it to the WebMethod?

I'm aware that it is possible to "stream" responses to the client using WSE (albeit very ugly), but this problem is with requests from the client.

You are done. Gigabyte transfer and WSE 3.0 never worked together - you basically NEED streaming in WCF for that.

You can just try to go 64 bit with the process and load up tons of memory so that the GC is not so much an issue. Upgrade to .NET 4.0 (should be painless mostly) and use the non-blocking GC. Hit the process with 12-30gb memory and you should survive a little more.

Historically all ASP (.net) stuff is "batched", first colecting all data then sending it. THis means large items just are not handled too well.

As a workaround, after some testing, it appears possible to have both WCF services and old-style ASMX Web Service in the same web application, both with .asmx extensions, which would let me implement the file transfer web service in WCF using streaming, and leave the rest of the services untouched, and preserve the original URI people are using to connect.

It requires some real ugly BuildProvider and IHttpHandler hacking, but it does work.

In short:

1. You implement a proxy build provider that inspects the .asmx file to determine whether it is a WebService or ServiceHost declaration. You then call the appropriate methods on the actual build provider in question (System.ServiceModel.Activation.ServiceBuildProvider or System.Web.Compilation.WebServiceBuildProvider). Note that you have to instantiate the target build providers using reflection, as they are internal. You also have call some internal BuildProvider methods to emulate what BuildManager does.

2. You implement an IHttpHandlerFactory that creates either a System.ServiceModel.Activation.HttpHandler using reflection (since it's also internal), or uses the convenient, already public System.Web.Services.Protocols.WebServiceHandlerFactory to create a legacy ASMX IHttpHandler.

3. You configure them as follows (this assumes you are using non-Integrated mode on IIS7):

<?xml version="1.0"?>
<system.web>
    <compilation debug="true">
      <buildProviders>
        <remove extension=".asmx" />
        <add extension=".asmx" type="TestService.AsmxWcfSwitchingBuildProvider, TestService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"/>
      </buildProviders>
    </compilation>
    <httpHandlers>
      <remove verb="*" path="*.asmx"/>
      <add path="*.asmx" verb="*" type="TestService.AsmxWcfSwitchingHttpHandlerFactory, TestService, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" validate="false"/>
    </httpHandlers>
  </system.web>

4. Navigate to a legacy ASMX page. Verify that you can see the default page and WSDL.

5. Navigate to a WCF ASMX page. Verify that you can see the default page and WSDL.

Making sure you have parity with the original WSDL, and making sure your SOAP actions are correct, etc, is left as an exercise for the reader :)