Should I use Request.UrlReferrer when determining referrals

I came upon an interesting discussion with my team around the use of HttpRequest.UrlReferrer and wanted to solicit feedback from the commu开发者_如何学运维nity. According to the W3C spec:

The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained (the "referrer", although the header field is misspelled.) The Referer request-header allows a server to generate lists of back-links to resources for interest, logging, optimized caching, etc. It also allows obsolete or mistyped links to be traced for maintenance. The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.as input from the user keyboard.

The Request.UrlReferrer object does the work of converting referral strings that contain well formed URIs to an object with properties on every request.

According to our logs there are requests that come in that contain invalid data in the referral such as:

localhost
app:/BeamBackTest.swf
app:/multtiple.swf
app:/AFriendFeed.swf
ALToolBar
app:/index.html
mhtml:file://C:\Documents+and+Settings\User\Desktop\oracle\What+is+a+View+in+Oracle+-+Stack+Overflow.mht

Using Request.UrlReferrer would mean the above cases would be NULL. Is it better to discard the invalid data based on the W3C spec by using Request.UrlReferrer or preserve it by using Request.ServerVariables["HTTP_REFERER"] even though the data may be interesting, but potentially useless.

Well, it depends on what you're doing with the data. If you're doing something that requires a valid URI, then you might as well use Request.UrlReferrer since you have to strip it down anyway. But if all you're doing is logging it, and your logging tool can handle weird things, I'd recommend the second method- especially for the app:/ data, which is likely to yield useful patterns.

Its subjective based on your needs. We store everything even if its junk. One day we might know how to process it better and it won't be junk anymore. If we didn't store it because we didn't understand it at the time we'd never be able to get it back.