Robust way to put contents of any arbitrary text file in the database (using Django/Python)?

As part of my Django app, I have to get the contents of a text file which a user uploads (which could be any charset) and save it to my DB. I keep running into issues (like having to remove UTF8's BOM manually, or having to figure out how to account for non-printable characters, or having to figure out how to make all unicode characters work - not just Latin ones, etc.) and each of these issues requires its own hack.

Is there a robust way to d开发者_开发技巧o this that doesn't require each of these case-by-case fixes? Right now I'm just using file.read() to get the contents, then doing all of those workarounds to clean the contents, and then using .save() to save it to the DB (I have a model for this).

What else can I be doing?

Causes some overhead, but you could base64 encode the entire string before persisting to the db. Then no escaping is required.

If you want to explicitly steer away from any issues with encoding and just see files as bunches of binary data (not strings of text in a specific encoding) you might want to use your database's binary format.

For MySQL this is BINARY and VARBINARY: http://dev.mysql.com/doc/refman/5.0/en/binary-varbinary.html

For a deeper understanding of unicode & utf-8 issues (recommended) this is a nice read on the subject: http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF