Sanitizing strings to make them URL and filename safe?
I am trying to come up with a function that does a good job of sanitizing certain strings so that they are safe to use in the URL (like a post slug) and also safe to use as file names. For example, when someone uploads a file I want to make sure that I remove all dangerous characters from the name.
So far I have come up with the following function which I hope solves this problem and allows foreign UTF-8 data also.
/**
* Convert a string to the file/URL safe "slug" form
*
* @param string $string the string to clean
* @param bool $is_filename TRUE will allow additional filename characters
* @return string
*/
function sanitize($string = '', $is_filename = FALSE)
{
// Replace all weird characters with dashes
$string = preg_replace('/[^\w\-'. ($is_filename ? '~_\.' : ''). ']+/u', '-', $string);
// Only allow one dash separator at a time (and make string lowercase)
return mb_strtolower(preg_replace('/--+/u', '-', $string), 'UTF-8');
}
Does anyone have any tricky sample data I can run aga开发者_运维技巧inst this - or know of a better way to safeguard our apps from bad names?
$is-filename allows some additional characters like temp vim files
update: removed the star character since I could not think of a valid use
I found this larger function in the Chyrp code:
/**
* Function: sanitize
* Returns a sanitized string, typically for URLs.
*
* Parameters:
* $string - The string to sanitize.
* $force_lowercase - Force the string to lowercase?
* $anal - If set to *true*, will remove all non-alphanumeric characters.
*/
function sanitize($string, $force_lowercase = true, $anal = false) {
$strip = array("~", "`", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")", "_", "=", "+", "[", "{", "]",
"}", "\\", "|", ";", ":", "\"", "'", "‘", "’", "“", "”", "–", "—",
"—", "–", ",", "<", ".", ">", "/", "?");
$clean = trim(str_replace($strip, "", strip_tags($string)));
$clean = preg_replace('/\s+/', "-", $clean);
$clean = ($anal) ? preg_replace("/[^a-zA-Z0-9]/", "", $clean) : $clean ;
return ($force_lowercase) ?
(function_exists('mb_strtolower')) ?
mb_strtolower($clean, 'UTF-8') :
strtolower($clean) :
$clean;
}
and this one in the wordpress code
/**
* Sanitizes a filename replacing whitespace with dashes
*
* Removes special characters that are illegal in filenames on certain
* operating systems and special characters requiring special escaping
* to manipulate at the command line. Replaces spaces and consecutive
* dashes with a single dash. Trim period, dash and underscore from beginning
* and end of filename.
*
* @since 2.1.0
*
* @param string $filename The filename to be sanitized
* @return string The sanitized filename
*/
function sanitize_file_name( $filename ) {
$filename_raw = $filename;
$special_chars = array("?", "[", "]", "/", "\\", "=", "<", ">", ":", ";", ",", "'", "\"", "&", "$", "#", "*", "(", ")", "|", "~", "`", "!", "{", "}");
$special_chars = apply_filters('sanitize_file_name_chars', $special_chars, $filename_raw);
$filename = str_replace($special_chars, '', $filename);
$filename = preg_replace('/[\s-]+/', '-', $filename);
$filename = trim($filename, '.-_');
return apply_filters('sanitize_file_name', $filename, $filename_raw);
}
Update Sept 2012
Alix Axel has done some incredible work in this area. His phunction framework includes several great text filters and transformations.
- Unaccent
- Slug
- Filter
Some observations on your solution:
- 'u' at the end of your pattern means that the pattern, and not the text it's matching will be interpreted as UTF-8 (I presume you assumed the latter?).
- \w matches the underscore character. You specifically include it for files which leads to the assumption that you don't want them in URLs, but in the code you have URLs will be permitted to include an underscore.
- The inclusion of "foreign UTF-8" seems to be locale-dependent. It's not clear whether this is the locale of the server or client. From the PHP docs:
A "word" character is any letter or digit or the underscore character, that is, any character which can be part of a Perl "word". The definition of letters and digits is controlled by PCRE's character tables, and may vary if locale-specific matching is taking place. For example, in the "fr" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.
Creating the slug
You probably shouldn't include accented etc. characters in your post slug since, technically, they should be percent encoded (per URL encoding rules) so you'll have ugly looking URLs.
So, if I were you, after lowercasing, I'd convert any 'special' characters to their equivalent (e.g. é -> e) and replace non [a-z] characters with '-', limiting to runs of a single '-' as you've done. There's an implementation of converting special characters here: https://web.archive.org/web/20130208144021/http://neo22s.com/slug
Sanitization in general
OWASP have a PHP implementation of their Enterprise Security API which among other things includes methods for safe encoding and decoding input and output in your application.
The Encoder interface provides:
canonicalize (string $input, [bool $strict = true])
decodeFromBase64 (string $input)
decodeFromURL (string $input)
encodeForBase64 (string $input, [bool $wrap = false])
encodeForCSS (string $input)
encodeForHTML (string $input)
encodeForHTMLAttribute (string $input)
encodeForJavaScript (string $input)
encodeForOS (Codec $codec, string $input)
encodeForSQL (Codec $codec, string $input)
encodeForURL (string $input)
encodeForVBScript (string $input)
encodeForXML (string $input)
encodeForXMLAttribute (string $input)
encodeForXPath (string $input)
https://github.com/OWASP/PHP-ESAPI https://www.owasp.org/index.php/Category:OWASP_Enterprise_Security_API
This should make your filenames safe...
$string = preg_replace(array('/\s/', '/\.[\.]+/', '/[^\w_\.\-]/'), array('_', '.', ''), $string);
and a deeper solution to this is:
// Remove special accented characters - ie. sí.
$clean_name = strtr($string, array('Š' => 'S','Ž' => 'Z','š' => 's','ž' => 'z','Ÿ' => 'Y','À' => 'A','Á' => 'A','Â' => 'A','Ã' => 'A','Ä' => 'A','Å' => 'A','Ç' => 'C','È' => 'E','É' => 'E','Ê' => 'E','Ë' => 'E','Ì' => 'I','Í' => 'I','Î' => 'I','Ï' => 'I','Ñ' => 'N','Ò' => 'O','Ó' => 'O','Ô' => 'O','Õ' => 'O','Ö' => 'O','Ø' => 'O','Ù' => 'U','Ú' => 'U','Û' => 'U','Ü' => 'U','Ý' => 'Y','à' => 'a','á' => 'a','â' => 'a','ã' => 'a','ä' => 'a','å' => 'a','ç' => 'c','è' => 'e','é' => 'e','ê' => 'e','ë' => 'e','ì' => 'i','í' => 'i','î' => 'i','ï' => 'i','ñ' => 'n','ò' => 'o','ó' => 'o','ô' => 'o','õ' => 'o','ö' => 'o','ø' => 'o','ù' => 'u','ú' => 'u','û' => 'u','ü' => 'u','ý' => 'y','ÿ' => 'y'));
$clean_name = strtr($clean_name, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u'));
$clean_name = preg_replace(array('/\s/', '/\.[\.]+/', '/[^\w_\.\-]/'), array('_', '.', ''), $clean_name);
This assumes that you want a dot in the filename. if you want it transferred to lowercase, just use
$clean_name = strtolower($clean_name);
for the last line.
Try this:
function normal_chars($string)
{
$string = htmlentities($string, ENT_QUOTES, 'UTF-8');
$string = preg_replace('~&([a-z]{1,2})(acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml);~i', '$1', $string);
$string = html_entity_decode($string, ENT_QUOTES, 'UTF-8');
$string = preg_replace(array('~[^0-9a-z]~i', '~[ -]+~'), ' ', $string);
return trim($string, ' -');
}
Examples:
echo normal_chars('Álix----_Ãxel!?!?'); // Alix Axel
echo normal_chars('áéíóúÁÉÍÓÚ'); // aeiouAEIOU
echo normal_chars('üÿÄËÏÖÜŸåÅ'); // uyAEIOUYaA
Based on the selected answer in this thread: URL Friendly Username in PHP?
This isn't exactly an answer as it doesn't provide any solutions (yet!), but it's too big to fit on a comment...
I did some testing (regarding file names) on Windows 7 and Ubuntu 12.04 and what I found out was that:
1. PHP Can't Handle non-ASCII Filenames
Although both Windows and Ubuntu can handle Unicode filenames (even RTL ones as it seems) PHP 5.3 requires hacks to deal even with the plain old ISO-8859-1, so it's better to keep it ASCII only for safety.
2. The Lenght of the Filename Matters (Specially on Windows)
On Ubuntu, the maximum length a filename can have (incluinding extension) is 255 (excluding path):
/var/www/uploads/123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345/
However, on Windows 7 (NTFS) the maximum lenght a filename can have depends on it's absolute path:
(0 + 0 + 244 + 11 chars) C:\1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234\1234567.txt
(0 + 3 + 240 + 11 chars) C:\123\123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890\1234567.txt
(3 + 3 + 236 + 11 chars) C:\123\456\12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456\1234567.txt
Wikipedia says that:
NTFS allows each path component (directory or filename) to be 255 characters long.
To the best of my knowledge (and testing), this is wrong.
In total (counting slashes) all these examples have 259 chars, if you strip the C:\ that gives 256 characters (not 255?!). The directories where created using the Explorer and you'll notice that it restrains itself from using all the available space for the directory name. The reason for this is to allow the creation of files using the 8.3 file naming convention. The same thing happens for other partitions.
Files don't need to reserve the 8.3 lenght requirements of course:
(255 chars) E:\12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901.txt
You can't create any more sub-directories if the absolute path of the parent directory has more than 242 characters, because 256 = 242 + 1 + \ + 8 + . + 3. Using Windows Explorer, you can't create another directory if the parent directory has more than 233 characters (depending on the system locale), because 256 = 233 + 10 + \ + 8 + . + 3; the 10 here is the length of the string New folder.
Windows file system poses a nasty problem if you want to assure inter-operability between file systems.
3. Beware of Reserved Characters and Keywords
Aside from removing non-ASCII, non-printable and control characters, you also need to re(place/move):
"*/:<>?\|
Just removing these characters might not be the best idea because the filename might lose some of it's meaning. I think that, at the very least, multiple occurences of these characters should be replaced by a single underscore (_), or perhaps something more representative (this is just an idea):
"*?->_/\|->-:->[ ]-[ ]<->(>->)
There are also special keywords that should be avoided (like NUL), although I'm not sure how to overcome that. Perhaps a black list with a random name fallback would be a good approach to solve it.
4. Case Sensitiveness
This should go without saying, but if you want so ensure file uniqueness across different operating systems you should transform file names to a normalized case, that way my_file.txt and My_File.txt on Linux won't both become the same my_file.txt file on Windows.
5. Make Sure It's Unique
If the file name already exists, a unique identifier should be appended to it's base file name.
Common unique identifiers include the UNIX timestamp, a digest of the file contents or a random string.
6. Hidden Files
Just because it can be named doesn't mean it should...
Dots are usually white-listed in file names but in Linux a hidden file is represented by a leading dot.
7. Other Considerations
If you have to strip some chars of the file name, the extension is usually more important than the base name of the file. Allowing a considerable maximum number of characters for the file extension (8-16) one should strip the characters from the base name. It's also important to note that in the unlikely event of having a more than one long extension - such as _.graphmlz.tag.gz - _.graphmlz.tag only _ should be considered as the file base name in this case.
8. Resources
Calibre handles file name mangling pretty decently:
- /src/calibre/utils/filenames.py
- /src/calibre/library/save_to_disk.py
Wikipedia page on file name mangling and linked chapter from Using Samba.
If for instance, you try to create a file that violates any of the rules 1/2/3, you'll get a very useful error:
Warning: touch(): Unable to create file ... because No error in ... on line ...
I've always thought Kohana did a pretty good job of it.
public static function title($title, $separator = '-', $ascii_only = FALSE)
{
if ($ascii_only === TRUE)
{
// Transliterate non-ASCII characters
$title = UTF8::transliterate_to_ascii($title);
// Remove all characters that are not the separator, a-z, 0-9, or whitespace
$title = preg_replace('![^'.preg_quote($separator).'a-z0-9\s]+!', '', strtolower($title));
}
else
{
// Remove all characters that are not the separator, letters, numbers, or whitespace
$title = preg_replace('![^'.preg_quote($separator).'\pL\pN\s]+!u', '', UTF8::strtolower($title));
}
// Replace all separator characters and whitespace by a single separator
$title = preg_replace('!['.preg_quote($separator).'\s]+!u', $separator, $title);
// Trim separators from the beginning and end
return trim($title, $separator);
}
The handy UTF8::transliterate_to_ascii() will turn stuff like ñ => n.
Of course, you could replace the other UTF8::* stuff with mb_* functions.
I have adapted from another source and added a couple extra, maybe a little overkill
/**
* Convert a string into a url safe address.
*
* @param string $unformatted
* @return string
*/
public function formatURL($unformatted) {
$url = strtolower(trim($unformatted));
//replace accent characters, forien languages
$search = array('À', 'Á', 'Â', 'Ã', 'Ä', 'Å', 'Æ', 'Ç', 'È', 'É', 'Ê', 'Ë', 'Ì', 'Í', 'Î', 'Ï', 'Ð', 'Ñ', 'Ò', 'Ó', 'Ô', 'Õ', 'Ö', 'Ø', 'Ù', 'Ú', 'Û', 'Ü', 'Ý', 'ß', 'à', 'á', 'â', 'ã', 'ä', 'å', 'æ', 'ç', 'è', 'é', 'ê', 'ë', 'ì', 'í', 'î', 'ï', 'ñ', 'ò', 'ó', 'ô', 'õ', 'ö', 'ø', 'ù', 'ú', 'û', 'ü', 'ý', 'ÿ', 'Ā', 'ā', 'Ă', 'ă', 'Ą', 'ą', 'Ć', 'ć', 'Ĉ', 'ĉ', 'Ċ', 'ċ', 'Č', 'č', 'Ď', 'ď', 'Đ', 'đ', 'Ē', 'ē', 'Ĕ', 'ĕ', 'Ė', 'ė', 'Ę', 'ę', 'Ě', 'ě', 'Ĝ', 'ĝ', 'Ğ', 'ğ', 'Ġ', 'ġ', 'Ģ', 'ģ', 'Ĥ', 'ĥ', 'Ħ', 'ħ', 'Ĩ', 'ĩ', 'Ī', 'ī', 'Ĭ', 'ĭ', 'Į', 'į', 'İ', 'ı', 'IJ', 'ij', 'Ĵ', 'ĵ', 'Ķ', 'ķ', 'Ĺ', 'ĺ', 'Ļ', 'ļ', 'Ľ', 'ľ', 'Ŀ', 'ŀ', 'Ł', 'ł', 'Ń', 'ń', 'Ņ', 'ņ', 'Ň', 'ň', 'ʼn', 'Ō', 'ō', 'Ŏ', 'ŏ', 'Ő', 'ő', 'Œ', 'œ', 'Ŕ', 'ŕ', 'Ŗ', 'ŗ', 'Ř', 'ř', 'Ś', 'ś', 'Ŝ', 'ŝ', 'Ş', 'ş', 'Š', 'š', 'Ţ', 'ţ', 'Ť', 'ť', 'Ŧ', 'ŧ', 'Ũ', 'ũ', 'Ū', 'ū', 'Ŭ', 'ŭ', 'Ů', 'ů', 'Ű', 'ű', 'Ų', 'ų', 'Ŵ', 'ŵ', 'Ŷ', 'ŷ', 'Ÿ', 'Ź', 'ź', 'Ż', 'ż', 'Ž', 'ž', 'ſ', 'ƒ', 'Ơ', 'ơ', 'Ư', 'ư', 'Ǎ', 'ǎ', 'Ǐ', 'ǐ', 'Ǒ', 'ǒ', 'Ǔ', 'ǔ', 'Ǖ', 'ǖ', 'Ǘ', 'ǘ', 'Ǚ', 'ǚ', 'Ǜ', 'ǜ', 'Ǻ', 'ǻ', 'Ǽ', 'ǽ', 'Ǿ', 'ǿ');
$replace = array('A', 'A', 'A', 'A', 'A', 'A', 'AE', 'C', 'E', 'E', 'E', 'E', 'I', 'I', 'I', 'I', 'D', 'N', 'O', 'O', 'O', 'O', 'O', 'O', 'U', 'U', 'U', 'U', 'Y', 's', 'a', 'a', 'a', 'a', 'a', 'a', 'ae', 'c', 'e', 'e', 'e', 'e', 'i', 'i', 'i', 'i', 'n', 'o', 'o', 'o', 'o', 'o', 'o', 'u', 'u', 'u', 'u', 'y', 'y', 'A', 'a', 'A', 'a', 'A', 'a', 'C', 'c', 'C', 'c', 'C', 'c', 'C', 'c', 'D', 'd', 'D', 'd', 'E', 'e', 'E', 'e', 'E', 'e', 'E', 'e', 'E', 'e', 'G', 'g', 'G', 'g', 'G', 'g', 'G', 'g', 'H', 'h', 'H', 'h', 'I', 'i', 'I', 'i', 'I', 'i', 'I', 'i', 'I', 'i', 'IJ', 'ij', 'J', 'j', 'K', 'k', 'L', 'l', 'L', 'l', 'L', 'l', 'L', 'l', 'l', 'l', 'N', 'n', 'N', 'n', 'N', 'n', 'n', 'O', 'o', 'O', 'o', 'O', 'o', 'OE', 'oe', 'R', 'r', 'R', 'r', 'R', 'r', 'S', 's', 'S', 's', 'S', 's', 'S', 's', 'T', 't', 'T', 't', 'T', 't', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'W', 'w', 'Y', 'y', 'Y', 'Z', 'z', 'Z', 'z', 'Z', 'z', 's', 'f', 'O', 'o', 'U', 'u', 'A', 'a', 'I', 'i', 'O', 'o', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'U', 'u', 'A', 'a', 'AE', 'ae', 'O', 'o');
$url = str_replace($search, $replace, $url);
//replace common characters
$search = array('&', '£', '$');
$replace = array('and', 'pounds', 'dollars');
$url= str_replace($search, $replace, $url);
// remove - for spaces and union characters
$find = array(' ', '&', '\r\n', '\n', '+', ',', '//');
$url = str_replace($find, '-', $url);
//delete and replace rest of special chars
$find = array('/[^a-z0-9\-<>]/', '/[\-]+/', '/<[^>]*>/');
$replace = array('', '-', '');
$uri = preg_replace($find, $replace, $url);
return $uri;
}
In terms of file uploads, you would be safest to prevent the user from controlling the file name. As has already been hinted at, store the canonicalised filename in a database along with a randomly chosen and unique name which you'll use as the actual filename.
Using OWASP ESAPI, these names could be generated thus:
$userFilename = ESAPI::getEncoder()->canonicalize($input_string);
$safeFilename = ESAPI::getRandomizer()->getRandomFilename();
You could append a timestamp to the $safeFilename to help ensure that the randomly generated filename is unique without even checking for an existing file.
In terms of encoding for URL, and again using ESAPI:
$safeForURL = ESAPI::getEncoder()->encodeForURL($input_string);
This method performs canonicalisation before encoding the string and will handle all character encodings.
I recommend* URLify for PHP (480+ stars on Github) - "the PHP port of URLify.js from the Django project. Transliterates non-ascii characters for use in URLs".
Basic usage:
To generate slugs for URLs:
<?php
echo URLify::filter (' J\'étudie le français ');
// "jetudie-le-francais"
echo URLify::filter ('Lo siento, no hablo español.');
// "lo-siento-no-hablo-espanol"
?>
To generate slugs for file names:
<?php
echo URLify::filter ('фото.jpg', 60, "", true);
// "foto.jpg"
?>
*None of the other suggestions matched my criteria:
- Should be installable via composer
- Should not depend on iconv since it behaves differently on different systems
- Should be extendable to allow overrides and custom character replacements
- Popular (for instance many stars on Github)
- Has tests
As a bonus, URLify also removes certain words and strips away all characters not transliterated.
Here is a test case with tons of foreign characters being transliterated properly using URLify: https://gist.github.com/motin/a65e6c1cc303e46900d10894bf2da87f
and this is Joomla 3.3.2 version from JFile::makeSafe($file)
public static function makeSafe($file)
{
// Remove any trailing dots, as those aren't ever valid file names.
$file = rtrim($file, '.');
$regex = array('#(\.){2,}#', '#[^A-Za-z0-9\.\_\- ]#', '#^\.#');
return trim(preg_replace($regex, '', $file));
}
Depending on how you will use it, you might want to add a length limit to protect against buffer overflows.
I don't think having a list of chars to remove is safe. I would rather use the following:
For filenames: Use an internal ID or a hash of the filecontent. Save the document name in a database. This way you can keep the original filename and still find the file.
For url parameters: Use urlencode() to encode any special characters.
This is a nice way to secure an upload filename:
$file_name = trim(basename(stripslashes($name)), ".\x00..\x20");
Here's CodeIgniter's implementation.
/**
* Sanitize Filename
*
* @param string $str Input file name
* @param bool $relative_path Whether to preserve paths
* @return string
*/
public function sanitize_filename($str, $relative_path = FALSE)
{
$bad = array(
'../', '<!--', '-->', '<', '>',
"'", '"', '&', '$', '#',
'{', '}', '[', ']', '=',
';', '?', '%20', '%22',
'%3c', // <
'%253c', // <
'%3e', // >
'%0e', // >
'%28', // (
'%29', // )
'%2528', // (
'%26', // &
'%24', // $
'%3f', // ?
'%3b', // ;
'%3d' // =
);
if ( ! $relative_path)
{
$bad[] = './';
$bad[] = '/';
}
$str = remove_invisible_characters($str, FALSE);
return stripslashes(str_replace($bad, '', $str));
}
And the remove_invisible_characters dependency.
function remove_invisible_characters($str, $url_encoded = TRUE)
{
$non_displayables = array();
// every control character except newline (dec 10),
// carriage return (dec 13) and horizontal tab (dec 09)
if ($url_encoded)
{
$non_displayables[] = '/%0[0-8bcef]/'; // url encoded 00-08, 11, 12, 14, 15
$non_displayables[] = '/%1[0-9a-f]/'; // url encoded 16-31
}
$non_displayables[] = '/[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]+/S'; // 00-08, 11, 12, 14-31, 127
do
{
$str = preg_replace($non_displayables, '', $str, -1, $count);
}
while ($count);
return $str;
}
I have entry titles with all kinds of weird latin characters as well as some HTML tags that I needed to translate into a useful dash-delimited filename format. I combined @SoLoGHoST's answer with a couple of items from @Xeoncross's answer and customized a bit.
function sanitize($string,$force_lowercase=true) {
//Clean up titles for filenames
$clean = strip_tags($string);
$clean = strtr($clean, array('Š' => 'S','Ž' => 'Z','š' => 's','ž' => 'z','Ÿ' => 'Y','À' => 'A','Á' => 'A','Â' => 'A','Ã' => 'A','Ä' => 'A','Å' => 'A','Ç' => 'C','È' => 'E','É' => 'E','Ê' => 'E','Ë' => 'E','Ì' => 'I','Í' => 'I','Î' => 'I','Ï' => 'I','Ñ' => 'N','Ò' => 'O','Ó' => 'O','Ô' => 'O','Õ' => 'O','Ö' => 'O','Ø' => 'O','Ù' => 'U','Ú' => 'U','Û' => 'U','Ü' => 'U','Ý' => 'Y','à' => 'a','á' => 'a','â' => 'a','ã' => 'a','ä' => 'a','å' => 'a','ç' => 'c','è' => 'e','é' => 'e','ê' => 'e','ë' => 'e','ì' => 'i','í' => 'i','î' => 'i','ï' => 'i','ñ' => 'n','ò' => 'o','ó' => 'o','ô' => 'o','õ' => 'o','ö' => 'o','ø' => 'o','ù' => 'u','ú' => 'u','û' => 'u','ü' => 'u','ý' => 'y','ÿ' => 'y'));
$clean = strtr($clean, array('Þ' => 'TH', 'þ' => 'th', 'Ð' => 'DH', 'ð' => 'dh', 'ß' => 'ss', 'Œ' => 'OE', 'œ' => 'oe', 'Æ' => 'AE', 'æ' => 'ae', 'µ' => 'u','—' => '-'));
$clean = str_replace("--", "-", preg_replace("/[^a-z0-9-]/i", "", preg_replace(array('/\s/', '/[^\w-\.\-]/'), array('-', ''), $clean)));
return ($force_lowercase) ?
(function_exists('mb_strtolower')) ?
mb_strtolower($clean, 'UTF-8') :
strtolower($clean) :
$clean;
}
I needed to manually add the em dash character (—) to the translation array. There may be others but so far my file names are looking good.
So:
Part 1: My dad’s “Žurburts”?—they’re (not) the best!
becomes:
part-1-my-dads-zurburts-theyre-not-the-best
I just add ".html" to the returned string.
why not simply use php's urlencode? it replaces "dangerous" characters with their hex representation for urls (i.e. %20 for a space)
There are already several solutions provided for this question but I have read and tested most of the code here and I ended up with this solution which is a mix of what I learned here:
The function
The function is bundled here in a Symfony2 bundle but it can be extracted to be used as plain PHP, it only has a dependency with the iconv function that must be enabled:
Filesystem.php:
<?php
namespace COil\Bundle\COilCoreBundle\Component\HttpKernel\Util;
use Symfony\Component\HttpKernel\Util\Filesystem as BaseFilesystem;
/**
* Extends the Symfony filesystem object.
*/
class Filesystem extends BaseFilesystem
{
/**
* Make a filename safe to use in any function. (Accents, spaces, special chars...)
* The iconv function must be activated.
*
* @param string $fileName The filename to sanitize (with or without extension)
* @param string $defaultIfEmpty The default string returned for a non valid filename (only special chars or separators)
* @param string $separator The default separator
* @param boolean $lowerCase Tells if the string must converted to lower case
*
* @author COil <https://github.com/COil>
* @see http://stackoverflow.com/questions/2668854/sanitizing-strings-to-make-them-url-and-filename-safe
*
* @return string
*/
public function sanitizeFilename($fileName, $defaultIfEmpty = 'default', $separator = '_', $lowerCase = true)
{
// Gather file informations and store its extension
$fileInfos = pathinfo($fileName);
$fileExt = array_key_exists('extension', $fileInfos) ? '.'. strtolower($fileInfos['extension']) : '';
// Removes accents
$fileName = @iconv('UTF-8', 'us-ascii//TRANSLIT', $fileInfos['filename']);
// Removes all characters that are not separators, letters, numbers, dots or whitespaces
$fileName = preg_replace("/[^ a-zA-Z". preg_quote($separator). "\d\.\s]/", '', $lowerCase ? strtolower($fileName) : $fileName);
// Replaces all successive separators into a single one
$fileName = preg_replace('!['. preg_quote($separator).'\s]+!u', $separator, $fileName);
// Trim beginning and ending seperators
$fileName = trim($fileName, $separator);
// If empty use the default string
if (empty($fileName)) {
$fileName = $defaultIfEmpty;
}
return $fileName. $fileExt;
}
}
The unit tests
What is interesting is that I have created PHPUnit tests, first to test edge cases and so you can check if it fits your needs: (If you find a bug, feel free to add a test case)
FilesystemTest.php:
<?php
namespace COil\Bundle\COilCoreBundle\Tests\Unit\Helper;
use COil\Bundle\COilCoreBundle\Component\HttpKernel\Util\Filesystem;
/**
* Test the Filesystem custom class.
*/
class FilesystemTest extends \PHPUnit_Framework_TestCase
{
/**
* test sanitizeFilename()
*/
public function testFilesystem()
{
$fs = new Filesystem();
$this->assertEquals('logo_orange.gif', $fs->sanitizeFilename('--logö _ __ ___ ora@@ñ--~gé--.gif'), '::sanitizeFilename() handles complex filename with specials chars');
$this->assertEquals('coilstack', $fs->sanitizeFilename('cOiLsTaCk'), '::sanitizeFilename() converts all characters to lower case');
$this->assertEquals('cOiLsTaCk', $fs->sanitizeFilename('cOiLsTaCk', 'default', '_', false), '::sanitizeFilename() lower case can be desactivated, passing false as the 4th argument');
$this->assertEquals('coil_stack', $fs->sanitizeFilename('coil stack'), '::sanitizeFilename() convert a white space to a separator');
$this->assertEquals('coil-stack', $fs->sanitizeFilename('coil stack', 'default', '-'), '::sanitizeFilename() can use a different separator as the 3rd argument');
$this->assertEquals('coil_stack', $fs->sanitizeFilename('coil stack'), '::sanitizeFilename() removes successive white spaces to a single separator');
$this->assertEquals('coil_stack', $fs->sanitizeFilename(' coil stack'), '::sanitizeFilename() removes spaces at the beginning of the string');
$this->assertEquals('coil_stack', $fs->sanitizeFilename('coil stack '), '::sanitizeFilename() removes spaces at the end of the string');
$this->assertEquals('coilstack', $fs->sanitizeFilename('coil,,,,,,stack'), '::sanitizeFilename() removes non-ASCII characters');
$this->assertEquals('coil_stack', $fs->sanitizeFilename('coil_stack '), '::sanitizeFilename() keeps separators');
$this->assertEquals('coil_stack', $fs->sanitizeFilename(' coil________stack'), '::sanitizeFilename() converts successive separators into a single one');
$this->assertEquals('coil_stack.gif', $fs->sanitizeFilename('cOil Stack.GiF'), '::sanitizeFilename() lower case filename and extension');
$this->assertEquals('copy_of_coil.stack.exe', $fs->sanitizeFilename('Copy of coil.stack.exe'), '::sanitizeFilename() keeps dots before the extension');
$this->assertEquals('default.doc', $fs->sanitizeFilename('____________.doc'), '::sanitizeFilename() returns a default file name if filename only contains special chars');
$this->assertEquals('default.docx', $fs->sanitizeFilename(' ___ - --_ __%%%%__¨¨¨***____ .docx'), '::sanitizeFilename() returns a default file name if filename only contains special chars');
$this->assertEquals('logo_edition_1314352521.jpg', $fs->sanitizeFilename('logo_edition_1314352521.jpg'), '::sanitizeFilename() returns the filename untouched if it does not need to be modified');
$userId = rand(1, 10);
$this->assertEquals('user_doc_'. $userId. '.doc', $fs->sanitizeFilename('亐亐亐亐亐.doc', 'user_doc_'. $userId), '::sanitizeFilename() returns the default string (the 2nd argument) if it can\'t be sanitized');
}
}
The test results: (checked on Ubuntu with PHP 5.3.2 and MacOsX with PHP 5.3.17:
All tests pass:
phpunit -c app/ src/COil/Bundle/COilCoreBundle/Tests/Unit/Helper/FilesystemTest.php
PHPUnit 3.6.10 by Sebastian Bergmann.
Configuration read from /var/www/strangebuzz.com/app/phpunit.xml.dist
.
Time: 0 seconds, Memory: 5.75Mb
OK (1 test, 17 assertions)
Solution #1: You have ability to install PHP extensions on server (hosting)
For transliteration of "almost every single language on the planet Earth" to ASCII characters.
Install PHP Intl extension first. This is command for Debian (Ubuntu):
sudo aptitude install php5-intlThis is my fileName function (create test.php and paste there following code):
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Test</title>
</head>
<body>
<?php
function pr($string) {
print '<hr>';
print '"' . fileName($string) . '"';
print '<br>';
print '"' . $string . '"';
}
function fileName($string) {
// remove html tags
$clean = strip_tags($string);
// transliterate
$clean = transliterator_transliterate('Any-Latin;Latin-ASCII;', $clean);
// remove non-number and non-letter characters
$clean = str_replace('--', '-', preg_replace('/[^a-z0-9-\_]/i', '', preg_replace(array(
'/\s/',
'/[^\w-\.\-]/'
), array(
'_',
''
), $clean)));
// replace '-' for '_'
$clean = strtr($clean, array(
'-' => '_'
));
// remove double '__'
$positionInString = stripos($clean, '__');
while ($positionInString !== false) {
$clean = str_replace('__', '_', $clean);
$positionInString = stripos($clean, '__');
}
// remove '_' from the end and beginning of the string
$clean = rtrim(ltrim($clean, '_'), '_');
// lowercase the string
return strtolower($clean);
}
pr('_replace(\'~&([a-z]{1,2})(ac134/56f4315981743 8765475[]lt7ňl2ú5äňú138yé73ťž7ýľute|');
pr(htmlspecialchars('<script>alert(\'hacked\')</script>'));
pr('Álix----_Ãxel!?!?');
pr('áéíóúÁÉÍÓÚ');
pr('üÿÄËÏÖÜ.ŸåÅ');
pr('nie4č a a§ôňäääaš');
pr('Мао Цзэдун');
pr('毛泽东');
pr('ماو تسي تونغ');
pr('مائو تسهتونگ');
pr('מאו דזה-דונג');
pr('მაო ძედუნი');
pr('Mao Trạch Đông');
pr('毛澤東');
pr('เหมา เจ๋อตง');
?>
</body>
</html>This line is core:
// transliterate
$clean = transliterator_transliterate('Any-Latin;Latin-ASCII;', $clean);
Answer based on this post.
Solution #2: You don't have ability to install PHP extensions on server (hosting)
Pretty good job is done in transliteration module for CMS Drupal. It supports almost every single language on the planet Earth. I suggest to check plugin repository if you want to have really complete solution sanitizing strings.
This post seems to work the best among all that I have tied. http://gsynuh.com/php-string-filename-url-safe/205
This is a good function:
public function getFriendlyURL($string) {
setlocale(LC_CTYPE, 'en_US.UTF8');
$string = iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', $string);
$string = preg_replace('~[^\-\pL\pN\s]+~u', '-', $string);
$string = str_replace(' ', '-', $string);
$string = trim($string, "-");
$string = strtolower($string);
return $string;
}
This is the code used by Prestashop to sanitize urls :
replaceAccentedChars
is used by
str2url
to remove diacritics
function replaceAccentedChars($str)
{
$patterns = array(
/* Lowercase */
'/[\x{0105}\x{00E0}\x{00E1}\x{00E2}\x{00E3}\x{00E4}\x{00E5}]/u',
'/[\x{00E7}\x{010D}\x{0107}]/u',
'/[\x{010F}]/u',
'/[\x{00E8}\x{00E9}\x{00EA}\x{00EB}\x{011B}\x{0119}]/u',
'/[\x{00EC}\x{00ED}\x{00EE}\x{00EF}]/u',
'/[\x{0142}\x{013E}\x{013A}]/u',
'/[\x{00F1}\x{0148}]/u',
'/[\x{00F2}\x{00F3}\x{00F4}\x{00F5}\x{00F6}\x{00F8}]/u',
'/[\x{0159}\x{0155}]/u',
'/[\x{015B}\x{0161}]/u',
'/[\x{00DF}]/u',
'/[\x{0165}]/u',
'/[\x{00F9}\x{00FA}\x{00FB}\x{00FC}\x{016F}]/u',
'/[\x{00FD}\x{00FF}]/u',
'/[\x{017C}\x{017A}\x{017E}]/u',
'/[\x{00E6}]/u',
'/[\x{0153}]/u',
/* Uppercase */
'/[\x{0104}\x{00C0}\x{00C1}\x{00C2}\x{00C3}\x{00C4}\x{00C5}]/u',
'/[\x{00C7}\x{010C}\x{0106}]/u',
'/[\x{010E}]/u',
'/[\x{00C8}\x{00C9}\x{00CA}\x{00CB}\x{011A}\x{0118}]/u',
'/[\x{0141}\x{013D}\x{0139}]/u',
'/[\x{00D1}\x{0147}]/u',
'/[\x{00D3}]/u',
'/[\x{0158}\x{0154}]/u',
'/[\x{015A}\x{0160}]/u',
'/[\x{0164}]/u',
'/[\x{00D9}\x{00DA}\x{00DB}\x{00DC}\x{016E}]/u',
'/[\x{017B}\x{0179}\x{017D}]/u',
'/[\x{00C6}]/u',
'/[\x{0152}]/u');
$replacements = array(
'a', 'c', 'd', 'e', 'i', 'l', 'n', 'o', 'r', 's', 'ss', 't', 'u', 'y', 'z', 'ae', 'oe',
'A', 'C', 'D', 'E', 'L', 'N', 'O', 'R', 'S', 'T', 'U', 'Z', 'AE', 'OE'
);
return preg_replace($patterns, $replacements, $str);
}
function str2url($str)
{
if (function_exists('mb_strtolower'))
$str = mb_strtolower($str, 'utf-8');
$str = trim($str);
if (!function_exists('mb_strtolower'))
$str = replaceAccentedChars($str);
// Remove all non-whitelist chars.
$str = preg_replace('/[^a-zA-Z0-9\s\'\:\/\[\]-\pL]/u', '', $str);
$str = preg_replace('/[\s\'\:\/\[\]-]+/', ' ', $str);
$str = str_replace(array(' ', '/'), '-', $str);
// If it was not possible to lowercase the string with mb_strtolower, we do it after the transformations.
// This way we lose fewer special chars.
if (!function_exists('mb_strtolower'))
$str = strtolower($str);
return $str;
}
There is 2 good answers to slugfy your data, use it https://stackoverflow.com/a/3987966/971619 or it https://stackoverflow.com/a/7610586/971619
// CLEAN ILLEGAL CHARACTERS
function clean_filename($source_file)
{
$search[] = " ";
$search[] = "&";
$search[] = "$";
$search[] = ",";
$search[] = "!";
$search[] = "@";
$search[] = "#";
$search[] = "^";
$search[] = "(";
$search[] = ")";
$search[] = "+";
$search[] = "=";
$search[] = "[";
$search[] = "]";
$replace[] = "_";
$replace[] = "and";
$replace[] = "S";
$replace[] = "_";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
$replace[] = "";
return str_replace($search,$replace,$source_file);
}
加载中,请稍侯......
精彩评论