XML Cleansing
When writing XML, all text should be converted to UTF8 if it's not coming from a known source. This blog post provides a ruby function to do the conversion.
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
So I have an announcement to make: if you are a programmer working in 2003 and you don't know the basics of characters, character sets, encodings, and Unicode, and I catch you, I'm going to punish you by making you peel onions for 6 months in a submarine.
UTF8 encoded text file
useful for testing compatability.
ongoing · Characters vs. Bytes
Icelandic Character Encoding (and other Joys of PDF)
a brief intro to encodings in PDF
Secure UTF-8 Input in Rails - igvita.com
don't trust incoming UTF-8 input in rails
Tip #246 - Working with Unicode (the same, rewritten for legibility) : vim online
Tony Finch - no-longer-simple mail transport protocol
good overview of additions to the basic SMTP protocol. Explains header encoding, mime types, quoted printable, base64, etc
Apple's MacRoman character set and equivalent Unicode and HTML characters
RFC 3629 UTF-8, a transformation format of ISO 10646
UTF-8: in RFC form!
Unicode: Index of /Public/MAPPINGS
Mappings from a wide range of encodings into Unicode
Well, I'm Back: String Theory
UTF-16 is the devil's work. UTF-8 rules.
PDF Techniques for Web Content Accessibility Guidelines 1.0
Section "Ensure that all characters codes map reliably to Unicode." descibes an appropriate algorithm for converting all text in a pdf to unicode, regardless of encoding
SMS Tutorial: Introduction to SMS Messaging/Short Message Service, Concatenated SMS/Long SMS and EMS
the start of an interesting tutorial on SMS
A List Apart: Articles: The Trouble With EM ’n EN (and Other Shady Characters)
examines the different dash and space characters (among others) available in Unicode and when they should be used
Change end-of-line format for dos-mac-unix - Vim Tips Wiki - a Wikia wiki
Shades of Gray: Articles in Character Encodings
Series of blog posts by James Earl Gray II on character encodings and ruby
Shades of Gray: Bytes and Characters in Ruby 1.8
some excellent advice on using UTF-8 in ruby 1.8
Unicode 88
Early article on the options for a new text encoding scheme, tentatively called Unicode.
Encoding issues with Ruby 1.9 and Rails 2.3
rails 2.3 marks all active record attributes as encoded in "ASCII-8BIT", regardless of what is in database.yml.
#2476 ASCII-8BIT encoding of query results in rails 2.3.2 and ruby 1.9.1 - Ruby on Rails - rails
http://speeple.com/unicode-maps.txt
handy list of unicode code points, grouped by latin character and other ranges
How do I replace accented Latin characters in Ruby? - Stack Overflow
W3C I18N FAQ: Multilingual Forms
Second, it may be a good idea for the script that receives the form data to check that the data returned indeed uses UTF-8 (in case something went wrong, e.g. the user changed the encoding).
GEntrans: Home page
a gstreamer based transcoder. written in python, has some helpful documentation on building transcoding pipelines
MySQL :: Unicode and Other Funny Characters
How to port your gem to Ruby 1.9 - 世界線航跡蔵
MySQL :: Re: utf8_unicode_ci vs utf8_general_ci
So when you need better sorting order - use utf8_unicode_ci, and when you utterly interested in performance - use utf8_general_ci.
Ruby 1.9 character encoding field notes · Fingertips
The biggest problem with receiving data from external sources is trust. Sources can lie about their encoding or provide broken data. Sometimes it gets mangled accidentally and sometimes someone is attacking your application with a carefully crafted payload.
Soapbox
interesting rant about ruby 1.9's encoding support. Most it it resonates strongly with my opinions.