Behdad Esfahbod's daily notes on GNOME, Pango, Fedora, Persian Computing, Bob Dylan, and Dan Bern!

My Photo
Location: Toronto, Ontario, Canada

Ask Google.

Contact info
Hacker Emblem Become a Friend of GNOME I Power Blogger
follow me on Twitter
July 2003
August 2003
October 2003
November 2003
December 2003
March 2004
April 2004
May 2004
July 2004
August 2004
September 2004
November 2004
March 2005
April 2005
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
July 2008
August 2008
October 2008
November 2008
December 2008
January 2009
March 2009
April 2009
May 2009
June 2009
July 2009
August 2009
November 2009
December 2009
March 2010
April 2010
May 2010
June 2010
July 2010
October 2010
November 2010
April 2011
May 2011
August 2011
September 2011
October 2011
November 2011
November 2012
June 2013
January 2014
May 2015
Current Posts
McEs, A Hacker Life
Wednesday, April 23, 2008
 UTF-8 Bit Manipulation


My reasoning was that the current code is not worth changing without strong profiling data showing measurable gain in real-world use cases. That's all.

Now to your solution and questions. You approach has two and a half major issues that make it unusable in real code:So while it theoretically works, using nine integer operations, in practice it's unusable. Oh, your function produces the exact same values as in the glib table BTW. That's good.

Here is my solution that can be written as valid C code using 13 simple 32-bit operations:
def behdad_utf8_skipper(c):
v = c ^ 0xff
r = (v > 0xF) << 2
v >>= r
s = (v > 0x3) << 1
v >>= s
r |= s
r |= (v >> 1)
return (0x11234561 >> (r << 2)) & 7
It's basically a negation followed by a log2 straight from bithacks, followed by a table lookup. I particularly like the beautiful final constant.

I leave it to others to measure if this is faster than the lookup-table in glib. Enjoyed working this out though. Everyone, go crazy, shove a few ops off!

Labels: , ,

Your second objection is valid (the undefinedness of too many shifts). Your first objection (to the 64-bit math) isn't as strong as you might think, since the 64-bit shift can often be done with only two 32-bit shift instructions (using the carry bit).

Your third objection, that x*3 should be written (x<<1)+x, is bogus. On almost all modern processors, the former is faster; on the few where it is not (and I can't think of any), gcc will automatically replace the multiply by shifts and adds.
Thanks Joe.

I'm not sure I understand your point about 64-bit shift. Isn't that only valid for shifting one bit? How do you do "i64>>j" using two 32-bit shift instructions?

Regarding "x*3", I wasn't suggesting that it should be written as "(x<<1)+x". All I was saying was that there are architectures on which gcc does this. So, when counting instructions, multiplications are not necessarily the same as addition, bitwise, or shift operations. That's all. Now when you say all modern processors, do you mean ARM too?

Post a Comment

<< Archive
<< Home