I am keeping an eye on the pubsubhubbub technology since its first day, it is a very simple, yet interesting technology to make RSS/Atom feeds really fast without polling. I wrote my own component called 'SubHub', it is running for a couple of weeks now, it is receiving tons of Atom and RSS messages from popular news-sources like nasa, cnn, bbc and the few thousand other, through pubsubhubbub. Thanks to this hard-to-spell technology, I do not poll at all :-)
The developers envisioned a distributed network of hubs and streams and sure it is distributed by design. What I wanted to know is how distributed is it, since my first guess was that Google will have a little dominance in it. For example blogger.com rss feeds (including Dummy Warhead's) are pointing to google's pubsubhubbub server. The WordPress guys were also very active and they implemented their WordPress plugin, which is turned on in all the wordpress.com blogs, but they only use it for their own content while Google's service is used by anyone... and almost everyone.
Enough talking. Let's see, who is doing most of the work in pubsubhubbub:
The graph was generated from the web server log using the usual linux commands.
So the two upcoming players are SuperFeedr and Pheedo, looks like they are special services for special customers. There might be some minor players in the technology, but I haven't found them yet.
Wednesday, 24 October 2012
Friday, 9 December 2011
PostgreSQL - enums vs varchar
Work-related topic today. OVirt stores java enum values as integer in database. Java 1.5 enums are kind of great tools in the language, but if you accidentaly re-order them, the ordinal changes, which is tragic for the records persisted in the database.
One solution is that you add a final integer field to your enum, set it from a private constructor, add a getter, and use it in your database persistence code. Ah yes, this lots of code to be written and maintained... and basically we are just lazy, just like anyone else. [Anyway, this solution is actually used by Ovirt today.]
Another possible solution that for example JPA supports is that you persist your enum as varchar, which is great, but varchars are bigger than integers. In this case it is kind of difficult to rename anything in the enum, names are more human-friendly than numbers, so you may not want to do this very often.
So lots of guys came up with the question: why not database enums. Enums are a very unique solution in PostgreSQL. [Relational databases are full of unique solutions, but fortunately not as much as NoSQL-databases, at least they somewhat agree in SQL syntax :-) ] PostgreSQL has enum. Enum is cool, it takes only 4 bytes, and can be read and written as character data, it can also save your application from invalid data, e.g. a 'rde' inserted accidentally as color;
When extending an enum, you can add or remove new elements with 'ALTER TYPE', however, this command is a new feature in the latest and greatest [therefore not that much widespread] version 9.1, it is not available in PostgreSQL 9.0. [This could be a problem for OVirt, since it targets PostgreSQL 8.4]
Ok, enough talking, let's see the data.
Insert performance
As you can see, with enums you can win a nice 10-15% on performance, but this is an edge-case. In this benchmark, I inserted 100.000 records in a single transaction. This is a very rare in applications.
When using multiple transactions, the difference dissappears, since the transaction overhead is significant.
Disk use
Sure, this is why we have come to enum. There is no significant difference in the index size since varchars are not stored in the index. However, the small storage requirement is still great, the database server can cache more useful information in the RAM, which results in faster responses if the database size is big. I mean, bigger than your RAM. [Not sure if this is a significant advantage, oVirt does not need a huge database]
Update performance
Again, enum beats varchar with roughly 10 percent. [This is a relatively small win for OVirt, but e.g. updating statuses on Virtual Machines is a very frequent operation.]
Select
Select is another very frequent operation. As mentioned, enums takes less space, more data can fit in the memory.
Let's see some query plans, the first one is for the varchar column that has an index on it.
temp=# EXPLAIN ANALYZE select * from vi where birdtype = 'cuckoo';
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on vi (cost=7.44..22.59 rows=412 width=10) (actual time=0.032..0.032 rows=0 loops=1)
Recheck Cond: ((birdtype)::text = 'cuckoo'::text)
-> Bitmap Index Scan on vi_birdtype_idx (cost=0.00..7.34 rows=412 width=0) (actual time=0.029..0.029 rows=0 loops=1)
Index Cond: ((birdtype)::text = 'cuckoo'::text)
Total runtime: 1.790 ms
(5 rows)
And then the same with the enum column.
temp=# EXPLAIN ANALYZE select * from vi where birdtype = 'eagle';
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on vi (cost=707.02..1998.61 rows=19967 width=10) (actual time=2.465..7.674 rows=20000 loops=1)
Recheck Cond: ((birdtype)::text = 'eagle'::text)
-> Bitmap Index Scan on vi_birdtype_idx (cost=0.00..702.03 rows=19967 width=0) (actual time=2.378..2.378 rows=20000 loops=1)
Index Cond: ((birdtype)::text = 'eagle'::text)
Total runtime: 8.738 ms
(5 rows)
One solution is that you add a final integer field to your enum, set it from a private constructor, add a getter, and use it in your database persistence code. Ah yes, this lots of code to be written and maintained... and basically we are just lazy, just like anyone else. [Anyway, this solution is actually used by Ovirt today.]
Another possible solution that for example JPA supports is that you persist your enum as varchar, which is great, but varchars are bigger than integers. In this case it is kind of difficult to rename anything in the enum, names are more human-friendly than numbers, so you may not want to do this very often.
So lots of guys came up with the question: why not database enums. Enums are a very unique solution in PostgreSQL. [Relational databases are full of unique solutions, but fortunately not as much as NoSQL-databases, at least they somewhat agree in SQL syntax :-) ] PostgreSQL has enum. Enum is cool, it takes only 4 bytes, and can be read and written as character data, it can also save your application from invalid data, e.g. a 'rde' inserted accidentally as color;
When extending an enum, you can add or remove new elements with 'ALTER TYPE', however, this command is a new feature in the latest and greatest [therefore not that much widespread] version 9.1, it is not available in PostgreSQL 9.0. [This could be a problem for OVirt, since it targets PostgreSQL 8.4]
Ok, enough talking, let's see the data.
Insert performance
As you can see, with enums you can win a nice 10-15% on performance, but this is an edge-case. In this benchmark, I inserted 100.000 records in a single transaction. This is a very rare in applications.
When using multiple transactions, the difference dissappears, since the transaction overhead is significant.
Disk use
Sure, this is why we have come to enum. There is no significant difference in the index size since varchars are not stored in the index. However, the small storage requirement is still great, the database server can cache more useful information in the RAM, which results in faster responses if the database size is big. I mean, bigger than your RAM. [Not sure if this is a significant advantage, oVirt does not need a huge database]
Update performance
Again, enum beats varchar with roughly 10 percent. [This is a relatively small win for OVirt, but e.g. updating statuses on Virtual Machines is a very frequent operation.]
Select
Select is another very frequent operation. As mentioned, enums takes less space, more data can fit in the memory.
Let's see some query plans, the first one is for the varchar column that has an index on it.
temp=# EXPLAIN ANALYZE select * from vi where birdtype = 'cuckoo';
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on vi (cost=7.44..22.59 rows=412 width=10) (actual time=0.032..0.032 rows=0 loops=1)
Recheck Cond: ((birdtype)::text = 'cuckoo'::text)
-> Bitmap Index Scan on vi_birdtype_idx (cost=0.00..7.34 rows=412 width=0) (actual time=0.029..0.029 rows=0 loops=1)
Index Cond: ((birdtype)::text = 'cuckoo'::text)
Total runtime: 1.790 ms
(5 rows)
temp=# EXPLAIN ANALYZE select * from vi where birdtype = 'eagle';
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on vi (cost=707.02..1998.61 rows=19967 width=10) (actual time=2.465..7.674 rows=20000 loops=1)
Recheck Cond: ((birdtype)::text = 'eagle'::text)
-> Bitmap Index Scan on vi_birdtype_idx (cost=0.00..702.03 rows=19967 width=0) (actual time=2.378..2.378 rows=20000 loops=1)
Index Cond: ((birdtype)::text = 'eagle'::text)
Total runtime: 8.738 ms
(5 rows)
No significant difference, because of the same index size / efficency.
Saturday, 3 December 2011
Cassandra on a pile of wreck
NoSQL databases traded ACID compliance for speed and scalablity and I was very interested in how good deal is that for mere mortals like me.
I am experimenting with Cassandra for a while. Version 1.0 was released recently (probably a month ago) and bugfix releases followed quickly, so now the current release is 1.0.5.
A special challange is the server farm I have: no uniform servers. Actually, you could call them a pile of wreck, they would not fit even desktop-class, but that does not matter. There is a trick to ballance the servers right.
I am experimenting with Cassandra for a while. Version 1.0 was released recently (probably a month ago) and bugfix releases followed quickly, so now the current release is 1.0.5.
A special challange is the server farm I have: no uniform servers. Actually, you could call them a pile of wreck, they would not fit even desktop-class, but that does not matter. There is a trick to ballance the servers right.
- dummywarhead - this is first cassandra node, the seed of the cluster running Fedora 15, it's hardware is a 512 GB sata hard disk, 2 GB DDR3 RAM and a dual core AMD CPU.
- maroon - the only branded server in my the farm, an IBM XSeries 226 with hyper-threaded 3Ghz Xeon processor 1 GB DDR2 RAM and an old 250 GB sata hard drive. Runs ubuntu. (Yeah, I was lazy and I have just left it running ubuntu, it does not really matter from performance point of view)
I really like the IBM casis, but the server generates terrible noise when under load.
As it turned out, I can expect 50% of dummywarhead's CPU performance from this server, and exactly half as much memory. - Switch: TP-LINK 5-port gigabit switch - this should not be a bottleneck in the system. Cat-6 cables, gigabit network interfaces installed in all of the nodes.
- Load generator: Lenovo T520 Tankpad: 4 GB ram, 4 cores. (Red Hat work laptop)
- Since I do not do updates, the repair on read chance is set to zero.
- I abandoned cassandra's thrift client after the first few tests and moved to hector. Cassandra's client seemed to be unable to deal with dead connections, database topology discovery, etc. Hector does not have any problem with all these.
Cassandra's client also seem to have_a_problem with javaNamingConventions, it was a bit confusing. - It seems that Cassandra (up to 1.0.5) assumes that hosts are unique, so when a new node joins the network, it ballances the database for equal load automatically. In order to get a good result out of the test, I had to update the tokens before starting the loadtest. The logic is quite simple, you just have to distribute the keys between 0 and 2^127, this will determine the load on the servers. Since I assumed that dummywarhead can do twice as much load as maroon, I generated token 56713727820156410577229101238628035242 for dumywarhead and 113427455640312821154458202477256070484 for maroon. This will put 66% of the load on the first, and 33% of the load on the second node.
On this chart you see a single node test. The first thing you notice is that the read times (red) get a little slower until the half of the test, it goes unstable and at the end of the test it is 10 times slower than at the half of the test. This is because of the amount of memory I have in the test server. When it ran out of the memory, in each round it had to find something on disk more and more often.
The second interesting thing here is that the writes are so nice and stable.
On this second test I have let cassandra ballance the database the way she wanted. I started up both dummywarhead and maroon, and started the test after maroon successfuly joined the ring.
As you can see, in this case the collapse of read operations happened much later and did not reach the 200 second limit that the previous single node test did. However, this it is far from being half as much...
While watching the load on the servers, maroon started to do heavy read operations soon since it ran out of RAM. The other server, dummywarhead did not start read operations at all until the end of the test.
I made the first two charts so that it is easy for you to compare the two. The third chart is still belong to the second experiment. I could not resist to do some modifications and I used the nodetool move command to re-allign the load according to the real capabilities of the servers. So this is the same test, continued.
What you see here is that the response time got crazy long for a while when I started to re-allign the cluster, it is a huge peak. And then, when the operations finished, the response time dropped again. The sharp-eyed may notice that the write operations also got 20% faster after re-allign.
So far so good... but.
Joining nodes to the cluster under load takes a lot of time. Really a lot, even with small databases. I am sure one more server to a 100-node cluster is no problem, but adding a second node to a single node really kicks. And then it reballances the cluster to the (bad) defaults, which drops performance. So at the moment runing cassandra on non-uniform nodes seems to be difficult. Plus, scaling out did not immediately made things better, it dropped performance for a while.
The Cassandra cookbook says that the move operation actually removes the node and re-adds it to the cluster with the new token. This may have changed since Cassandra 0.5, it is not that heavy operation, as far as I can tell, it did not remove the node, it only moved it to 'Moving' state for a while.
One can play a lot with this software, it has some interesting concepts.
The second interesting thing here is that the writes are so nice and stable.
On this second test I have let cassandra ballance the database the way she wanted. I started up both dummywarhead and maroon, and started the test after maroon successfuly joined the ring.
As you can see, in this case the collapse of read operations happened much later and did not reach the 200 second limit that the previous single node test did. However, this it is far from being half as much...
While watching the load on the servers, maroon started to do heavy read operations soon since it ran out of RAM. The other server, dummywarhead did not start read operations at all until the end of the test.
I made the first two charts so that it is easy for you to compare the two. The third chart is still belong to the second experiment. I could not resist to do some modifications and I used the nodetool move command to re-allign the load according to the real capabilities of the servers. So this is the same test, continued.
What you see here is that the response time got crazy long for a while when I started to re-allign the cluster, it is a huge peak. And then, when the operations finished, the response time dropped again. The sharp-eyed may notice that the write operations also got 20% faster after re-allign.
So far so good... but.
Joining nodes to the cluster under load takes a lot of time. Really a lot, even with small databases. I am sure one more server to a 100-node cluster is no problem, but adding a second node to a single node really kicks. And then it reballances the cluster to the (bad) defaults, which drops performance. So at the moment runing cassandra on non-uniform nodes seems to be difficult. Plus, scaling out did not immediately made things better, it dropped performance for a while.
The Cassandra cookbook says that the move operation actually removes the node and re-adds it to the cluster with the new token. This may have changed since Cassandra 0.5, it is not that heavy operation, as far as I can tell, it did not remove the node, it only moved it to 'Moving' state for a while.
One can play a lot with this software, it has some interesting concepts.
Sunday, 6 November 2011
JSON vs XML
A few days ago I asked a few guys at work that if they were given a choice, would they choose JSON or XML as a format for a REST API. Everyone responded in favor of JSON. What is surprising in this is that they were all java developers, not a single javascript guy. Maybe it is too early to say this, but it seems that after the initial success of XML in the previous decade and stagnation in the last few years, the XML as format is slowly getting to decline, while more compact and simple, schemaless formats seem to rise: JSON and YAML.
JSON and YAML implementations are just as simple as their formats, you can choose from very many of them.
Question: How do JSON parsers compare to XML parsers from performance perspective?
I wrote sample inputs in XML and JSON. The same data, basically. For XML, I have made two input files. The first is the 'traditional' XML, no attributes at all. The second uses some attributes. The third is the JSON version. I used very few XML apis, only the ones packaged into JDK, this is a bit unfair because the alternative XML apis may be a little faster than the default. I have made a test a few years age and that showed them not so much different, so I was more interested in JSON. With JSON format, I used 5 different parsers: json.org, jackson, jsonlib, jsonj and gson.
Even the slowest JSON parser is faster than the XML parsers of JDK, also the format is more compact, looks like it is win. Jackson is much faster than the rest. It's website is right, it is very fast.
Test source code: here.
JSON and YAML implementations are just as simple as their formats, you can choose from very many of them.
Question: How do JSON parsers compare to XML parsers from performance perspective?
I wrote sample inputs in XML and JSON. The same data, basically. For XML, I have made two input files. The first is the 'traditional' XML, no attributes at all. The second uses some attributes. The third is the JSON version. I used very few XML apis, only the ones packaged into JDK, this is a bit unfair because the alternative XML apis may be a little faster than the default. I have made a test a few years age and that showed them not so much different, so I was more interested in JSON. With JSON format, I used 5 different parsers: json.org, jackson, jsonlib, jsonj and gson.
Even the slowest JSON parser is faster than the XML parsers of JDK, also the format is more compact, looks like it is win. Jackson is much faster than the rest. It's website is right, it is very fast.
Test source code: here.
Saturday, 29 October 2011
turning simple into difficult: date formats
SimpleDateFormat is a nice little class that helps programmers turn a date to string and vice versa. It is very simple to use and it is packaged with standard java since 1.2 probably... that was very long ago. And of course we use it a lot.
At the other hand, SimpleDateFormat is kind of pain in the back because it is not thread-safe and never going to be. I was always wondering why, it seems to be a terribly wrong design decision. It makes things a bit difficult with SimpleDateFormat. Most of the time you just want to create a final static SimpleDateFormat field with your preferred format. If you do this, your code is probably already broken, under high load it will produce stupid results and you will not know why. (Unless you use PMD with the proper rules) If your class needs to be threadsafe -most of them do- the best thing to do is to create a new instance of SimpleDateFormat when you need it, as a local varible in a method.
What is painful with this? It's performance sucks.
This is what everyone knows. Let's see how to deal with the problem:
- Workaround :-)
You can have a final static SimpleDateFormat object, and anytime you want to use it, you clone it and use the clone to do the job. Funny right?
This is a theoretical solution. Do not use this! The java culture chose a completely different approach and year by year thousands of java developers get heart-attack after finding a final static SimpleDateFormat field in the code. - Commons Lang is around for a good while as well. Life would be hell without the apache commons projects. Commons-lang provides some support (FastDateFormat) to deal with date formating. Unfortunately it does not support parse. Only format :-( but even that is more than nothing.
Now of course the choice depends on your taste, but I thought I will help you with a short performance benchmark on the topic:
new SimpleDateFormat 100000 times : 170 ms
SimpleDateFormat clone 100000 times : 68 ms
DateFormatUtils 100000 times : 62 ms
FastDateFormat 100000 times : 55 ms
Update: the two lines updated from Zsombor's ThreadLocal idea. Wow, this must be some mistake!
SimpleDateFormat form pre-created ThreadLocal 100000 times : 29 ms
SimpleDateFormat in ThreadLocal check each time 100000 times : 28 ms
Update: the two lines updated from Zsombor's ThreadLocal idea. Wow, this must be some mistake!
SimpleDateFormat form pre-created ThreadLocal 100000 times : 29 ms
SimpleDateFormat in ThreadLocal check each time 100000 times : 28 ms
For me it is surprising again and again, how bad the SimpleDateFormat performs when you create it again and again. More than half of the processing is just creating the SimpleDateFormat object, an enormous waste of resources. And then once it is created, it is very quick.
Another funny link for jarvana to find out how many packages has a FastDateFormat. I can't decide if it is tragic or funny...
Test code: here.
java version: sun jdk 1.6.22
OS: linux (fedora 15)
Hardware: Lenovo T520 with Intel core i7 vpro (thank you Red Hat :) )
Another funny link for jarvana to find out how many packages has a FastDateFormat. I can't decide if it is tragic or funny...
Test code: here.
java version: sun jdk 1.6.22
OS: linux (fedora 15)
Hardware: Lenovo T520 with Intel core i7 vpro (thank you Red Hat :) )
Friday, 12 August 2011
...and...
...and I am back.
I moved to Brno, Czech Republic and now I am working for Red Hat.
Since I am not good in czech and most people in Brno are not good in english, it took a few weeks, but now I have internet connection at home and I will continue my performance check soon. Recently I have been playing with Cassandra and Infinispan. Cool stuff :-)
Wednesday, 15 June 2011
Serialization speed
Serialization is probably the most simple way to send data to an output stream. The three most typical use cases:
So... what this tells me is that when you have problems with session replication, your problem is not the serialization, it is more likely the IO speed. While serialization produced 600 MB in just 4 seconds, a gigabit network with tcp/ip connection can transport roughly 60-70 MB/sec, so it will need at least 10 secs to transport your data.
If a component is slow, use it sparingly :-)
Java version: 64 bit Sun/Oracle JDK 1.6
OS: Linux (fedora)
- Session replication in servlet containers
- A simple way to save information to disk
- RMI uses it heavily
Of course session replication is kind of important and it would be nice if it was quick. Therefore, I was very interested in the cost of serialization in general, with small data structures and bigger ones alike.
Serialization is smart, it does not serialize the same object twice, it just sends a reference to an already serialized object, this is the mechanism that allows deserialization reconstruct the same object-graph on the other side of the stream. The small drawback of this smart behavior is that if you send through lots of data on the ObjectOutputStream, you should call reset() time after time again, otherwise the ObjectOutputStream will hold a reference to all objects you have serialized and sooner or later you get an OOM. The reset method clears the ObjectOutputGraph's object/ID tracking and subsequent calls to writeObject will actually serialize the data again. So in my tests this method is called after each serialization. Well, there are a couple of ways you can shoot yourself on the leg with serialization :-)
And for IO stream, I implemented something similar to commons-io NullOutputStream, since it is the serialization speed I want to know, not the IO speed. Here comes the google eye-candy:
So... what this tells me is that when you have problems with session replication, your problem is not the serialization, it is more likely the IO speed. While serialization produced 600 MB in just 4 seconds, a gigabit network with tcp/ip connection can transport roughly 60-70 MB/sec, so it will need at least 10 secs to transport your data.
If a component is slow, use it sparingly :-)
Java version: 64 bit Sun/Oracle JDK 1.6
OS: Linux (fedora)
Subscribe to:
Posts (Atom)