500k reads per second on 1 datanode

just did some benchmarking on multi-threaded ndbd (binary called ndbmtd)
that is in the coming 6.4 release.

quite happy with results

--- results

[jonas@n1 run]$ flexAsynch -ndbrecord -temp -t 8 -p 512 -r 5 -a 2
insert average: 374200/s min: 374200/s max: 374200/s stddev: 0%
update average: 370947/s min: 370947/s max: 370947/s stddev: 0%
delete average: 395061/s min: 395061/s max: 395061/s stddev: 0%
read average: 537178/s min: 531948/s max: 543092/s stddev: 0%

---

this flexAsynch command will run with
- 8 threads
- 512 parallel transactions per thread
- 8 byte records.

note: during the reads, the datanode was *not* maxed out.

---

this was run on two identical computers,
2-socket, 4 cores per socket Intel(R) Xeon(R) CPU X5355 @ 2.66GHz

api-program was running on computer 1 (n1)
datanode was running on computer 2 (n2)

--- configuration

[cluster_config]
DataMemory=2000M
IndexMemory=150M

SendBufferMemory=8M
ReceiveBufferMemory=8M
LongMessageBuffer=64M

NoOfReplicas=1
ndb_mgmd=n1
ndbd=n2
mysqld=n1,n1,n1,n1
Diskless=1
MaxNoOfExecutionThreads=6
MaxNoOfConcurrentTransactions=16384

forks, add-on patch-sets and features

so far little is happening in this area with MySQL Cluster.
would be interesting to get patches to cluster from a(ny) (huge-web) company...
wonder if that will ever happen...
maybe we don't use enough buzz-words

---

it could also be that we add features in a high enough pace ourselves,
preliminary benchmarks of our multi-threaded ndbmtd(4 threads)
shows up to 3.7 times better throughput than singled threaded ndbd.

create(drop) node(group) pushed

this friday.

no (currently :) known bugs

end of think-period

today, I think I finally cracked how to create(drop) a nodegroup.
basic concept is to
- temporary block gcp
- create(drop) the node group
- unblock gcp

(the same concept is btw used for adding a starting node to gcp)
the block should last for micro seconds

now it's only implementing it...

---

very happy that I now know how to proceed,
I've spent quite a lot of time trying to figure out a 100% safe
way of doing it...(wo/ blocking gcp)
but this solution will be efficient and fairly easy to implement.
(if any protocol dealing with (multi)node-failures can be considered easy)

status of create/drop node(group)

status:
create/drop nodegroup now works with one noticeable exception
replication cant be connected while the nodegroup is added.
i'll try to find time to fix this next week.

howto:
- start a 2 node cluster
- create table T1
- stop ndb_mgmd, add 2 nodes, start ndb_mgmd
- either stop the 2 running nodes and restart all 4
or rolling restart the 2 running nodes, and then start the 2 new nodes
- ndb_mgm> create nodegroup n1,n2
- alter table T1 add partitions partitions 2

Tata! fully online scaling of the cluster

howto backwards:
- drop table T1
- ndb_mgm> drop nodegroup X
- ndb_mgm> n1,n2 stop -a
- stop ndb_mgmd, remove the 2 nodes, start ndb_mgmd
- either stop the 2 running nodes and restart them
or rolling restart them

(A nodegroup is allowed to be dropped if it does not contain any data)

side effect:
- I added the possibility to specify nodegroups per node in the config-file
(this I intend to use for testing, but maybe someone might find it interesting)

future:
- magnus is working on "online configuration change" in the ndb_mgmd
once this is complete/functional, we can add the "add node"-command
so that the entire procedure can be done wo/ node restarts.

---

In pain

I miss bk

summer months

june: customer issues and bugs
july: vacation
august: must complete add node (I haven't started, but work has been done by stewart)

----

No need to comment this...
if you want to make me happy, choose one of the earlier posts