After lunch, we start off with Hidden Scalability Gotchas in Memcached and Friends by Neil Gunther (Performance Dynamics Company), Shanti Subramanyam (Oracle Corporation), and Stefan Parvu (Oracle Finland).
Scaling up versus scaling out. Bigger or more. There is no “best approach” – you need to be quantitative, with controlled measurements and numbers to see the cost-benefit.
Data isn’t information, you need to transform it. Capacity planning has “planning” in it. Like with finance, you need a model. Metrics + models = information.
Controlled Measurements
You want to take measurements in a known environment with a specific workload. Using production time series data is like predicting the stock market using Google finance graphs.
You need throughput measured in steady state. No load vs vusers with varying throughput…
So they did some controlled tests.
Memcached scaling is thread limited. Past about 4-6 threads, throughput levels off.
By using a SPARC multicore friendly hash map patch, it did scale up to maybe 30 threads.
Quantifying Scalability
1. Equal bang for the buck. Ideal parallelism is a linear load vs capacity graph, but really it plateaus and degenerates at some point. But there’s an early art of the graph that looks like this.
2. Cost of sharing resources – when the curve falls away from linear.
3. Resource limitation – where the curve tops out (Amdahl’s law)
4. Degradation/negative return – more capacity makes things worse after a point.
Formula: c(N)=N/(1+a(N-1)+bN(N-1))
N is the number of threads. 1=concurrency, a=contention , b= coherency
Run it through excel USL analysis and calculate a and b.
As memcached versions came out, the concurrency was improved, but N didn’t budge. People can say they make improvements, but if it doesn’t affect the data, then bah.
Anyway, the model is semi predictive but not perfect. If you know whether your problem is a contention (like queuing) or coherence (like point to point transfers) issue you know what to look for in your code.
Memcached Gotchas
Throw more hardware at it! Well, current strategies are around old cheap hardware with single CPUs. As multicore arrives, if you can’t use all the cores, you won’t utilize your hardware fully.
As memcached is thread limited it’ll be a problem on multicore.
Take controlled measurements with steady state throughput to analyze data.
Quantify scalability using a model. Reduce contention and coherency.
Follow them at:
There’s a lot of discussion about the model predictability because they had a case where the model predicted one thing until there were higher order data points and then it changed. The more data, the more the model works – but he stresses you need to trust the model with what data you have. You’re not predicting, you’re explaining with the model. It’s not going to tell you exactly what is wrong… Lots of questions, people are mildly confused.
the agile admin 