网络科技

    今日:952| 主题:246304
收藏本版
互联网、科技极客的综合动态。

[其他] Make MyRocks 2X less slow

[复制链接]
不变的信仰 发表于 2016-10-20 00:13:41
121 2

立即注册CoLaBug.com会员,免费获得投稿人的专业资料,享用更多功能,玩转个人品牌!

您需要 登录 才可以下载或查看,没有帐号?立即注册

x
Fixing mutex contention has been good for my career. I had the workload, an RDBMS that needed a few improvements and support from a great team. Usually someone else found the bugs and I got to fix many of them. Sometimes I got too much credit because a good bug report is as valuable as the bug fix. These days I don't see many mutex contention bugs but I have begun to see more bugs from memory contention. My perf debugging skills need refreshing. They are far from modern. Thankfully we have    Brendan Gregg.  
  For someone who debugs performance, shared_ptr is a gift. Mistakenly passing shared_ptr by value means the reference count will be changed too much and that is not good on a concurrent workload. I have encountered that at least twice in RocksDB and MyRocks. I even encountered it in MongoDB with    SERVER-13382.  
  I have twice made MyRocks 2X less slow. First with    issue 231peak compaction throughput was doubled and now with    issue 343we almost double range-scan throughput (for long range scans with many concurrent queries). Someone important recently reported a disappointing performance result when comparing MyRocks with InnoDB. After a few days with sysbench I was able to reproduce it. This should be easy to fix.  
      Not mutex contention  
  In this bug, with sysbench read-only and read-write the peak QPS for MyRocks saturated long before InnoDB. While MyRocks and InnoDB had similar QPS at low concurrency, the QPS at high concurrency was almost 2X better for InnoDB. This was only an issue for longer range scans (try --oltp-range-size=10000) and the default was a shorter range scan (--oltp-range-size=100). My first guess was mutex contention. There was an odd pattern in vmstat where the    context switch ratealternated every second for MyRocks but was steady for InnoDB. Spikes in context switch rate sometimes mean mutex contention but I did not see that with PMP. What next?  
  The next guess is memory system contention but my debugging skills for that problem are weak. I have told myself many times this year that I need to refresh my skills. So I started with this    blog postfrom Brendan Gregg and tried    perf statand found that InnoDB completed almost twice the number of instructions compared to MyRocks in the same time period. Why is    IPCalmost 2X better for InnoDB? Results from perf    are here.  
  I then tried a different perf stat command to get other hardware perf counters and    results are here. This also shows that InnoDB completed twice the number of instructions while both have a similar value for bus-cycles, so MyRocks uses 2X the number of bus-cycles per instruction. That can explain why it is slower. What are bus-cycles? Most of the documentation only explained this is as    [Hardware event]and without more details I can't look that up in Intel manuals. I asked internally and learned the magic code, 0x013c, that    leads to more information. Start with    this article(and consider subscribing to LWN, I do).  
  The next step was to get call graphs when bus-cycles was incremented. I used the command below to find the offending code. Disabling that code fixes the problem, but work remains to make that code performant. InnoDB and MyRocks have similar code to count rows read and InnoDB doesn't fall over because of it. I want to make MyRocks not fall over because of it.
  perf record -ag -p $( pidof mysqld ) -e bus-cycles -- sleep 10
      Useful commands  
  I used all of these commands today:
    perf stat -a sleep 10
    perf stat -e cycles,instructions,cache-references,cache-misses,bus-cycles -a sleep 10
    perf stat -e 'syscalls:sys_enter_*' -a sleep 10
    perf stat -e L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores -a sleep 10
    perf stat -e dTLB-loads,dTLB-load-misses,dTLB-prefetch-misses -a sleep 10
    perf stat -e LLC-loads,LLC-load-misses,LLC-stores,LLC-prefetches -a sleep 10
    perf top -e raw_syscalls:sys_enter -ns comm
    perf list --help
    perf record -ag -p $( pidof mysqld ) -e bus-cycles -- sleep 10
友荐云推荐




上一篇:Google app beta tests ‘Upcoming’ tab for timely information
下一篇:Building Your Startup: Refining Email Templates
酷辣虫提示酷辣虫禁止发表任何与中华人民共和国法律有抵触的内容!所有内容由用户发布,并不代表酷辣虫的观点,酷辣虫无法对用户发布内容真实性提供任何的保证,请自行验证并承担风险与后果。如您有版权、违规等问题,请通过"联系我们"或"违规举报"告知我们处理。

AllenJX 发表于 2016-10-20 19:46:41
这次必须是沙发!
回复 支持 反对

使用道具 举报

兰山 发表于 2016-10-21 00:09:36
怎么我回帖都没人理我呢?
回复 支持 反对

使用道具 举报

*滑动验证:
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

我要投稿

推荐阅读

扫码访问 @iTTTTT瑞翔 的微博
回页顶回复上一篇下一篇回列表手机版
手机版/CoLaBug.com ( 粤ICP备05003221号 | 文网文[2010]257号 )|网站地图 酷辣虫

© 2001-2016 Comsenz Inc. Design: Dean. DiscuzFans.

返回顶部 返回列表