Oracle is pleased to announce the general availability of the Oracle RDBMS Server 11gR2
Pre-Install RPM for Oracle Linux 6 x86_64 (64 Bit) architecture.

This package replaces the previous Oracle Validated RPM.

You can complete most preinstallation configuration tasks by using the Oracle RDBMS Server 11gR2
Pre-install RPM, available from the Unbreakable Linux Network, or via the Oracle Public Yum
repository.

When it is installed, the Oracle RDBMS Pre-install RPM does the following:

* Automatically installs any additional packages needed for installing Oracle Grid
Infrastructure and Oracle Database 11gR2 (11.2.0.3).

* Creates an oracle user, and creates the oraInventory (oinstall) and OSDBA (dba) groups for
that user. For security purposes, this user has no password by default and cannot login
remotely. To enable remote login, please set a password using the “passwd” tool.

* Sets and verifies sysctl.conf settings, system startup parameters, user limits, and driver
parameters to the minimum acceptable values based on recommendations from the Oracle Database
Installation Guide and the Oracle Validated Configurations program.

Important Note

Removal of the Oracle RDBMS Server 11gR2 Pre-install RPM will restore the following files to
the version prior to inital installation:

* /etc/security/limits.conf
* /boot/grub/grub.conf
* /etc/sysctl.conf

If you make further changes to these files after installing the Oracle RDBMS Server 11gR2
Pre-install RPM, please make a backup of your changed files and manually restore them after the
package removal is complete.

x86_64:
oracle-rdbms-server-11gR2-preinstall-1.0-3.el6.x86_64.rpm

SRPMS:

http://oss.oracle.com/ol6/SRPMS-updates/oracle-rdbms-server-11gR2-preinstall-1.0-3.el6.src.rpm

Description of changes:

[1.0-3.el6]
- Added smartmontools as dependency, bz13653

[1.0-2.el6]
- kernel.shmall=1073741824 as per bz7256
- kernel.shmmax=4398046511104 as per bz7256
- stack hard = 32768 as per doc max limit

[1.0-1.el6]
- Renamed rpm to oracle-rdbms-server-11gR2-preinstall
- Included xorg-x11-utils xorg-x11-xauth as dependency bz13653
- Included kernel-uek as dependency.
- fs.aio-max-nr=1048576 to match document,bz13653
- kernel.shmall=2097152 to match document,bz13653
- kernel.shmmax=536870912 to match document,bz13653
- nofile soft = 1024 to match document,bz13653
- nofile hard = 65536 to match document,bz13653
- nproc soft = 2047 to match document,bz13653
- nproc hard = 16384 to match document,bz13653
- stack soft = 10240 to match document,bz13653
- stack hard = 10240 to match document,bz13653

[1.0.0-3.el6]
- removed util-linux and added util-linux-ng (fork of util-linux)
- removed openssh and added openssh-clients bz13173
- removed 32 bit dependency for x86_64 as per st docs.
- removed kernel-uek-headers/kernel-headers
- disable login for oracle user for bug12623491
- Merge fix
- Removed msgmni, msgmnb, msgmax for bz11029
- Increase stack limit for oracle user bz11683
- bugfix for bug11656858
- added compat-libcap1 dependency bz12221
- move link creation to install part bz11030
- removed comment related to bugdb6820451
- removed flowcontrol settings bz11508
- Removed 10G related info from oracle-rdbms-server-11gR2-preinstall.param
- Changed kernel.semmni to 128 as per 11203crs-cvu_prereq.xml
- removed vm.min_free_kbytes
- removed readme

[1.0.0-1.el6]
- Changed requirement for x86_64 arch
/lib/libaio.so.1
libodbc.so.2()(64bit)
/usr/lib/libodbc.so.2
/usr/lib/gcc/x86_64-redhat-linux/4.4.4/libstdc++.a

Software Accessibility

All packages are available via ULN (http://linux.oracle.com) or the Oracle Public
Yum Repository (http://public-yum.oracle.com)

For more information, please visit http://linux.oracle.com

Related posts:

1. Install Oracle on Linux
2. Oracle De-install Utility
3. Oracle Database 11.2.0.2 Installation on Enterprise Linux x86 5.2 Part 1

Related posts:

1. Oracle 11.2.0.2新特性
2. Oracle Metalink reference – Database
3. Easy way to install Oracle Database on Oracle Linux

Related posts:

1. Oracle数据库产品发布计划
2. isqlplus desupport on 11g
3. Oracle Enterprise Manager 10g Grid Control Certification Checker

近期遇到的几个安装时的错误案例，都是跟下载的软件不完整有关的。因此建议在安装前务必先确认安装软件是正确、完整的。

可以在下载界面查看“view digist”，里面有每个文件的MD5，SHA-1校验值，可以用检查工具对下载的软件进行检查。

检查工具可以下载微软的FCIV，地址：Availability and description of the File Checksum Integrity Verifier utility

Happy Installation.

Related posts:

1. Oracle De-install Utility
2. Oracle Database补丁安装最佳方法

sqlplus / as sysdba

SQL*Plus: Release 10.2.0.1.0 – Production on M Mar 8 17:22:36 2010

Copyright (c) 1982, 2003, Oracle. All Rights Reserved.

ERROR:
ORA-1031: Insufficient privileges.

检查sqlnet.ora文件的配置：
SQLNET.AUTHENTICATION_SERVICES=(NTS)

重新修改配置如下：
SQLNET.AUTHENTICATION_SERVICES=(nts)

关于AUTHENTICATION_SERVICES参数有如下的常规设置：

none for no authentication methods, including Microsoft Windows native operating system authentication. When SQLNET.AUTHENTICATION_SERVICES is set to none, a valid user name and password can be used to access the database.

all for all authentication methods.

nts for Microsoft Windows native operating system authentication.

Related posts:

1. 11gR2 for Windows Release
2. Cisco VPN Client x64 for Windows 7
3. Config Oracle 10g VIP on Unix

New Relic’s multitenant, SaaS web application monitoring service collects and persists over 100,000 metrics every second on a sustained basis, while still delivering an average page load time of 1.5 seconds.  We believe that good architecture and good tools can help you handle an extremely large amount of data while still providing extremely fast service.  Here we’ll show you how we do it.

•  New Relic is Application Performance Management (APM) as a Service
•  In-app agent instrumentation (bytecode instrumentation, etc.)
•  Support for 5 programming languages (Ruby, Java, PHP, .NET, Python)
•  175,000+ app processes monitored globally
•  10,000+ customers

The Stats

•  20+ Billion application metrics collected every day
•  1.7+ Billion web page metrics collected every week
•  Each “timeslice” metric is about 250 bytes
•  100k timeslice records inserted every second
•  7 Billion new rows of data every day
•  Data collection handled by 9 sharded MySQL servers

Architecture Overview

• Language-specific agents (Ruby, Java, PHP. .NET, Python) send application metrics back to New Relic once every minute
• The “collector” service digests app metrics and persists them in the right MySQL shard
• Real User Monitoring javascript snippet sends front-end performance data to the “beacon” service for every single page view
• Customers log into http://rpm.newrelic.com/ to view their performance dashboard
• The amount of data we collect every day is staggering.  Initially all data is captured at full resolution for each metric.  Over time we reperiodize the data, going from minute-by-minute to hourly and then finally to daily averages.  For our professional accounts, we store daily metric data indefinitely, so customers can see how they’ve improved over the long haul.
• Our data strategy is optimized for reading, since our core application is constantly needing to access metric data by time series.  It’s easier to pay a penalty on write to keep the data optimized for faster reads, to ensure our customers can quickly access their performance data any time of the day. Sharding our database helps by distributing customers across multiple servers.  Within each server we have individual tables per customer to keep the customer data close together on disk and to keep the total number of rows per table down.
• New Relic manages several types of alerts for monitoring systems.  Customers can set thresholds on their APDEX score and error rate.  New Relic also has an availability monitoring feature, so customers can get alerted on downtime events as short as 30 seconds.  We send email alerts primarily, with several customers using our PagerDuty.com integration for more complex on-call rotations with SMS integration.
• Let’s take a single web transaction from a customer request all the way through the New Relic stack.
  1. An end user views a page on Example.com, who uses New Relic to monitor their app performanc
  2. The application running Example.com is running with New Relic agents installed (for Ruby, Java, PHP, .NET or Python
  3. Detailed performance metrics are captured for each transaction, including time spent in each component, database queries, external API calls, etc
  4. These back-end metrics are persisted in the customer’s New Relic agent for up to one minute, where they are then sent back to the New Relic data collection service
  5. Meanwhile, embedded in the web page is the New Relic Real-User Monitoring JavaScript code, which tracks the performance of this single customers experience
  6. When the page is fully rendered within the customer’s browser, the New Relic beacon gets a request providing performance metrics on the back-end, network, DOM processing and page rendering times.
  7. An engineer working on Example.com logs into New Relic and sees up-to-the-minute application performance metrics as well as the end-user experience for every single customer, including browser and geographic information.

Platform

Web UI

• Ruby on Rails
• nginx
• Linux
• 2 @ 12 core Intel Nehalem CPUs w/ 48Gb RAM

Data Collector And Web Beacon Services

Interesting MySQL Stats:

Challenges

Lessons Learned

Related posts:

1. Scaling Bumper Sticker: A 1 Billion Page Per Month Facebook RoR App
2. 37signals Architecture
3. Facebook: An Example Canonical Architecture For Scaling Billions Of Messages

详见官方网址：http://www.oracle.com/technetwork/indexes/documentation/index.html

Related posts:

1. Oracle Grid Control 11gR1下载
2. Install Oracle on Linux
3. Oracle blogs

当前的软件实现是指定一个Map（映射）函数，用来把一组键值对映射成一组新的键值对，指定并发的Reduce（化简）函数，用来保证所有映射的键值对中的每一个共享相同的键组。

映射和化简

简单说来，一个映射函数就是对一些独立元素组成的概念上的列表（例如，一个测试成绩的列表）的每一个元素进行指定的操作（比如，有人发现所有学生的成绩都被高估了一分，他可以定义一个“减一”的映射函数，用来修正这个错误。）。事实上，每个元素都是被独立操作的，而原始列表没有被更改，因为这里创建了一个新的列表来保存新的答案。这就是说，Map操作是可以高度并行的，这对高性能要求的应用以及并行计算领域的需求非常有用。

而化简操作指的是对一个列表的元素进行适当的合并（继续看前面的例子，如果有人想知道班级的平均分该怎么做？他可以定义一个化简函数，通过让列表中的奇数(odd)或偶数(even)元素跟自己的相邻的元素相加的方式把列表减半，如此递归运算直到列表只剩下一个元素，然后用这个元素除以人数，就得到了平均分）。虽然他不如映射函数那么并行，但是因为化简总是有一个简单的答案，大规模的运算相对独立，所以化简函数在高度并行环境下也很有用。

分布和可靠性

MapReduce通过把对数据集的大规模操作分发给网络上的每个节点实现可靠性；每个节点会周期性的把完成的工作和状态的更新报告回来。如果一个节点保持沉默超过一个预设的时间间隔，主节点（类同Google档案系统中的主服务器）记录下这个节点状态为死亡，并把分配给这个节点的数据发到别的节点。每个操作使用命名文件的原子操作以确保不会发生并行线程间的冲突；当文件被改名的时候，系统可能会把他们复制到任务名以外的另一个名字上去。（避免副作用）。

化简操作工作方式很类似，但是由于化简操作在并行能力较差，主节点会尽量把化简操作调度在一个节点上，或者离需要操作的数据尽可能近的节点上了；这个特性可以满足Google的需求，因为他们有足够的带宽，他们的内部网络没有那么多的机器。

用途

在Google，MapReduce用在非常广泛的应用程序中，包括“分布grep，分布排序，web连接图反转，每台机器的词矢量，web访问日志分析，反向索引构建，文档聚类，机器学习，基于统计的机器翻译……”值得注意的是，MapReduce实现以后，它被用来重新生成Google的整个索引，并取代老的ad hoc程序去更新索引。

MapReduce会生成大量的临时文件，为了提高效率，它利用Google档案系统来管理和访问这些文件。

2011年，Google发表Snappy压缩函式库开源项目；Snappy是Google MapReduce的一部分，以减少Network I/O 或 Disk I/O提升效能。^[1]

其他实现

• Hadoop － Apache软件基金会的开放源码项目，提供与MapReduce档案系统类似的功能。http://hadoop.apache.org/

http://zh.wikipedia.org/zh-cn/MapReduce

No related posts.

While developers are now rocking mobile development on the client side, their next challenge is how to code those tricky backend bits. A company facing those same exact problems right now isMedialets, a mobile rich media ad platform. What they do is help publishers create high quality interactive ads, though for our purposes their ad stuff isn’t that interesting. What I did find really interesting about their system is how they are tackling the problem of defeating the mobile device data deluge.

Each day Medialets munches on billions of new objects embedded in a stream of terabytes of raw event data flowing in from millions of mobile devices. All that data must be: generated on the mobile device; transmitted over lossy connections punctuated by long periods of disconnection; crunched; made available to reporting systems; fed back into control systems that must be able to respond within milliseconds to requests.

This will become a common paradigm for systems featuring mobile devices. The question is, how you can make it happen?

Now that’s interesting.

To help us understand more about how Medialets works, Joe Stein, Engineering Manager for Server Platforms at Medialets, was kind enough to talk to me about what they are doing. Joe also runs an excellent Hadoop podcast and blog called All Things Hadoop, check it out.

Joe has worked on this problem a lot and has some great ideas on how to build an effective mobile data cruncher using tools like Hadoop, MySQL, HBase, Cassandra, Ruby, Python, and Java…

Site

• http://www.medialets.com – home page.
• http://www.medialytics.com – insights dashboard for analyzing events from applications.

What Is Medialets?

Medialets delivers rich media advertising to mobile devices like the iPhone, iPad, and Android. Rich media means advertisements can be complex applications that embed, using a SDK, event generation and advertising functionality. The idea is that ads run inside the platform and instead of being lame adsense type ads, these can be fully interactive while providing the same brand quality they have on TV, except it’s on the mobile device. Applications can have you do things like shake the device or play a little football game with Michael Strahan. All this activity generates data that must be streamed back to their server farm for processing.

In addition to advertising they also provide very elaborate app based analytics for their publishers.

To see examples of their ads go here.

The Stats

• 2-3 TB of new data every day (uncompressed).
• Tens of billions of analytic events created per day. An event means somebody shook, turned off, rotated, etc. the app.
• 200+ Premium Applications run on tens of millions of mobile devices (iPhone, iPad, and Android).
• Over 700 billion analytics events already processed.
• During typical MapReduce jobs the can crunch on over 1 million events per second
• Data is often available in www.medialytics.com within 1 hour of coming in from mobile devices
• Approximately 100 servers total.
• Mobile is growing. Smartphone sales have outpaced PC sales for the first time and Medialets is seeing that in their growth. iPhone, iPad and Android devices increased nearly 300% in fourth quarter 2010. Android continues to grow market share, but iOS still dominates for premium mobile inventory.
• A dozen people are in engineering, mostly on the client side and Medialytics. The infrastructure team is 1-2 people.

Infrastructure

• Ad Servers instances running on quad core x 16GB w/ 1TB
• Event Tracking Servers  on 16 core x 12 GB w/ 10TB
• Event Processing, Job Execution, Log Collection & Aggregate Pre-Processing each 16 core 24GB RAM w/ 6TB of space
• Log Collection & Aggregate Pre-Processing and Post-Processing. Hadoop Cluster nodes each 16 core x 12GB RAM w/ 2x2TB (JBOD).
• www.medialytics.com varying configurations all with 16 cores from 12 to 96GB of RAM w/ 1-2TB

Open Source Systems & Tools In Production

• Linux
• Haproxy
• Tomcat
• MySQL
• Apache
• Passenger
• RoR
• Memcached
• Gearman
• Hadoop
• Pig
• Nagios
• Ganglia
• Git
• Jira

Languages

• C/C++ – ad server
• Java – data transformation
• Ruby – a lot server side Ruby. Probably more Ruby than C++, but moving more to Python.
• Python – a lot of MapReduce is being moved into Python using Python streaming.
• Scala
• Bash

Architecture

• The system was built over a couple years, mostly out of custom software, though they use Hadoop for the heavy lifting on the analytics side and MySQL as the database. Custom software was required to scale when they first started, but they are now considering more off-the-shelf products as these have evolved.
• The system is real-time in the sense that as data trickles in it’s made available as fast possible and as real as possible in reports.
• Not cloud based. They run on physical hardware. They couldn’t do what they need to do in a multitenancy environment in the cloud. Machines with 16 cores and terabytes of disk space is almost not enough.  They spend a lot of  time building their software to take advantage of the hardware, disk IO, CPU, parallel processing, utilizing as much memory as they can.
• Everything (almost) they do is asynchronous. You don’t need to be connected to a network to see an ad or for them to capture the analytics. Ads can be flighted (scheduled) weeks before a campaign kicks off and you can be on a subway and you’ll still be able to see the ad. Data collected during the ad is eventually passed to the server side.
• Publishers are responsible for building the app and integrate in the Medialets SDK. The SDK is specifically for analytics and advertisements. When they want run analytics or run ads in ad slots they use Medialets’ tools.
• There are three basic subsytems: ad serving, event processing, and reporting.
• Servers break down into a few tiers:
  • Forward facing tiers:
    • Ad Serving Tier – designed just to run ad servers.
    • Tracking Tier – handles data dumps and data loads.
  • Asynchronous processing tiers:
    • Hadoop Cluster –  runs on its own set of servers.
    • Java and Ruby Processes – take data that comes in and turns the data into a form usable by Hadoop to get the different aggregations going.
• Keep things as lean and mean as possible with only as much redundancy as needed.
  • Hardware is configured based on what the software is doing. Not all the machines are high end or low end. Different parts of the system have different hardware needs.
  • When responding to ad service requests there’s very little disk IO or computation. There’s a C++ service doing a lookup on static memory. So the ad serving machines have a lot of memory and lower end processors.
  • Where they have data intake they have a lot of sockets blocking and writing to disk, so they need very little memory.
  • The Hadoop MapReduce tier needs everything you can throw at it.
• The event handling flow goes something like:
  • Applications on mobile devices generate events. There are application events, ad events, and custom events. Custom events can be created by applications as key-value pairs and they’ll aggregate on that with no custom coding required.
  • Tracking servers receive events and write them into a log file of objects. The files represent a snap shot of time over which event data is collected, say seven minutes, for example.
  • Java servers read these files, unpack them, and run the objects through a series of thread pools so they are transformed and merged with all the necessary meta-data. Different processes pick up the different event types and process them.
  • The previous step creates a new file where the events are now complete after having been merged with meta-data. That new file is pushed into HDFS (the file system used by Hadoop).
  • MapReduce is run on these HDFS files to dedupe the data. After the deduplication occurs the data set is clean.
  • MapReduce jobs produce data that is then imported into databases that the analytics UI makes available to customers. Dozens of different types of aggregations are run. Standard metrics are provided, via Medialytics, showing what an app is doing from from both an ad and app reporting perspective. They aggregate along many different dimensions, for example, by device and by platform. Some of this data is feedback to the ad serving systems.
• Data duplication is a key design point on mobile devices and asynchronous systems.
  • A phone OS doesn’t guarantee an application it will get time to generate events. In the OnClose hook the publisher will run some cleanup logic, Medialets has some cleanup logic, then the event is written to a server which takes a couple milliseconds to respond, and then the local app has to update the local database. Even though this is a quick trip you are in this 15 millisecond window where the OS doesn’t guarantee all the functionality will execute. The OS could kill the process or there’s a crash. This will lead to duplicate events on replay depending on where the failure occurs. The iPhone’s new background feature complicates the accounting. If an app is backgrounded for 30 seconds or less it’s still considered the same run. There are a lot of variations.
  • Data can still come in from a phone that’s been off for 3 months. They have to be able know if they’ve seen that data before, a common occurrence if apps aren’t used all the time.
  • Hadoop is used to calculate metrics and aggregation, but it is also used for deduplication. When processing data they look at over a million events per second in order to dedupe the data. Everyday they get 10 billion events. They have to look at every event and decide if 1) they have seen the event before; 2) are they going to see the event again in the data set they are processing. Using MapReduce that data is broken up across a bunch of servers, which means you don’t really know until you’ve reduced everything that any of the data was duplicated.
• Advertising
  • Some types of event data are streamed back in real-time. For another class of data the event has to be opened and closed before an event is created. To know how many times someone used an app in a day, for example, there must be an open and a corresponding close event.
  • Users in the mobile world are truly unique. In the Internet world users, unless they have logged in, are almost impossible to quantify. A computer is used by many people and you can’t really identify who is using a device. A physical device like the phone is usually used by a single person, so aggregating on a device basis is essentially aggregating on a user basis.
  • They can gather stats like conversions along different dimensions. They can tell how long it takes someone to upgrade from version of the OS to another. Who are the different types of people? Then they can overlay that to the different types of apps that they have.
  • Ads are either brand or direct response advertisements, there’s a mix. A movie ad, for example, when clicked on instead of going to a website could bring up a local application from the device. That allows you to buy the tickets right in the app. Having different interceptors and the capability of creating a rich user experience makes it possible to monetize interaction flows in new ways.
  • When an app is launched an ad should be displayed immediately. Async is used for pushing data to the servers, but ads can be served synchronously. A lot of apps have a sponsor logo you see when the app opens and that must be served immediately. The first onload call is a sync call to get the ads. Their ad serving system can serve ads in under 200 milliseconds with a 3% variance.
• 70%-80% of their storage costs are saved because they store data in compressed sequence files.
  • HDFS has a sequence file format. Compression can be either on a row or a block basis. Let’s say you have a sequence file with 10 blocks in it and a block is defined as data that will go into your mapper (for MapReduce). If compression is on a block level and there are 10 blocks then those 10 blocks can be pushed to mappers in parallel and HDFS will handle all the decompression and streaming automatically.
  • Data that comes out of the reducer can be stored in the sequence file, say it’s the result of the deduplication process, or it can be stored in an uncompressed format that’s ready to load in another database.
  • A lot of people leave the data uncompressed. To use Hive the data has to be uncompressed so you can interact with it. It’s a matter of how much do you want to spend and where do you want the inefficiencies of your system to come into play.
  • It’s a con to have a lot of data sitting on disk that isn’t being interacting with in an uncompressed format. They selectively decide which format to use depending on if the decompression phase is worth the overhead compared to how long they keep that data on disk.
• Job Execution System
  • Built in Ruby, before suitable off-the-shelf systems were available.
  • They built a job processing system to implement workflow processing. A workflow is a set of jobs that have different tasks and steps and operate on different events. App data must be processed in many different ways and the results have to go into a few different systems, some into tables, and some into other reporting systems. All that is automated and scripted.
• Aggregated data is stored in MySQL for viewing by publishers and advertisers. They are reaching a limit of how much they can shard. They are looking at MongoDB and GridFs (which is part of MongoDB).
  • GridFs is looking good to store, scale, and serve their media files, which is making them consider using MondoDB to store aggregate results sets.
  • They are also looking at Cassandra and HBase to store their aggregate results sets. They would consider using the same infrastructure also for their tracking and event capture servers, which currently is completely custom written.
  • Cassandra looks attractive because it works across multiple datacenters. They would use this feature to have multiple clusters in the same dacenter and have writes occur on one cluster and reads in another, so the different traffic loads won’t step on each other. They don’t want to mix different types of traffic, so they don’t want to do MapReduce jobs, writing from HBase, and reading from HBase all on the same machines.
  • HBase is an attractive option because they already write so much data to HDFS that having those files available in HBase would be exciting. They’ve had reliability concerns over fsync, making sure data is written to disk, but those concerns have been addressed in recent releases. HBase doesn’t allow partitioning the data by different uses, which is what is attractive about Cassandra as Cassandra supports having different kinds of racks within the cluster.
  • Since they are already using HDFS moving all the data into Cassandra after it is processed isn’t attractive, that would double their hard drive requirements.
  • They like the idea of the coprocessors so they don’t have to move the data across the network. Each job is 2-3 terabytes so to truly parallelize that without moving data is very attractive.
  • TTL deletes in Cassandra are very attractive. Cassandra can easily handle their write load, so it can be used to store incoming events. Then all the work processes can take the mobile data out of Cassandra, merge it with meta-data from other databases, to make for a fully joined objects, then that could be written to HBase, then they could do MapReduce aggregations and write the results to MongoDB.
  • An alternative dedupe design would be to write it all to HBase and just pick the last one as the winner. Once these systems are in-place they’ll rethink some of their existing processes to see how they could take advantage.
  • A lot of prototyping trying was undertaken to figure out if they should keep with their existing software, move to another database, or stay with MySQL. They may end up with MongoDB, Cassandra, and HBase, they just want to find the right mix of capabilities for the new products they are building, and figure out how they can continue to scale without soaking up a lot of developer time.
• AdServers are written in C++
  • This layer provides a standard ad scheduling facility so ads can be mapped to slots, rotated, targeted, etc. Targeting can be based on platform, resolution, geography and other dimensions.
  • There’s an object cache for database data that is used to make ad delivery decisions.
  • 99% of the time the cache is sufficient, 1% of the time they have to hit the database.
  • Reads are in a few microseconds, but increase to 200 microseconds when they have to hit the database.
  • Also responsible for pacing ads so a campaign can be delivered across the lifetime of an app. If an app is run a million times a day, for example, and the ad campaign is for a million impressions, they don’t want to use it up in one day. Let’s say the advertiser wants the campaign to run for a month. The ad server looks at the analytic data and the rate requests are coming in to calculate what pace ads should be delivered.
  • A lot of decisions are precomputed. A human will slot target, saying where an add should be displayed.
  • Decisions like geographical distribution are calculated on the fly. If you have a set of ad impressions to give out you’ll want some to go to Canada and some to go the US, for example.
• Java Servers
  • Join meta-data with the data sets that come in. Meta-data is pulled from cache at 95% hit rate. When they have a new advertisement that went live, for example, they’ll hit the database.
  • Going to the database they want to be as optimistic as possible and interrupt as few as threads as possible, so they use atomic variables and CAS (compare and set) when doing very heavy read operations and very few writes. This switch increased performance 15%-20% because they are no longer blocking on writes.
  • For the amount of reads they were doing they benchmarked it and found a Mutex took to long. Semaphores ended up blocking on the writer. Say there were 10 threads and 9 could read so no threads would block, but the 10th thread had to write which would block all the threads. This increased latency and didn’t perform well compared to looping and doing the compare and set. It’s possible because they are continually processing gigabytes of data that inside the JVM something was being blocked.
  • Used concurrent memoizer pattern to create thread pools that handle cache requests. The cache is a pool that will load data when required. The load uses CAS to block the actual reads that are occurring.
  • SEDA is used to process data through all it’s different transformations. Each thread pool performs a state transformation on a chunk of data and then the data is forwarded onto another thread pool for the next transformation. For example, the first stage is to read the data off disk and serializing it onto an object array. These operations are not latency sensitive.
• Using Ruby
  • When using Ruby one must fork to really multiprocess functions effectively.
  • Rinda is used to create concurrencies across forked processes. Sometimes a database is used for coordination.
  • This hides any memory leak problems or green thread problems common to interpreters.
• Monitoring
  • Mix of internal tools and Nagios.
  • Do a lot of trending of their own logs with Ganglia across all their different tiers.
  • They take a very proactive monitoring approach and spend a lot of R&D effort so they can know they have problems before they occur.
  • Trending feeds into their monitoring. If one of their ad servers stops responding within 10 milliseconds, for more than 1 or 2 requests, within a second, they need to know about that.
  • If request latencies go up from 200 milliseconds to 800 milliseconds on average they want to know.
  • They log a lot so debugging of problems occurs through the logs.

Lessons Learned

• Turn Data into Products. Development knows the data they have. That knowledge can be used to help the product team create new products to sell to customers. There’s always a big gap between R&D and the business. Help the business be in tune with what R&D is doing. Have them understand the power of Hadoop, the data they crunch and how fast they can crunch it. A good example is a new Conversion Attribution product. If a user sees a static ad, clicks on it and downloads the app, it’s likely that static ad wasn’t the sole reason for the conversion. If, for example, the user experienced a rich media ad the day before (or two weeks earlier) then that ad would be apportioned some credit for the conversion, based on publisher configurable criteria. This kind of capability is only possible with a robust data processing infrastructure. Without R&D making it known that this sort of feature is even possible, it’s not likely these new kinds  of high value-add products could be developed and sold to customers.
• Explore New Tools. It’s a complex world of new tools. All these new technologies make it challenging to know where to put functionality. A feature can be done in so many different ways and would be done differently depending on the tool (HBase, Cassandra, MongoDB). Should the dedupe still be done in MapReduce or should it be done using coprocessors? Is it worth doubling disk by supporting two different databases? Is partitioning data by usage pattern really or win or can all the traffic patterns work on the same system? Prototype your options and think about how your architecture could change with each new tool, working alone or working together.
• Monitor and Capacity Plan Proactively. Turn monitoring data around into planning for infrastructure. If you don’t do that you’ll just keep firefighting issues. Build your proactive alerts and trending data so even before it becomes a warning you can see it coming. Sometimes you just need another server. More data simply means more servers. It’s a cost of doing business.  Knowing what server to put where in order to keep all the different dataflows going appropriately, that’s the trick. It’s important to compare the trends of CPU and load and to really look at the infrastructure as a whole and create a plan based on that.
• Look at Data from a Product Operations Perspective. Look at new applications and see how they are similar to what has been implemented before as a way to figure out how you need to scale. Just because a new app comes on board doesn’t mean need new ad servers, new data nodes, and new tracking servers need to be added. It depends. A lot of different ads make a small amount of ad requests but send a ridiculous amount of data. Trend logs to look at spikes in the data and see how latency is related to the spikes. This tells you where and how different parts of the system need to scale .

Related Articles

• Joe Stein’s AllThingsHadoop Twitter Feed
• Tackling Big Data Problems at Scale by Joe Stein
• Published Q4 2010 mobile figures http://www.medialets.com/medialets-data-spotlight-mobile-rich-media-momentum-q4-2011/
• Hadoop, BigData and Cassandra with Jonathan Ellis – HBase is to OLAP as Cassandra is to OLTP
• Twitter’s Plan To Analyze 100 Billion Tweets

http://highscalability.com/blog/2011/3/8/medialets-architecture-defeating-the-daunting-mobile-device.html

Related posts:

1. TripAdvisor Architecture – 40M Visitors, 200M Dynamic Page Views, 30TB Data
2. Oracle Metalink Reference – Data guard
3. Data Recovery Advisor – oracle 11g new feature

From the time Mollom opened its digital inspection system they’ve rejected over 373 million spams and in the process they’ve learned that a stunning 90% of all messages are spam. This spam torrent is handled by only two geographically distributed machines that handle 100 requests/ second, each running a Java application server and Cassandra. So few resources are necessary because they’ve created a very efficient machine learning system. Isn’t that cool? So, how do they do it?

To find out I interviewed Benjamin Schrauwen, cofounder of Mollom, and Johan Vos, Glassfish and Java enterprise expert. Proving software knows no national boundaries, Mollom HQ is located inBelgium (other good things from Belgium: Hercule Poirot, chocolate, waffles).

Statistics

• Serving 40,000 active websites, many of which are very large customers like Sony Music, Warner Brothers, Fox News, and The Economist. A lot of big brands, with big websites, and a lot of comments.
• Find 1/2 million spam messages each day.
• Handle 100 API calls/second.
• A spam check is low latency, taking between 30-50msecs. The slowest connection would be 500msec. The 95th percentile of latency is 250msecs. It’s really optimized for speed.
• Spam classification efficiency is at 99.95%. This means that only 5 in 10,000 spam messages were not caught by Mollom.
• Netlog, which is a social networking site in Europe, has their own Mollom setup in their own datacenter. Netlog handles about 4 million messages a day on custom classifiers that are trained on their data.

Platform

• Two production servers run in two different datacenters for failover.
  • One server is on the East coast and one is on the West coast.
  • Each server is an Intel Xeon Quad core, 2.8GHz, 16GB RAM, 4 disks of 300 GB, RAID 10.
• SoftLayer – the machines are hosted by SoftLayer.
• Cassandra – a NoSQL database selected for it’s write performance and ability to operate across multiple datacenters.
• Glassfish – open source application server for the Java EE platform. They picked Glassfish for it’s enterprise ready features like replication and failover.
• Hudson – provides for continuous testing and deployment of the backend across all their servers.
• Java – From the start Mollom was written in Java.
• Munin – used to measure and plot metrics concerning server health.
• MySQL - JPA (The Java Persistence API) is used for regular data sets and Cassandra is used for large data sets.
• Pingdom – used for uptime monitoring.
• Zendesk – used for support.
• Drupal – used for the main website with a custom E-commerce module.
• Unfuddle – Subversion hosting used for source code control by their distributed development team.

What Is Mollom?

Mollom is a web service for filtering out various types of spam from user generated content: comments, forum posts, blog posts, polls, contact forms, registration forms, and password request forms. Spam determination is not only based on the posted content, but also on the past activity and reputation of the poster. Mollom’s machine learning algorithms act as your 24×7 digital moderator, so you don’t have to.

How Is It Used?

Applications like Drupal, for example, integrate Mollom using a Module that installs itself into content editing integration points so content can be checked for spam before being written to the database. The process looks like:

• When a user submit comments to a website an API call is made to the backend servers.
• The content is analyzed, if it is spam the site will be told to block it, or if the backend is unsure it will advise the site to show a CAPTCHA, which they also serve.
• When the CAPTCHA is filled in correctly the content will be accepted. In most cases a human will not see a CAPTCHA, the content will directly be accepted as ham, ham being good content and spam being bad content.
• CAPTCHA is only displayed with the machine learning algorithms are not 100 percent sure, so for the most part humans are not inconvenienced.

Dashboard

Mollom includes a pretty nifty dashboard for each account that shows you how much ham has been accepted and how much spam has been rejected. The amount of spam that one sees in the graph is really depressing.

Operations Process

Installation. Installation is quite easy for Drupal. Install it like any other module. Create an account on the Mollom website. Get a pair of security keys, configure these keys into the module, and select which parts of the system you want to protect with Mollum. That’s about it.

Daily. I check regularly to see if spam has gotten through. It’s not 100% so some spam does get through, but very little. If spam does get through there’s a way to tell Mollom that this post was really spam and should be deleted. This is what you would have to do anyway, but in the process you are helping train Mollom’s machine learning algorithms about what is spam.

Allows for anonymous user interaction. With a good spam checker it’s possible to have a site where people can interact anonymously, which is what a lot of people using certain types of sites really prefer. Once you require registration engagement goes way down and registration doesn’t stop spammers anyway.

Not Everything Is Rosy

Dealing with false positives is Mollom’s biggest downside. Spam detection is a difficult balancing act between rejecting ham and accepting spam. Mollom’s machine learning algorithms seems to work quite well, but there is a problem sometimes with good posts being rejected, you get the dreaded:Your submission has triggered the spam filter and will not be accepted. Currently there is no recourse. Few things piss off a user more than having their glorious comment rejected as spam when it’s obvious, to a human, that it’s not. A user will only try a few times to get around the problem and then they will simply give up and walk away.

The problem is there is no way to fix the problem. To protect the machine learning algorithms from being gamed, Mollom does not allow you to present an example of a wrongly rejected chunk of content that should be accepted, though they are working on adding this in the future.

It’s a tough decision. Static CAPTCHA systems, that is systems that only require a user pass a test to submit content, simply do not work once a site has been targeted for serious attack. User registration doesn’t work. Moderating every post requires a very high burden, especially for “hobby” sites, given a site can have thousands of spams a day. And spam completely kills a site, so the risk of angering some users has to be balanced against having no users in the end because of a bombed-out site.

Business Model

• What makes Mollom a no-brainer is it is free to try. You only pay once your site starts accepting over 100 ham messages a day. For a small site this may never happen.
• Once you pass the free threshold there is Mollom Plus (1 EUR a day) and Mollom Premium (3600 EUR/year/site) pricing tiers, which seemed pretty reasonable.
• The free websites are not a resource drain as you might expect, they are are actually a source of critical training data. All websites using Mollom are constantly feeding back data to the backend classifiers. The more people that using Mollom the better is the training through all the feedback it is getting from users. Without free sites Mollom wouldn’t be as accurate as it is.

Architecture

• Mollom is very engineering driven. A major emphasis has been put on making Mollom as efficient as possible both in code and server resource usage.
• Physically each server can handle all requests, but they have complete failover. Work is distributed between the machines. If one goes down then work moves to the other machine.
  • They used to have 3 servers, but since they improved the performance a lot they can use the third server as a staging server.
  • Each server can handle a full 100 connections per second and each connection runs the entire pipeline: full text analysis, calculating the reputations of the authors, and serving CAPTCHAS.
• Really optimized for low latency. Since the spam detection is part of the process of content being submitted to a site, if it took a long time it would be really annoying to users.
• Mollom has went through several phases of evolution:
  1. Initially a small team of two people worked part time on the algorithms, the classifiers, the real business problems they were trying to solve. To build out the infrastructure on the backend they used their own implementation of thread-pool, connection-pool, resource management. They found they were spending too much time on supporting all this stuff and making it scale. They then switched to Glassfish, a Java application server, so they would have worry a lot less about memory management, REST handling, XML parsing, and database connection pooling.
  2. The main problem in the past was disk bandwidth. They need to track reputations for all IP addresses and for all URLs on the Internet, so this was a massive datastore with a lot of random access.
  3. In the early days they used a cheap virtualized machine and everything was in MySQL, which didn’t scale.
  4. They then moved to solid state disk and stored everything in files. Solid state solved the write problem, but there were issues:
     1. It’s really expensive.
     2. It’s very sensitive to the type of file system installed.
     3. Writes are fast, but iterating over the data, which they did quite often, to clean up the data or to train a new classifier over millions of millions of small objects, was still very slow.
  5. They then moved away from solid state and moved to Cassandra.
• Cassandra is now used as their database for write heavy loads and as a caching layer:
  • Runs on a RAID 10 disk configuration (striped and mirrored), which are very good for heavy read/writes.
  • Cassandra is optimized for writes and Mollom has a lot more writes than reads.
  • Designed to be distributed inside a datacenter and across datacenters.
  • A downside is there is no standard NoSQL interface which makes it difficult to write applications.
  • Cassandra’s row caching saves them from having to add another caching layer in their system, which removes a lot of application code.
  • Cassandra has an aging feature which will automatically deletes data after a period of time. Europe has strict privacy laws which require certain data be removed after a period of time. This feature is a huge win. It’s an extremely expensive operation and Cassandra handling it removes a lot of application code.
• The path a blog comment takes through the system:
  • A request can come from any client. Clients load balance requests across servers. This part is explained later. The typical client is a Drupal system. Requests can be XML-RPC or REST.
  • Requests are handled by a Glassfish application server and follow a typical application server workflow: requests are handled by servlets and are delegated to session beans.
  • Paying customers are serviced first, free customers may experience longer delays.
  • The request is parsed and analyzed. More on this later. A spam score is determined and returned to the user. So there are different functional parts of Mollom: spam checking and CAPTCHA handling. CAPTCHA includes generating, serving, and processing the responses. Different sessions beans are responsible for the parts of the Mollom functionality.
  • Classifiers are fully in RAM. A small piece of content is blown up and broken into thousands and thousands of tiny tokens that could identify spam. These classifiers have in the area of a few million tokens in RAM. The classifiers need to run really quickly so they have to be in RAM.
  • What is kept in Cassandra are reputation scores, frequencies, URLs, and IP addresses. Cassandra’s new row caching feature now acts as their caching layer. Previously they implemented an internal cache, but that was removed.
  • Both machines in both datacenters run Cassandra and the Glassfish application server. Cassandra is constantly replicating data between datacenters.
  • The in-memory datastructures are not replicated directly. They write through to Cassandra which then replicates. The cache on the other side has a timeout so it will go to Cassandra and pick up the new data. It’s eventually consistent. There’s a small window of inconsistency, but the models aren’t negatively impacted over such a short period of time.
  • Consistency is managed on a case-by-case basis. For reputation and IP addresses eventual consistency is fine. Session data, including CAPTCHA sessions, is kept strictly consistent so it will follow correctly when a machine fails over.
• Client Side Load Balancing
  • Mollom uses client side load balancing that distributes load based on latency, etc. As a startup they didn’t have the money to buy a large load balancer. They also had a goal to be able to do global load balancing over multiple datacenters, which would have required an expensive and complicated setup.
  • Management of the client list is through an API. Each client has a list of available servers it can use.
  • Each client can get a different list of servers to use. Paying customers can be provided a list of closer servers to reduce latency.
  • When a server fails the clients will try the next server in the list.
  • Client side load balancing helped with migration from the old system to the new Glassfish based system. New users were given the address of the migrated machines and old users still worked on the old machines and could be migrated in orderly fashion by updating their server lists. This allowed testing so they could test functionality and then scaling and performance. They looked at responses times, how many connections were in the connection queue, and how long connections would stay in the queue. They could test what would happen if they increased threads in the thread pool, changed the number of JDBC connections, and other configurations. Once everyone migrated the old servers were shutdown. Very little downtime was experienced during the transition, which is key for a highly available system. While the system is down spams are getting through.
  • A downside of the client side approach is that if third party clients are written poorly then there will be problems. The client could get a server list, for example, and iterate over it in reverse order, which is wrong. They now work closely with client developers and provide quality reference code examples so developers can learn best practices.
• Machine Learning
  • Mollom is a set of learning systems. A stand-alone CAPTCHA solution, which neither considers user behavior nor point of origin, can never achieve this level of informed protection, and generally requires users to solve a CAPTCHA on every post. Using Mollom’s text analysis, users must only solve CAPTCHAs when Mollom is unsure about a post.
  • The average message length is about 500 characters and this is blown up into 3,000 features. The spaminess of a post is determined by looking at the reputation of the IP address or Open ID, they look at the user ID, sentiment, language, profanity, specific words and combinations of words, they look at how well the text has been written, etc. All these are based on classifiers. Some of the classifiers are statistical in nature that can learn in automatically. Some classifiers that are rule based to ensure they can never go wrong. It’s a combination of all these tests in the end that determine the spam score.
  • They learn from this process and the classifiers and internal metrics are updated in real-time.
• Glassfish handles work scheduling and was designed to handle multicore workloads.
  • The key is to create a design the works in parallel as much as possible and has as small a lock window as possible.
  • The number concurrent HTTP connections is tuned so they have an appropriately sized pool of available connections.
  • They uses 16 threads per server.
  • Most of the calls are handled by stateless session beans which work well for concurrency management.
  • They keep a number of session beans in a pool, but let Glassfish decide how many should be in a pool. At peak load there will be more session beans in the pool so requests can be handled efficiently. 32 session beans can be running in parallel at any given moment.
  • All the classifiers are actually session beans that are being reused by different threads.
  • Each session bean has their own client connection to Cassandra so they will not block each other.
  • When a user doesn’t respond to a CAPTCHA that session is cleaned up and the Mollom learns that was probably spam.
  • There is one instance of each classifier per server.
  • There is a very short period of lock contention on session cleanup where classifiers are being updated.
  • Updated classifiers are written back to Cassandra every 1/2 hour.
• Application Integration
  • Mollom uses an open API which can be integrated into any system.
  • Libraries: Java, PHP, Ruby, and more.
  • Integrated solutions: Drupal, Joomla, WordPress, and other content management systems.
  • Third parties generate new bindings based on example code produced by Mollom.
• To monitor the health of a server they continually monitor, using Munin:
  • What is the size of the heap after garbage collection?
  • What is the number of available connections?
  • What is the number of threads available in the thread pool? Make sure there’s no one thread waiting for a long time for a lock.
• When you look at their architecture, Mollom is trying to build an architecture that can transparently work out of multiple datacenters a set themselves up to scale-out when they’ve outgrown a single server system:
  • Client side load balancing is used to select and failover servers.
  • Glassfish clustering will be used to failover at the application tier and to make is easy to add and remove machines.
  • Cassandra will be used manage the data tier across datacenters.
• Netlog’s installation of Mollom has some interesting characteristics. It process more than the main Mollom.com servers, but the distribution of spam is completely different because their people communicating are part of social network. The spam distribution on Netlog 90% ham and 10% spam, where it’s exactly the reverse out in the cruel world of blogs.  The interesting implication is that it takes less resources to process ham so they can actually perform more work on the same servers.
• They initially tried virtualized servers, and thought of using Amazon, but they found IO was the main bottleneck using shared virtualized servers. IO latency and bandwidth were real problems so they decided to go scale-up and use bigger machines and bigger disks.
  • Surprisingly they are not CPU bound. Only two of the 8 cores are computing. The others are just doing IO.
  • Mollom’s traffic is fairly constant, so having a dedicated servers is more cost efficient. They see Amazon more of a way to handle peak loads.
• Development Process
  • They are distributed team. Three people are in Belgium, someone in Texas, Boston, and Germany.
  • Scrum is used as the development process and they are very happy with it. The scrum meeting happens over Skype at 2PM. They’ve found as they’ve grown they need more process.
  • Developers develop locally and submit code into Unfuddle.
  • Hudson is used as their continuous integration environment. Hudson made their migration from the old to the new Glassfish system easier because tests had to be passed before deployment. They didn’t lose much time because problems were found before production deployment.
  • They have many tests: unit tests, system tests, Drupal tests. Only when these tests are passed by Hudson can the system be deployed.
  • Deployment is still done manually to reduce potential downtime.
  • Whenever they found a problem with garbage collection times it was always due to a memory leak in their application. In case of a memory leak they take a core dump. To analyze a core dump from a 16GB machine is not that easy, you probably can’t analyze it on your local machine, so what they do is rent a large memory instance on Amazon to analyze the dump. It takes about 2 hours to process the heap dump. They compare two dumps, one after 10 hours of execution time and one after 20 hours of execution time. If there are major differences then it’s probably because of a memory leak.

Future Directions

• Mollom API uses XML-RPC, they are now testing a REST implementation to make it easier for services to mashup with Mollom.
• Now that they’ve transitioned to Cassandra it will be easier for them to scale-out when growth dictates.
• Soon to be released are enterprise features that make it possible to manage hundreds of websites as a unit. It will be easy to moderate across a set of websites by sentiment, spam score, or delete all the comments from certain IP addresses.
• They’ve had some talks about getting into the streaming data business like Twitter, but they are restricted by Europe’s stricter privacy policies.
• They will experiment using Glassfish for load balancing in each datacenter.
• If the load goes up 10x they will have to add more Cassandra nodes. Disk IO is the bottleneck. Only when they have to grow more than 10x will they need to add more application servers.

Lessons Learned

• Efficiency leads to happiness. Mollom takes high performance engineering very seriously. They are proud that Mollom is extremely cost effective. It can handle many many requests on a single server, with low latency, which makes customers happy, which makes them happy because they don’t have to maintain lots of machines, and costs are low. They’ve made this a priority from the start and have chosen the right technologies to achieve their goals. This enabled them to take the profit they’ve made and invest in marketing, building a user base, and building new products on top of Mollom.
• Free for breadth, pay for depth. Machine learning requires a lot of example data to be able to detect spam successfully. To get that data Mollom offers a free service to customers, who provide the breadth of data needed to better train the learning algorithms, they are a constant source of intelligence and feedback. Larger customers provide the revenue and also benefit from the data learned from the free clients. This model seems peculiar to big data and machine learning, which as we all know is the future of everything.
• Remove non domain-specific obstacles. Big systems take a lot of infrastructure work. The infrastructure effort often takes work away from work on a product’s true value producing domain related features (classifiers, reputation systems, client libraries). Mollom consciously tried to get out of the infrastructure business as much as possible with their selections of Cassandra and Glassfish.
• Be careful with client side code. Code on the client side is attractive because it uses other people’s resources instead of yours. The problem is that code can be poorly written which will make your system look bad. Work closely with client developers and provide quality reference code examples so developers can learn best practices.
• Prioritize paying customers. Paying customers get a better quality of service. They are handled first in the queues and experience less delay through the system. Paying customers have access to failover server and free customers only have one server.
• Reduce code by letting the stack do the heavy lifting. In the early days the Mollom code base was a lot bigger than it is now. Cassandra removed a lot of complex code by handling replication and row caching and Glassfish removed a lot application code and will handle clustering. Simplify over time.
• Minimize lock contention. Mollom spent a lot of time working on minimizing lock contention in the Glassfish server as this became the major bottleneck. Lock as little is possible to maintain full parallelism.

Related Articles

• Mollom API
• Mollom Drupal Module
• Mollom Technical Whitepaper
• Mollom’s Twitter Account
• Episode #072 – Mollom.com’s GlassFish backend with Dries and Johan
• Mollom gets a new backend
• Fighting spam with Mollom on Glassfish
• To learn more about big data and machine learning take a look at some materials from the Strata Conference.

http://highscalability.com/blog/2011/2/8/mollom-architecture-killing-over-373-million-spams-at-100-re.html

Related posts:

1. Friends For Sale Architecture – A 300 Million Page View/Month Facebook RoR App
2. Playfish’s Social Gaming Architecture – 50 Million Monthly Users And Growing
3. Mailinator Architecture