Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight
• Infrastructure Q1: Cloud and big data woo enterprises
• Migrating media applications to the private cloud: best practices for businesses

Derrick covered some of the techniques being used by Ancestry.com back in June, and today we can reveal the latest weapon in MyHeritage’s arsenal: automated record matching.

Both platforms lean heavily on records as a way of augmenting the drier names and dates that make up family trees, but the Israel-based MyHeritage – which already has its own angle by explicitly treating the service like a social network – reckons it now has the edge.

According to CEO Gilad Japhet, MyHeritage has had its Record Matching tech ready for some time, but needed to set up a server farm, then clear a backlog of four billion historical records (including the world’s largest historical newspaper collection, acquired through the company’s FamilyLink buy last year), before launching it today.

“They come from original documents, birth records, marriage certificates, passenger lists going through Ellis Island, tombstones – in a few cases user contributed, as some people take snapshots of gravestones and upload them – public information, census records, newspaper articles and books. Record Matching covers both text-based and structured records, those that can be filled into a regular database,” he told me.

As an example, let’s say you don’t know the date of birth or death for your grandfather, but you do know his name. MyHeritage has a big database of wills, but again, you’re lacking dates. So the service would use its already-existing Smart Matching technology to compare the known information with that on other family trees, perhaps pinning down dates through other relatives’ connections.

Then, armed with that, it would find what it can in those historical records, using semantic analysis to deal with the free-text newspaper cuttings for example.

The smart thing, and one that Japhet hopes will pull in more subscribers and pay-as-you-go credit users, is that Record Matching works automatically and provides snippets of information for free. If you’re a user, you’ll just get an email telling you what’s been found. If you want to see the full record, you pay, but it doesn’t require that step to prove its worth.

So why did MyHeritage decide to shun the cloud for all this?

“We found it wasn’t very efficient to run this in the cloud because the CPU power you get is typically smaller, as a lot of these servers are virtual,” Japhet said. “We wanted serious number-crunching capabilities, and found it more efficient for us to purchase high-end servers, put together a large farm, run it all and accumulate the matches. It’s an ongoing real-time system.”

Japhet also claims other advantages over Ancestry.com, an older and larger service (38 million family trees to MyHeritage’s 23 million). For one thing, he points out that MyHeritage is available in 38 languages and its rival in just half a dozen – that makes a difference when you consider the international aspect of genealogical research.

What’s more, MyHeritage intends to “launch a massive crowdsourcing based transcription system” for its users within the next year, he added. And so the battle for family history continues.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• The evolution of consumer-media cloud storage
• Controversy, courtrooms and the cloud in Q1
• AWS Storage Gateway jolts cloud-storage ecosystem

It you’ve used Ancestry.com recently, you’re probably thankful for its efforts. According to Head of Engineering Scott Sorenson, Ancestry.com has more than 10 billion records that are part of a 4-petabyte (or 4-million gigabyte) data store. If you’re searching for “John Smith,” he explained, it probably has about 60 million for “Smith” and about 4 million for “John Smith,” but you’re only interested in the relative handful that are relevant to your John Smith.

Making models smarter

That’s why Ancestry.com is using machine learning to make sorting through those records a lot less like finding a needle in a haystack and a lot more like having that needle — and any others made from the same batch of steel — delivered right to your door. Here’s how the process works, in a nutshell:

1. Crawl digital records (e.g., newspapers, birth records, death records, census data, ship manifests, etc.) online and extract relevant data
2. (Or 1(a)) Scan, upload and index physical records (via a partner in China)
3. Stitch together new records with user data to add more context
4. And this is key, constantly analyze user behavior in order to make its algorithms smarter

As users make judgments about the records they’re presented, Sorenson said, Ancestry.com’s algorithms get better at performing their particular tasks. So, a system for extracting data from newspaper pages might be able to better recognize the various sections of the page (so as to ignore the ads, for example) and then be able to adjust for mistakes in the section it is analyzing. And as with Google’s search algorithms, the more that users interact with records, the better Ancestry.com’s sorting algorithms are able to determine those records relevance to any given user.

Spit in a tube, pay $99, learn your past

Oh, but Ancestry.com has decided that merely storing and analyzing historical records is just the beginning with regard to providing accurate genealogy information. It also will sequence your DNA, focusing on 700,000 markers important to determining one’s race, lineage and other factors. That service, which simply requires users to swab their cheek or spit in a tube and send it to the lab, costs only $99 (a full genome sequence would cost at least 10 times that, by the way), but could revolutionize the accuracy of Ancestry.com’s models.

Right now, Sorenson said, the DNA service can tell users their race and what country they’re from, and also connect them with other relatives who share a DNA profile. (If your privacy red flag has gone up reading this, Sorenson did note the following: all communications with relatives are optional and initially anonymous; all DNA information is disassociated from personal information; and users get their sequence results via an encrypted key “that we treat with a higher level of security than we’d store your credit card information.”)

Connecting with distant relatives can be valuable, though. A third cousin, for example, might have ancestral information that you don’t, which will help make your family tree that much more accurate. But Sorenson said when it really gets interesting is when Ancestry.com can combine DNA data with record data in family trees. Someone’s DNA might indicate he’s from France, Sorenson explained, but cross-checking that against that person’s family data will let the service discover he’s actually from the Normandy region.

Going forward, Sorenson said Ancestry.com expects its DNA service to take off like a rocket. The company is investing between $10 million and $15 million into that service over the next couple years, and has bioinformatic scientists on staff trying to scale algorithms designed to handle hundreds of samples to work with hundreds of thousands or even millions of samples. In that regard, though, Ancestry.com isn’t alone — the steady drop in the price of genome sequencing has everyone in the sector anticipating skyrocketing data volumes.

What’s next: Telling stories and making genealogy real-time

OK, so it has billions of records and our DNA, what more can Ancestry.com possibly want or need to provide us information on our ancestors? Nothing, actually.

It just needs to make better use of what it does have and the new technologies available for working with that information. Genealogy has traditionally been “dusty,” Sorenson explained, but Ancestry.com is trying to tell the stories behind those dusty records. If you’ve seen the NBC program “Who Do You Think You Are?”, on which Ancestry.com traces celebrities’ ancestral roots, you have an idea of what Sorenson is talking about.

For example, by improving its image-processing capabilities, Ancestry.com could extract more information than just name, data and location from old records that it already knows how to process. It could tell someone that his grandfather was the only person on the block to own a radio, or whether he owned his home. Combined with socioeconomic and other external data, Sorenson said, Ancestry.com could “create a really vivid picture” of what it was like to live during a specific time.

By using location data from cell phones, Sorenson said Ancestry.com could deliver a mobile experience that’s far more than a translation of the web on a smaller screen by making genealogy a geospatial pursuit. For example, Sorenson, explained, if a user takes a picture of a gravestone, Ancestry.com would like to provide him with relevant historical data related to that place, and maybe even some nearby points of interest.

Some might think Ancestry.com’s practices and plans toe the privacy line, but if someone has to toe that line, this might be the company to do it. In a fast-paced world it’s easy to get tied up in the moment and in our own little worlds — especially with big data being used elsewhere on the web to keep our attention firmly on one site or another. Using personal data to let users dig into decades into their family histories ends up looking very refreshing.

Feature image courtesy of Shutterstock user tovovan.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

• Connected world: the consumer technology revolution
• Takeaways from the second quarter in cloud and data
• The importance of putting the U and I in visualization