The Challenges of Leveraging Data Streams in the Financial Services Industry

Data has long been viewed as a strategic asset by financial service institutions. Driven by a continuing need to address regulatory compliance and fraud detection, these institutions have developed deep expertise in analytics around both streaming and non-streaming data sources. Each customer transaction, act of fraud, credit default, and complaint provides organizations the experience from which to learn and create value from their data and better address their regulatory compliance requirements.

Financial fraud is on the rise. The Nilson Report shows that losses from worldwide fraud on credit cards, debit cards and prepaid cards hit $16.31 billion in 2014, a growth of 19% from 2013 with the U.S. alone accounting for 48.2% of this total. Fraudsters use very sophisticated and technology-enabled techniques. Some of the major victims of fraud are credit card companies, insurance companies, retail merchants, and telecom service providers. Fraud analysis is used to take various forms of stored data and streaming data and convert them into actionable insights to detect fraud and take necessary required actions.

read more

Exchange Market Blog Series, Part 3: Enterprise Level Exchanges

This is the third and final installment in a blog series examining exchange markets and how they are a good use case for graph databases.

In college, I was in a student group that wrote a research paper about redesigning election systems, which were traditionally dominated by two parties, to include viable third-party candidates. This was inspired by the two consecutive failed presidential bids by third-party candidate Ross Perot in 1992 and 1996. We found that the simplest way to reasonably include third-party candidates was to allow voters to rank two or more candidates, instead of voting for a single candidate.

The problem though was that if most voters preferred the same second-rank candidate, then that second-rank candidate would likely be elected. You can imagine in this scenario that candidates might purposely pursue becoming the second-rank candidate in order to win. In a market design like this, most candidates and political strategists would attempt to exploit the weaknesses of the system to engineer the outcome.

read more

Powerful Persistence: ThingSpan & SparkSQL

There are many reasons Spark is fast becoming the defacto standard for large scale data processing. While clever use of in memory computing, optimized execution and built in machine learning libraries are often cited as reasons for its meteoric rise in popularity, it’s the way Spark has embraced structured data and external sources that I find particularly impressive.

No matter what your reason for using Spark, it will almost certainly involve reading data from external sources. A common use case is to consume large quantities of unstructured operational data dumped into HDFS and fuse it with structured historical metadata that represents the system’s learning or knowledge over time. Typically, this knowledge repository is maintained in a database that can also be leveraged and updated by other applications, business systems etc.

Over the past few releases of Spark, SparkSQL and the Dataframes API have evolved as a powerful way to interact with structured data. At the lowest level it allows an external datastore to be represented as a set of Dataframes which are akin to virtual SQL like tables. This allows the use of SQL to access data from disparate datasources, even joining across tables that derive from totally separate physical datastores.

read more

Exchange Market Blog Series, Part 2: Designing an Exchange Market with the Help of “My Little Pony”

This is the second installment in a blog series examining exchange markets and how they are a good use case for using a graph database.

Civilization has designed exchange markets since the first early humans made stone tools and others recognized their value. In fact, non-monetary exchange markets predate all other markets by such a great deal of time that you might say that we are inherently predisposed towards them. Of course, transacting in exchange markets can be extremely difficult, because value is subjective, and we have a tendency to overvalue what we have. Successfully making a trade often involves tedious work to first find potential participants and then convince them to close the deal.

This challenge is well illustrated in an episode of the animated show, “My Little Pony: Friendship is Magic™,” titled “Trade Ya!” where one of the ponies named Rainbow Dash wants to trade for a rare book held by a book seller. In the Rainbow Falls Traders Exchange, no money is used. Instead, the ponies must trade only for goods or services. Rainbow Dash brings her most valuable possession, her lucky horseshoe, to trade for a first edition copy of the Daring Do book. Unfortunately, the book seller does not find her horseshoe valuable, so the rest of the episode involves Rainbow Dash going from vendor to vendor to enact a series of trades starting with her “valuable” horseshoe and ending with the rare book.

read more

Why Software is Our Greatest Crime-Fighting Superhero

The Federal Bureau of Investigation recently released its annual report, “Crime in the United States,” which stated that in 2014 there were 1.2 million violent crimes committed in America, 63.6% of which were aggravated assaults. In addition, 8.2 million property crimes were reported by law enforcement agencies; victims suffered financial losses of approximately $14.3 billion.

However, not all larceny occurs in “the real world,” per say. In a separate FBI report on Internet crime, cybercrimes accounted for nearly 270,000 documented incidents last year. These illicit activities, which include auto and real estate fraud, government impersonation scams and extortion, resulted in total losses of $800 million.

With the sheer volume of crimes being committed on a daily basis and the severity of the resulting financial damages, clearly more could be done to deter future incidents. Unfortunately, as technology advances, criminals become more sophisticated in their methods, and it becomes even more paramount to remain multiple steps ahead.

read more

Exchange Market Blog Series, Part 1: Kidney Exchanges

This is the first installment in a series of blogs examining exchange markets and how they are a good use case for using a graph database.

Bartering may seem like an ancient practice, but we all grew up exchanging non-monetary goods in a marketplace. For me, it was baseball cards, but for many it was records or clothes. We all intuitively know how to transact in markets when we want or need something, and the supply is limited.

An exchange is a market in which non-monetary goods are traded. When you trade in exchange markets, the matching is typically very rare and the value of the goods is generally very subjective. For example, my “Wade Boggs” rookie card may not be valuable to me but may be very valuable to my cousin’s friend. Networking connections and attracting people who want to transact is key to what keeps exchanges alive. Finding matches to ensure transactions are happening is what attracts users to an exchange.

read more