Buzzwords...

Did you learn about them from:

• A sales person
• In a class with examples from others
• A grandiose article
• Un@ culebrer@

Do they look like this?

What you really
need to know

Insights!

What are insights?

• Deep understanding
• Meaningful
• Non obvious
• Actionable
• Based on data

Ingredients

• Data
• Tasks
• Users 🫵🏼

How to extract insights

My insights toolbelt?

What do I use?

Let me show you
with examples

Covid Colombia

Colombian Companies

How did the Colombian Companies
perform during the pandemic?

We got some data

🎁🎁🎁 Navio

Use it to explore your data!

https://navio.dev

But John told me that I should visualize it

Centralized in Bogotá

... and around a handful of companies

Remeber

• Data + Tasks + Users
  • You are the expert 🫵🏼
• Interact with the data
• Aim for interactive information visualizations
• Ask lots of questions

If you have the insights...
tell stories

Presidential Election

Remeber

• If you have the insights, tell a story
• ... but allow for interactions
• ... be careful with assumptions

How can you find who are
the Innovation Influencers

Remember

• Try to open the black box
• If you don't understand it, don't use it
• Common sense beats Machine Learning

Other Insights!

Results Bogota Council

Navio

How do our Senators vote?

UMAP Playground

Trees

Reject
Buzzwords

Or: No coma Machine Learning, follow the insights

Story time

What car should I buy?

Normal procedure

Ask friends and family

Problem

That's inferring statistics from a sample n=1

Better approach

Data based decisions

Cars analysis data

• Scraped
• Asking price
• Few examples for rare cases

Story

A new niece 👶🏼!!!

What are Colombia's most common names?

Colombian National Registry

• Me: Hey, do you have data of Colombians' names?
• CNR: Sure of course!
• Me: Great, can I have all of them
• CNR: Of course, it is just $0.40 per name

A couple months later...

• Brother: What school do you like for your nephew?
• Me: I wonder which one has the best scores? 🤔🤔🤔🤔

What we learned

• No data 👉🏼 No Insights
• Dig deeper
• Ask questions
• The 👺 is in the details

Why should we
visualize?

Anscombe's quartet

Datasaurus!

The purpose of visualization
is insight,
not pictures

Ben Shneiderman

But what are insights?

• Deep understanding
• Meaningful
• Non obvious
• Actionable
• Based on data

Data Science ?

However

Popularity

Data Science is way more than Machine Learning!

The purpose of visualization
is insight,
not pictures

Ben Shneiderman

The purpose of data analysis
is insight,
not (just) models

John Alexis Guerra Gómez

Machine Learning?

What can you use ML for?

• Photos 🖼
• Videos 📹
• Document/Text Processing 📃
• Speech 👄👂🏼
• Structured data 💾?

What can I detect on photos 🖼?

• Objects 🐈 🐕 🍎
• Faces 👱🏽‍♂️👱‍♀️
• Celebrities 🍾
• Landmarks 🗼
• Text in images 🗼

How can I use it?

Develop locally

Pose Detection

Object Detection

How can I use it?

• Big companies 👉 ML as a service
• You pay per request 💵 ~US$1.0 per every 1000 requests

Demos

• Google Cloud Vision
• IBM Watson Visual Recognition
• IBM Watson Visual Recognition
• Microsoft Azure Computer Vision

What can I do with documents 📃?

• OCR 🖼 → 🔤
• Sentiment analysis 😆😡
• Topic extraction 🟡🟠🟣
• Entities detection
• Political Affiliation? 👔🎉
• Psychological Profile?

Demos

• Google Cloud Document AI
• Personality insights

What can I do with Speech 👄👂🏼?

• Speech recognition 👂🏼
• Speech generation 👄

That's hip, but...

The purpose of visualization is insight, not pictures

Ben Shneiderman

The purpose of data analytics is insight, not (just) models

John Alexis Guerra Gómez

Rappi

How is Rappi doing on Twitter?

• 30k tweets in a week of 2019

Approach 1

😡😠😒😐😏😃🥰?

• Machine learning 🎩! ???
• Detects sentiment ! ???

I hired a data 🐒 (might be me)

Analyzed 180 tweets

• 😡😠😒😐😏😃🥰

Here are some of them

Would you hire this data 🐒?

Well, actually

• It wasn't a data 🐒
• It was a 💻
• Would you use it?

Well, actually, actually

• It wasn't just any computer
• It was ML as a service by Google
• It costs $$$

Will you trust it?

• Spanish is just too complicated
• These models are trained with 🇪🇸 examples

Approach 2

Approach 3

It's up to you!

• Interactivity 👉 Ask questions
• Slice and dice
• Overview first, Zoom/Filter, then details on demand

¡No coma Machine Learning, coma 🍖!

Machine Learning

• Prediction vs Training
• How was it trained?
• Garbage in - garbage out

AI ?

Big Data?

You might have heard of the Vs of Big Data

• Volume
• Velocity
• Variety
• and Veracity and Value

Too ambiguous!! 🤦🏽‍♀️ Let's go beyond that

How Big is big?

Can you fit it in one computer?

Yes? 👉🏼 Then, is not really big 🤷🏽‍♀️

Why this criteria?

Big data 👉🏼 Big overhead

Example: photo collection

• One photo 👉🏼 10MB
• 1k photos in a 📱 👉🏼 10MB * 1k = 10000MB = 10GB
• 50k photos in your 💻 👉🏼 10MB * 50k = 500GB

Big Data? 🙅🏽‍♂️

How many blue photos are in my collection?

How do you compute this?

• Put all your photos in one 💻
• Go through all the collection and count the blue ones

Flickr scale

80+ trillion photos (80'''000''000'000.000)

That's big data

How many blue photos are on Flickr?

How do you compute this?

• Distribute the data among 100s of 💻💻💻s. (a cluster)
• Compute subtotals on each data part. (Map)
• Aggregate the subtotals into one big total. (Reduce)

How many computers do you need?

What if one computer breaks? ☢️

Conclusion

Big Data? 👉🏼 Only if it doesn't fit on one 💻

⚠️ Use it only if you must ⚠️

But don't panic!

Let me share a secret

🤫

My wife tells it to me all the time!

Size doesn't really matter

What matters are the insights 👍

Insights ?

Machine Learning?

Big Data Technologies

Technologies

• MapReduce (Hadoop, Hive, pig, Spark ...)
• NoSQL Databases (Redis, Cassandra, MongoDB, Neo4J)
• Distributed Relational (SQL) Databases (MySQL, PostgreSQL, Oracle, SqlServer)
• Many others

Hadoop

• Computing platform for big data
• Uses clusters for storing and processing the data

Hadoop Architecture

Spark

A distributed computing alternative of to map reduce.

• Easier to use
• Integrates better with traditional programming models

NoSQL Databases

• Scalable storage platforms that use techniques different to traditional SQL databases
• Sacrifices features for performance

Types of NoSQL

• Column Oriented: Cassandra, HBase, Redshift ...
• Key-value: Redis, memcached, Aerospike ....
• Document based: MongoDB, CouchDB, DynamoDB ...
• Graph based: Neo4J, Titan, ...

Bonus

Introduction to NoSQL for Web Developers

Distributed Relational DB

• You can also use traditional databases on a distributed way.
• Divides the database into shards.
• Usually doesn't scale that well.

Others

• Google DataFlow
• Google's replacement for MapReduce based on flows.
• Supposed to scale better.
• AFAIK can only be used with Google's Cloud.

Types of Visualization

• Infographics
• Scientific Visualization (sciviz)
• Information Visualization (infovis, datavis)

Infographics

Scientific Visualization

• Inherently spatial
• 2D and 3D

Information Visualization

Visualization Mantra

• Overview first
• Zoom and Filter
• Details on Demand

Visualization Science

Problem Abstraction

What/Why/How

• What is visualized?
  • data abstraction
• Why is the user looking at it?
  • task abstraction
• How is visualized?
  • idiom visual encoding and interaction

Abstract language avoids domain specific pitfals

What/Why/How to navigate systematically the design space

Marks and Channels

Analyze Idiom Structure

Marks

Other Insights

FDA

Task: Change in drug's adverse effects reports

User: FDA Analysts

State of the art

Health insurance claims

Task: Detect fraud networks

User: Undisclosed Analysts

Clustering

Overview

Ego distance

Who to follow on Twitter

Who am I?

PhD

Silicon Valley