Data Analysis

Big Data

• How big is big data? (StackExchange)

Good question; good answer.

• A Complete Tutorial to work on Big Data with Amazon Web Services (AWS)

A step by step process the author used to connect to a 24GB AWS instance via a laptop. Examples for Python and R users.

Business Intelligence

• Quartz/bad-data-guide - The Quartz guide to bad data

An exhaustive reference to problems seen in real-world data along with suggestions on how to resolve them.

BI Tools

• The Business Intelligence Toolkit

A section from the Creative Commons book Getting the Most Out of Information Systems: A Manager's Guide.

• 10+ MySQL Reporting Tools

These MySQL reporting tools fall into two broad camps – business intelligence suites where reporting is a major component, and tools that are specifically aimed at reporting. Also many of them are free.

• 33 Open Source and Free Business Intelligence Solutions

Free Open Source Business Intelligence Solutions

Pentaho Community Edition, OpenText Actuate Information Hub, Free Edition, ReportServer, JasperReports Business Intelligence, Jedox Base, SpagoBI, ART, Pentaho Reporting, JMagallanes, OpenReports, Seal Report, Openi, NextReports, RapidMiner, Mondrian, KNIME.

Free Cloud Business Intelligence Solutions

Watson Analytics, SAP Lumira Cloud, Power BI, Microstrategy Analytics Express and Birst Express for NetSuite.

Free Proprietary Business Intelligence Solutions

EspressReport Lite, SAP Lumira, QlikView Personal Edition, InetSoft, Qlik Sense Desktop, icCube, Tableau Public.

Open Source Commercial Business Intelligence Solutions

Pentaho, Jaspersoft, Palo, Actuate Corporation, TACTIC.

Data Analysis & Data Science Tools

• Datamaps.co - Create and download data maps.

Datamaps.co is a free and simple platform for creating visualizations with data maps. It allows you to upload CSV file with region data, and fully customize your map's appearance. Your map chart can be saved as PNG or SVG. With datamaps.co, you can create a custom map of World, USA, China, Canada and more are coming.

• Data Wrangler (Stanford Visualization Group)

Wrangler allows interactive transformation of messy, real-world data into the data tables analysis tools expect. Export data for use in Excel, R, Tableau, Protovis, ...

• HTSQL—A Database Query Language

HTSQL is designed for data analysts and other accidental programmers who have complex business inquiries to solve and need a productive tool to write and share database queries. HTSQL is free and open source software.

• Jigsaw: Visual Analytics for Exploring and Understanding Document Collections

Jigsaw is a visual analytics system to help analysts and researchers better explore, analyze, and make sense of such document collections.

• OpenRefine (Formerly Refine.)

Open Refine is a standalone open source desktop application for data cleanup and transformation to other formats, the activity known as data wrangling. It is similar to spreadsheet applications (and can work with spreadsheet file formats), however, it behaves more like a database.

Command Line

• Data Science at the Command Line

This hands-on guide demonstrates how the flexibility of the command line can help you become a more efficient and productive data scientist. You'll learn how to combine small, yet powerful, command-line tools to quickly obtain, scrub, explore, and model your data.

• Tutorial – Data Science at Command Line with R & Python (Scikit Learn)

Tutorial inspired by Jeroen Janssens' book Data Science at the Command Line.

IDEs in the Browser | Cloud IDEs

Also See Python in Your Browser: Tools (below).

• Cloud9 - Your development environment, in the cloud.

Cloud9 IDE is an online integrated development environment, published as open source from version 3.0. It supports hundreds of programming languages, including PHP, Ruby, Perl, Python, JavaScript with Node.js, and Go.

• Nitrous.io

Code and Collaborate in the Cloud, for Free! No installation required. Create a new environment with our cloud IDE in seconds.

Further Reading

• Cloud IDEs For Web Developers – Best Of

Compares "13 of the best Cloud IDEs," including the following (to name just a few): Akshell, Cloud9, Python Fiddle

D3

• D3.js - Data-Driven Documents

D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.

Excel

Text Analysis using Excel

• Excel VBA Macro Code to Find Word-Level N-Grams in a Text Entry

FEATURES EXAMPLE WORKBOOK: Analysis of Literature and Medicine Courses.xlsm

The featured workbook looks interesting and appears quite comprehensive.

• Text Analysis using Excel

A quick guide to the types of analysis you can run, ranging from easy to hard.

EXAMPLE: Use this formula to count the number of words in a text string. =LEN(A1)-LEN(SUBSTITUTE(A1," ",""))+1

• The Implementation of Word Clouds with Excel

Approach, algorithm, VBA code and performance optimization of the Word Cloud with Excel implementation.

FEATURES: Split Text Tool (Microsoft Excel 2007-2013, 115K)

NOTE: If the Split Text Tool downloads as a Zip file, change the file extension from .zip to .xlsm.

• Create dynamic Word Clouds / Tag Clouds in Microsoft Excel

FEATURES FOLLOWING DOWNLOADS:

- Word Clouds Bruce Springsteen (Microsoft Excel 2007-2013, 518.4K)

- Word Cloud Generic Template (Microsoft Excel 2007-2013, 78.7K)

- Word Cloud UDF (Microsoft Excel 2007-2013, 72.4K)

NOTE: If any of the tools downloads as a Zip file, change the file extension from .zip to .xlsm.

Python

Python is a very powerful programming language used for many different applications. Over time, the huge community around this open source language has created quite a few tools to efficiently work with Python. In recent years, a number of tools have been built specifically for data science. As a result, analyzing data with Python has never been easier.

Python in Your Browser: Propaganda

• Python FAQ: Why should I use Python 3? (fuzzy notepad)

Python 3 is great and you should use it. Here’s why...

• Running Python in a Browser Is More Awesome Than You Think

Shows simple code examples running on Trinket.

• Python Graphing Tutorials in Trinket

Trinket can be used to post interactive Python code which your students can run in the browser or copy and paste for the purpose of running locally on their computer.

Python in Your Browser: Tools

• PythonAnywhere

Host, run, and code Python in the cloud!

• Python Fiddle: Python Cloud IDE

The Python IDE for the web. Play around with and modify live example code. Share or demonstrate solutions to problems.

• repl.it: online REPL, Compiler & IDE

Powerful and simple online compiler, IDE, interpreter, and REPL. Write, run, save, and share code from your browser!

• Skulpt: Python. Client side.

Skulpt is an entirely in-browser implementation of Python. No preprocessing, plugins, or server-side support required, just write Python and reload.

• Your Python Trinket

Python in the browser. No installation required.

Python & Spreadsheets

• SheetSync: Welcome to SheetSync!

A python library to create, update and delete rows of data in a google spreadsheet.

• XlsxWriter: Creating Excel files with Python and XlsxWriter

XlsxWriter is a Python module for creating Excel XLSX files.

Python Tips, Tutorials & Tricks

• A Complete Tutorial to Learn Data Science with Python from Scratch

Python was originally a general purpose language. Over the years, with strong community support, dedicated libraries for data analysis and predictive modeling emerged.

• Dark Web OSINT With Python and OnionScan: Part One

Step-by-step instructions for setting up an environment to scan hidden services in the dark web!

Web Scraping With Python

• Scrapy

Scrapy is an open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

R

• Any Online R console? (StackExchange)

R Studio

• RStudio: Download RStudio

Take control of your R code. RStudio is the premier integrated development environment for R. It is available in open source and commercial editions and runs on the desktop (Windows, Mac, and Linux) or over the web with RStudio Server. Download RStudio (for Windows, Mac, or Linux).

R & Excel

• A million ways to connect R and Excel

In quantitative finance both R and Excel are the basis tools for any type of analysis. Whenever one has to use Excel in conjunction with R, there are many ways to approach the problem and many solutions. It depends on what you really want to do and the size of the dataset you’re dealing with.

• openxlsx: Read, Write and Edit XLSX Files

Simplifies the creation of Excel .xlsx files by providing a high level interface to writing, styling and editing worksheets. Through the use of Rcpp, read/write times are comparable to the xlsx and XLConnect packages with the added benefit of removing the dependency on Java.

R & Google Sheets

• googlesheets: Manage Google Spreadsheets from R

Interact with Google Sheets from R.

• Google Sheets R API

Access and manage Google spreadsheets from R with googlesheets.

Shiny

• Shiny - by RStudio

A web application framework for R.

- Turn your analyses into interactive web applications.

- No HTML, CSS, or JavaScript knowledge required.

Learn Data Science

Also See: Computer Productivity Hacks#Educate_Yourself

“	Academic credentials are important but not necessary for high-quality data science. The core aptitudes – curiosity, intellectual agility, statistical fluency, research stamina, scientific rigor, skeptical nature – that distinguish the best data scientists are widely distributed throughout the population. We’re likely to see more uncredentialed, inexperienced individuals try their hands at data science, bootstrapping their skills on the open-source ecosystem and using the diversity of modeling tools available. Just as data-science platforms and tools are proliferating through the magic of open source, big data’s data-scientist pool will as well. And there’s yet another trend that will alleviate any talent gap: the democratization of data science. Autodidacts – the self-taught, uncredentialed, data-passionate people – will come to play a significant role in many organizations’ data science initiatives.	”
	— James Kobielus, Data Scientist: Closing the Talent Gap

• Dataquest: Learn data science online.

Learn applied skills for a career in data science. Work with data, build projects, and gain experience. (No longer free.)

• Dataquest.io's Curated List Of Data Science Blogs

Analytics Vidhya | Dataaspirant | Dr. Randal S. Olson | Domino Data Lab's blog | Entrepreneurial Geekiness | no free hunch | Pythonic Perambulations | Rayli.Net | yhat | Beckerfuffle | Carl Shan | Data Mining: Text Mining, Visualization and Social Media | FastML | Garbled Notes | Joy Of Data | MLWave | Trevor Stephens | While My MCMC Gently Samples | Zac Stewart | Machined Learnings | Deep Learning | Nuit Blanche | Machine Learning (Theory) | Large Scale Machine Learning | Revolutions | KDnuggets | Wellecks | PyImageSearch | Data Mining Research | Trey Causey | Deep Learning | Machine Learning and Data Science | i am trask | Jonas Degrave | Yanir Seroussi | Life, Language, Learning | M.E.Driscoll | Models are illuminating and wrong | Christophe Bourguignat | samim | Andrej Karpathy blog | colah's blog | Sebastian Raschka | Edwin Chen | Simply Statistics | Will do stuff for stuff | Tombone's Computer Vision Blog | I’m a bandit | Data Science 101 | R-bloggers | Stitch Fix Tech Blog | DataGenetics | Dataquest Blog

• The Open-Source Data Science Masters

The open-source curriculum for learning Data Science. Foundational in both theory and technologies, the OSDSM breaks down the core competencies necessary to make data useful.

The Internet is Your Oyster: With Coursera, ebooks, Stack Overflow, and GitHub -- all free and open -- how can you afford not to take advantage of an open source education?

• Quizlet—Study Everywhere! | Subject: Data Science

Numerous "flash card"-style data science lessons.

Random Text, Number & String Generators

Tools

• GenerateData.com

Free, GNU-licensed, random custom data generator for testing software.

• RANDOM.ORG - True Random Number Service

Generate random lists, numbers, text strings, sequences and more.

Tutorials

• How to generate random character strings in a range in Excel?

Sometimes you may need to generate random strings, such as different passwords. This article tries to show you some tricks to generate different random strings in Excel.

Regular Expressions

• Regular expression (Wikipedia)

In theoretical computer science and formal language theory, a regular expression (abbreviated regex or regexp and sometimes called a rational expression) is a sequence of characters that forms a search pattern, mainly for use in pattern matching with strings, or string matching, i.e. "find and replace"-like operations.

Quick Tips

• Regular Expression to match cross platform newline characters

• What is a cross platform regex for removal of line breaks?

"The regex I use when I want to be precise is \r\n?|\n."

RegEx Tutorials

• RegexOne - Learn regular expressions with simple, interactive examples.

Includes:

- Interactive Tutorial › Learn to use Regular Expressions

- Practical Examples › Practice your Regular Expressions

- RegEx Cheatsheet › Regular Expressions in PHP & More

• Regular Expressions Tutorial: Learn How to Use and Get The Most out of Regular Expressions

Any non-trivial regex looks daunting to anybody not familiar with them. But with just a bit of experience, you will soon be able to craft your own regular expressions like you have never done anything else.

• The Many Uses of Regex

Regex is the gift that keeps giving. Once you learn it, you discover it comes in handy in many places where you hadn't planned to use it.

• Learn Regex The Hard Way

This is an in-progress book that quickly teaches you regular expressions.

RegEx Tools

• Regex101: Online regex tester and debugger: JavaScript, Python, PHP, and PCRE

Regex101 allows you to create, debug, test and have your expressions explained for PHP, PCRE, JavaScript and Python. The website also features a community where you can share useful expressions.

• RegExr: Learn, Build, & Test RegEx

Regular expression tester with syntax highlighting, contextual help, video tutorial, reference, and searchable community patterns.

• Pythex: a Python regular expression editor

Pythex is a real-time regular expression editor for Python, a quick way to test your regular expressions.

• Regex Tester

JavaScript regex tester. Highlights matches on the fly.

Small Data

Small Data is everything Big Data is not.

You Might Not Be Big Data If

- You were generated through human data entry. (Big Data came about in order to handle the exponential growth of machine-generated data, because we humans aren’t fast enough to outpace a good old relational database).
- You are an operational database. For instance, CRM is never Big Data, and ERP is never Big Data.
- You fit just fine in a MySQL database. Even if you have to put a lot of RAM in it, it’s still not Big Data.

• Forget Big Data–Small Data Is Where the Money Lies

For the majority of small and medium businesses, Big Data is the technology of the future, not the reality they experience today. To be blunt, most SMBs don’t even have a handle on the Small Data they’re already creating and collecting themselves. (And if many enterprise organizations are being honest, neither do they.) According to Forrester Research, most companies are analyzing a mere 12% of their existing data. That leaves a whopping 88% of data that businesses are flat out ignoring. Can you imagine the potential of actually leveraging that existing data to derive data-driven business insights? Instead of chasing the Big Data dream, SMBs should consider picking up the dollars that are effectively lying on the floor, and invest first in leveraging their Small Data.

• Forget big data, small data is the real revolution

Just as we now find it ludicrous to talk of "big software" – as if size in itself were a measure of value – we should, and will one day, find it equally odd to talk of "big data". Size in itself doesn't matter – what matters is having the data, of whatever size, that helps us solve a problem or address the question we have.

• Forget Big Data -- Small Data Is Driving The Internet Of Things

What is small data, you ask? Small data is a dataset that contains very specific attributes. Small data is used to determine current states and conditions or may be generated by analyzing larger data sets.

SQL

Cheat Sheets

• SQL Server Management Studio Query Window Keyboard Shortcuts

Simple-Talk's free wallchart of the most important SSMS keyboard shortcuts aims to help find all those curiously forgettable key combinations within SQL Server Management Studio that unlock the hidden magic that is available for editing and executing queries.

Fiddle with SQL in the Browser

• codingground: EXECUTE SQL ONLINE

I got this one to work. | Dave (talk) 18:01, 7 December 2016 (PST)

• Rextester | compile postgresql online

At Rextester you can not only compile PostgreSQL, but also several several other SQL flavors, too (SQL Server, MySQL, Oracle). Rextester also offers options for several other programming languages.

SQL & PowerShell

• Execute SQL Query with PowerShell

Scripting is very powerful. And for me, one of the best scripting languages is PowerShell (PoSH). Yes, PoSH takes a bit of getting used to, but once you pass the initial learning curve, you end up with a powerful tool in your hands.

• How do you run a SQL Server query from PowerShell?

Multiple examples; approaches.

• Practical PowerShell For SQL Server Developers and DBAs

Download the latest version of this PowerShell™ wallchart and read the accompanying in-depth article from Simple-Talk.

Also See:

• SQL & Excel

Text Extraction

PDF Conversion

Capture2Text conceptual illustration.

• Capture2Text

Capture2Text enables users to quickly OCR a portion of the screen using a keyboard shortcut. The resulting text will be saved to the clipboard by default.

• Detexter

Detexter is an app designed to extract text from PDF files.

• PDF Online - Convert PDF to Word (Free!) (A unit of BCL Technologies.)

I used this service to successfully convert the .US Locality Domain Name Registration Terms and Conditions form from PDF to Word format.

• PDF TEXT Extraction

Lists several options.

• pdf2htmlEX

Convert PDF to HTML without losing text or format.

• pdftotext.org (Now at pdftotext.github.io.)

Note: (Domain squatter has apparently hijacked the original pdftotext.org domain. But the service is still here: http://pdftotext.github.io/)

pdftotext.org is the best online service for easily extracting text from your PDF files. Conversion from PDF to TXT is really fast thanks to our in-browser conversion architecture. Your PDF files are never uploaded to the Internet, so even private PDF files are safe to convert with this service. The conversion is done locally in your browser – you can even convert when you are offline! There is no need for any registration or sign-up, and the service will always be free to use.

• SILVERCODERS DocToText

Extracts plain text from documents in all popular formats.

• Tabula

Tabula is a tool for liberating data tables locked inside PDF files. Tabula allows you to extract that data into a CSV or Microsoft Excel spreadsheet using a simple, easy-to-use interface. Tabula works on Mac, Windows and Linux.

Also See

Office Productivity Hacks#Portable Document Format (PDF)

Data Scrape ; Web Scrape

• Almost Scraping: Web Scraping for Non-Programmers

Tools and tips compiled by journalists from PBS and Omaha World-Herald.

• Open Source at Scrapinghub

Scrapinghub's list of open source scraping projects.

• Sitestalker

Monitor website links with ease. Sitestalker supervises websites and notifies you when your desired content hits the web.Stop wasting your time constantly refreshing websites.

Sitestalker is great for:

Finding jobs

Searching for an apartment

Getting the best bargains

Clipping

• Six tools for web scraping – To use for data journalism & creating insightful content

Tools for gathering data from public sources.

• 10 Web Scraping Tools to Extract Online Data

Web Scraping tools are specifically developed for extracting information from websites. They are also known as web harvesting tools or web data extraction tools. These tools are useful for anyone trying to collect some form of data from the Internet. Web Scraping is the new data entry technique that don’t require repetitive typing or copy-pasting.

R Scraper

• Web scraping with R and rvest (includes video & code)

Watch (video) how easy it is to import data from a Web page into R.

Text Search

• geekDadaji - A SEARCH INITIATIVE

Makes tools to search text content, including:

1. FALCON - Text Search Java Project: JSON based text search Java Project
2. HAWK - PDF Text Search Java Project: Taking initiative for Document Text Search

• Xpdf: A PDF Viewer for X

Xpdf is an open source viewer for Portable Document Format (PDF) files.

Windows installer: Short Programs/Scripts (Look for the xpdf3.exe / poppler.exe links in left sidebar.)

Text Transformation

• Transformy - Change your data with one example.

Do you have a list of text strings that you want to modify the format on? Copy and paste the list into the a box, then provide an example of how you want each text string formatted. The hope is that Transformy will do the rest.

Workflow

Git, GitHub

• What's the best practice for putting multiple projects in a git repository? (Stack Overflow)

Solution 1: A single repository can contain multiple independent branches, called orphan branches. Orphan branches are completely separate from each other; they do not share histories.

Solution 2: Avoid all the hassle of orphan branches. Create two independent repositories, and push them to the same remote. Just use different branch names for each repo.

#git, #github

See Also

• Career Portfolio Tips
• Computer Productivity Hacks
• Mobile Apps
• Office Productivity Hacks
• Social Networking Tips
• Spreadsheet Tricks
• .US Locality Domains
• User:Dave/Admin Notes
• User:Dave/CV Sandbox
• User:Dave/Web Hack Notes (Non-Wiki)
• User:Dave/Wikis In The Enterprise