ssgoe2017
A repository with the material we'll use at the Data Science Summer School Workshop
Further Reading
This is a list with material for further study.
It contains links for:
- Reproducible Science
- Programming
- Scientific Python
- Git and GitHub
- Linux and Linux Command Line
- Containers
- Machine Learning
- General Material
- Books
Reproducible Science
The following links, are focusing on Reproducible Research
Reproducible Science Curriculum
This is the curriculum, on which the workshop was based on.
- Introduction
- Organization
- The main documents are under the _episodes folder.
- Data Exploration
- Automation
- Publication & Sharing
iDigBio, Reproducible Science Workshop
This workshop is also based on the Reproducible Science Curriculum. However, it is using an older version of the program based on R. The “theoretical” topics are still the same though.
- Workshop Website
- Recordings of Sessions:
- Workshop Wiki
Reproducible Research: Walking the Walk
This is a hands-on tutorial, that took place on SciPy 2014. It includes examples of using Docker to create the same environment, and Dexy to automate document creation. All the material is available online, and there is also a recording for the whole duration of the workshop.
- Website
- GitHub repo
- YouTube Recordings: Part 1, Part 2
Reproducible Research – YouTube playlist
A number of presentation regarding Reproducible Research
Programming
An introduction to programming using Python
Programming Exercises:
- Advent of Code: 2015, 2016
- Advent of code is a website that contain programming challenges. You will get an input and then a form to put your output to be checked automatically.
- The cryptopals crypto challenges
- More advanced programming challenges, focusing on security topics
Scientific Python
Also a number of great videos and tutorials, for Python and generally for Data Science, can be found at the Enthough and PyData channels on YouTube:
For example:
Git and GitHub
To learn more about Git/Github you can check the following.
Git for Scientists: A Tutorial
An introduction to Git, structured specifically for scientists.
It uses tryGit, so you can follow along, even if you don’t have Git installed on your system.
Atlassian Git Tutorial
A longer tutorial on git, from Atlassian. The examples use Bitbucket, and not GitHub. However the commands are the same
Udacity Git Course
A free course on Git, from Udacity.
ProGit
The standard textbook to get more familiar with Git and Github
Linux and Linux Command Line
Introduction to Linux, edX course
A complete (free) course that will provide you with a good working knowledge of the Linux system and basic command line operations
Containers
Here are some resources regarding the use of general containers (Docker), or some containers specialized in scientific computing (Singularity)
Docker
Docker is platform based on container virtualization. To try Docker, you can either install it from the Docker Website or use the Play with Docker Website. Play with Docker, provides access to a testing virtual machine with docker installed. The machine is destroyed after a few hours. You can use it to follow the workshops presented here.
Introduction to Docker
A very good workshop from Jérôme Petazzoni, on PyCon2016, showing the basic Docker fuctionalitites.
Advanced Docker Workshop
An advanced workshop, from Jérôme Petazzoni, on PyCon2017
Lecture on Container Virtualization
This is a single lecture, getting into more details on container virtualization
Singularity
Singularity containers are containers specialized for HPC and scientific usecases. They are similar to Docker, but more optimized to this usecase.
Singularity: Scientific containers for mobility of compute
An article explaining what singularity containers are and their basic usage.
Singularity: Containers for Science, Reproducibility, and HPC
A youtube presentation of Singularity containers
Machine Learning
The two following (free) books, are considered to be the best books for Machine Learning – even though “Machine Learning” is not in their title.
General Material
- A Quick Guide to Software Licensing for the Scientist-Programmer
- Choose A Licence
- DataTau: A website with general news regarding Data Science.
- Good Enough Practices in Scientific Computing
- Best Practices in Scientific Computing
- Opening Science
- How to have a bad career in Research / Academia
- You and Your Research
Books
Some general (non-technical) book recommendations:
-
Who Moved My Cheese, by Dr Spencer Johnson: A book that teaches you how to deal with constant changes
-
The Dip, by Seth Godin: Thoughts about when to quit and when to stay.
-
So Good They Can’t Ignore You, by Cal Newport: On why it’s more important to focus on building your skills, and why “Following your passion” is not always a good advice.
-
Deep Work, by Cal Newport: By the same author, recommendations for how to work more focused and why that is important.
-
Thinking, Fast and Slow: A lot of information on how we think and behave.
-
How to Read a Book: Self explanatory title.
-
Incerto, by Nassim Nicholas Taleb: A philosophical and practical essay on uncertainty. It is not a “Data Science” book, but it might be the most critical book that you read, regarding Data Science.
Revised Agenda
Due to the issues with the Jupyter Notebooks, (and the lack of time), we’ll focus more on the other parts of the workshop, that are not so heavily dependent on Jupyter.
16:00 – 16:15
Discussion on the Forensic Exercise (Organization Exercise 01 html)
16:15 – 16:45
Project Organization
16:45 – 17:15
Organization Exercise 02 html
17:15 – 18:15
Publication & Sharing Exercise html
Workshop Agenda
S21 Introduction, 11:30 – 12:30
This would be the introductory session for concept of Reproducible Research
- The slides of the introduction are available here
- To follow allong, during the Jupyter section, you need to follow the instructions from here Workshop Preperation.
- Introduction to Jupyter
- Working with Notebooks
- Now, for the remaining time open (from your Jupyter Browser), the notebook
01_navigation.ipynband read / evaluate all of it’s cells
S22 Organization & Data Exploration 14:00 – 15:30
In this part we’ll talk about how to better organize a project. Your first (practical) task is to follow the instructions from the Organization Exercise 01.
- The slides of the organization lecture can be found here
- Organization Exercise 01 html / ipynb
- Organization Exercise 02 html / ipynb
- Data Exploration html / ipynb
S23 Automation, 16:00 – 17:30
Here, we’ll automate the analysis from the previous step by implementing functions
S24 Publication & Sharing, 17:30 – 18:30
Here we’ll see how to export a notebook, share it, and publish our results.
References
The majority of the material, is based on the Reproducible Science Curriculum, which was in turn based on Data Carpentry material for reproducible research.
Disclaimer
The opinions expressed in this article are the committer’s own and do not necessarily reflect the views of GWDG.