Julia is a high-level, high-performancedynamic programming language, focusing on numerical computing and general programming. It is relatively new—the four creators, Jeff Bezanson, Stefan Karpinski, Viral Shah, and Alan Edelman, set out to create it in 2009, with the first public reference to the language in 2012, when they published a blog post explaining their vision and their goals. 2012 is considered the official birth year of Julia, making it only six years old. Since its initial public release, Julia has received code contributions from hundreds of scientists, programmers, and engineers across the world. It is developed in the open, with the source code available on GitHub, and is one of the most popular repositories with almost 20,000 stars (at the time of writing, and counting). Julia v1.0, the much anticipated first stable release, came in August 2018 during the Julia conference in London, as the brilliant outcome of the collaboration between over 700 open source contributors and thousands of package creators and early users. By that time, the language had been downloaded over two million times already!
Julia came out as a fresh alternative to traditional scientific computing languages, which were either productive or fast, but not both. This is known as the two language problem, where the initial prototyping code is written in a dynamic, highly productive language (such as R or Python), which allows exploratory coding and quick iterations, skipping taxing build and compile times. But later on, the developers would be forced to rewrite their programs (or at least the performance critical parts of their programs), using a compiled language that would satisfy the high-performance requirements of scientific computing.
The creators of Julia thought that software development technology has evolved enough that it can support a language that combines both high productivity and high performance. This was their manifesto, underlying their goals for Julia:
"We want a language that's open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that's homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like MATLAB. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as MATLAB, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.""(Did we mention it should be as fast as C?)"
As incredible as it may seem, Julia has managed to satisfy all these demands, making for a unique language that is easy to learn, intuitive, friendly, productive, and fast. Let's take a closer look at all these features.
The topics we will cover in this chapter are:
- A quick look at Julia—what is it, the main features and strengths, and why it could be the best choice for your next project
- How to set up and interact with the Julia language on your local machine
- The best IDEs and editors for productive Julia development
- Getting starting with Julia by learning about its powerful REPL
- How to use the built-in package manager,
Pkg, to extend the language with third-party libraries
The Julia package ecosystem is under continuous development and new package versions are released on a daily basis. Most of the times this is great news, as new releases bring new features and bug fixes. However, since many of the packages are still in beta (version 0.x), any new release can introduce breaking changes. As a result, the code presented in the book can stop working. In order to ensure that your code will produce the same results as described in the book, it is recommended to use the same package versions. Here are the external packages used in this chapter and their specific versions:
In order to install a specific version of a package you need to run:
pkg> add [email protected]
pkg> add [email protected]
Alternatively you can install all the used packages by downloading the
Project.toml file provided with the chapter and using
pkg> instantiate as follows:
julia> download("https://raw.githubusercontent.com/PacktPublishing/Julia-Programming-Projects/master/Chapter01/Project.toml", "Project.toml") pkg> activate . pkg> instantiate
In a nutshell, Julia truly is a new breed of programming language that successfully manages to combine the high performance of compiled languages with the agility of the dynamic ones, through a friendly syntax that feels natural and intuitive right from the start. Julia is fast (programs are compiled at runtime to efficient native code for multiple platforms), general (the standard library supports, out of the box, powerful programming tasks including asynchronous I/O, process control, parallel, and distributed computing, logging, profiling, package management, and more), dynamic and optionally typed (it is dynamically-typed with optional type declarations and comes with a powerful read-eval-print loop (REPL) for interactive and exploratory coding). It is also technical (excelling at numerical computing) and composable (thanks to its rich ecosystem of packages that are designed to work together seamlessly and with high performance).
Although initially it focused on addressing the needs of high-performance numerical analysis and computational science, recent releases have positioned the language in the area of general computing, with many classes of specialized functions being moved out of the core into dedicated modules. As such, it is also a great fit for client and server-side programming, due to its powerful capabilities for concurrent, parallel, and distributed computing.
Julia implements a type system based on parametric polymorphism and multiple dispatch, it is garbage-collected, uses eager evaluation, packs a powerful regular expression engine, and can call C and Fortran functions without glue code.
Let's take a look at the most important features of the language, the parts that make Julia stand out. If you're considering Julia for your next project, you can use this as a quick checklist against your requirements.
The key to Julia's performance is the combination between the LLVM-based just-in-time (JIT) compiler and a series of strategic design decisions that allow the compiler to generate code that approaches, and most of the times matches, the performance of C.
quicksort and a few others. They are designed to evaluate compiler performance against common code patterns such as function calls, string parsing, sorting, iterations, recursion, and more. There is a plot of the benchmarks, available at https://julialang.org/benchmarks/, which illustrates Julia's consistent top performance across all of the tests. The following plot depicts this:
The creators of Julia have carefully picked the most successful elements of syntax from other languages, with the goal of producing expressive, concise, and readable code. Julia provides powerful and expressive language constructs for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python do. It builds upon the experience brought by existing mathematical programming languages but also borrows much from popular dynamic ones, such as Lisp, Perl, Python, Lua, and Ruby.
To give you a quick taste of idiomatic Julia, here's how to open a file, read it, output it, and then have the file automatically closed by Julia:
open(".viminfo") do io read(io, String) |> println end
In the preceding snippet, we open the
.viminfo file for reading passing
IOStream instance, into the underlying code block. The stream is then read into a
String that is finally displayed onto the console by piping it into the
println function. The code is very readable and easy to understand if you have some coding experience, even if this is your first time looking at Julia code.
do syntax (named after the
do part following the
open function) is inspired by Ruby's blocks—and it is, in fact, syntactic sugar for passing anonymous functions as method arguments. It is efficiently used in the preceding example to succinctly express a powerful design pattern for safely handling files, guaranteeing that the resources are not accidentally left open.
This goes to show the amount of attention that was put by the designers of the language to make Julia safe, beginner-friendly, expressive, concise, readable, and intuitive.
Julia's type system is a key feature of the language and one that has a major impact on both its performance and productivity. The type system is dynamic and optional, meaning that the developer can, but is not required to, provide type information to the compiler. If not provided, Julia will perform type inference, which is the process of deducing the types of later values from the types of input values. This is a very powerful technique, as it frees the programmers from having to worry about types, allowing them to focus on the application logic and making for a gentler learning curve. This is especially useful for prototyping and exploratory programming, when the complete set of constraints and requirements is not known beforehand.
However, understanding and correctly using the type system offers important performance benefits. Julia allows optionally adding type information, making it possible to indicate that a certain value must be of a specific kind. This is one of the cornerstones of the language, allowing performant method dispatching and facilitating the automatic generation of efficient, specialized code for different argument types. The type system allows the definition of rich type hierarchies, with user-defined types as fast and compact as the built-in ones.
If the languages of the 70s and 80s were designed under the strict requirements imposed by the limited CPU and RAM resources, the ones in the 90s and the 2000s had the optimistic outlook that these resources are forever expanding. However, the last decade had seen something of a stagnation in this regard, with a shift toward multi-CPU, multi-core, and distributed computing. In this regard, Julia's inception only 6 years ago gave it an edge compared to older languages, putting parallel and distributed computing at its center as one of its most important features.
One of the most serious barriers in the adoption of a new language is that it takes time for the ecosystem to catch up—and in the beginning, it cannot offer libraries of the quality and richness of the already established languages. This is less of an issue now, when Julia benefits from a large, enthusiastic and continuously growing developer community. But being able to seamlessly communicate with other languages is a very efficient way to enrich existing functionality and to effortlessly supplement any missing features.
Julia has the ability to directly call C and Fortran functions (that is, without glue code)—especially important for scientific computing, where these languages have a strong presence and a long history.
Optional packages extend this capability by adding support for calling functions written in other languages, most notably Python, via
PyCall. And there are others, supporting interaction with Java, C++, MATLAB, Rust, and more.
The REPL represents a language shell, an interactive computer programming environment at the command line. Julia has an excellent REPL, supporting sophisticated code inputting and evaluation. It includes powerful editing features such as searchable history, tab-completion, and syntax highlighting, to name just a few.
It also comes with three special modes—shell, which allows executing commands as if at the OS Terminal; help, which provides access to documentation without leaving the REPL; and pkg, used for installing and managing application dependencies.
Julia has superb cross-platform support, running on all major operating systems. The install process is straightforward—the language can be set up on your local machine, in a virtual machine (VM), in a Docker container, or on a server somewhere in the cloud.
Let's start by looking at local installation options for the big three operating systems (Windows, Linux, and macOS). Feel free to skip directly to the right one for you.
The simplest option is to download the Windows installer corresponding to your platform (32 or 64-bit), from https://julialang.org/downloads/. Get the
.exe and run it. Follow the standard installation procedure and in the end, you will have Julia installed as a program. Double-clicking
julia.exe opens a command prompt with the Julia REPL, just like the one illustrated here:
Chocolatey is a package manager for Windows, similar to
yum on Linux, or
brew on Mac. If you don't have it, get it by following the instructions at https://chocolatey.org.
$ choco search julia Chocolatey v0.10.11 Julia 1.0.0 [Approved] 1 packages found.
Installing is as simple as this:
$ choco install julia Chocolatey v0.10.11 Installing the following packages: julia By installing you accept licenses for the packages. Progress: Downloading Julia 1.0.0... 100% Julia v1.0.0 [Approved] Chocolatey installed 1/1 packages.
One of the recent additions to Windows 10 is the Subsystem for Linux. This allows setting up a Linux development environment, including most command-line tools, utilities, and applications—directly on Windows, unmodified, and without the overhead of running a VM.
In order to be able to use the Linux Subsystem your PC must be running the 64-bit version of Windows 10 Anniversary Update or later (build 1607+). It also needs to be enabled first—so open a PowerShell as an administrator and run the following:
$ Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
Once the subsystem is enabled (computer restart might be required) you can choose one of the Linux versions available, directly from the Windows Store. At the time of writing, five versions were available—Ubuntu, openSUSE , SLES, Debian, and Kali.
Ubuntu is the default option for Windows 10 and has the best user ratings in the Windows Store, so let's go with that. It can be installed from https://www.microsoft.com/en-us/store/p/ubuntu/9nblggh4msv6. Alternatively, you can just open a command prompt and type
$ bash. This will trigger the installation of the Ubuntu Linux Subsystem.
Once you find yourself at the shell prompt of your Linux subsystem, you can proceed and issue the commands for installing Julia. For Ubuntu you need to run the following:
$ sudo apt-get install julia
Make sure to confirm the required selections—then after a couple of minutes you should have Julia up and running.
Visit https://julialang.org/downloads/ and look for the macOS package (
.dmg). Once it's downloaded, double-click the
.dmg file and drag and drop the Julia app into the
/Applications folder. Now you can simply open the Julia app—which in turn will launch a new Terminal session, loading the Julia environment, as follows:
Homebrew is a well-known package manager for macOS in the line of
yum on Linux. It's not really necessary for installing Julia, but it's worth setting it up as it can be very useful during development since it can seamlessly install database servers, libraries, and other components for your projects.
As per the instructions at https://brew.sh, it can be installed by running the following command in a Terminal window:
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew cask install julia will download and install the latest version of Julia. In the process, it will also link the
julia binary to
/usr/local/bin/julia so you can interact with the language from the command line by simply typing
As soon as you get the confirmation that the installation was successful you can run
$ julia to start a new REPL session:
Julia is already available in the software repositories of the major Linux distributions, but unfortunately, these are not up to date. For example, at the time of writing, Ubuntu was providing v0.4.5 and Debian v0.4.7. The best approach is to use the generic Linux binaries provided on Julia's download page, at https://julialang.org/downloads/.
Please follow the instructions corresponding to your Linux distribution, as indicated at https://julialang.org/downloads/platform.html#generic-binaries.
Docker is a software technology that provides an additional layer of abstraction of operating-system-level virtualization. In plain English, Docker sets up containers that behave like VMs, but without the added overhead of starting and maintaining VMs. You can run Docker on all the major operating systems.
Docker is widely used as a development and deployment strategy, so many technologies are readily available in the form of Docker images, and Julia is no exception.
Start by installing Docker for your platform. The official Julia container can be found in the Docker store at https://store.docker.com/images/julia. Go get it.
If you need help setting up Docker or installing containers, follow the instructions at https://www.docker.com.
At the command prompt, type
$ docker pull julia. Once the Julia image is configured by Docker, run it with
$ docker exec -it --rm julia. This will start the container and load a new Julia REPL:
Julia Computing, the company behind the Julia programming language, offers a batteries included distribution. It's called JuliaPro and it's arguably the easiest way to get started with Julia straight away. It includes the compiler, a profiler, the Juno IDE, and over 160 top quality curated packages for plotting, data visualization, machine learning, databases, and more.
JuliaPro can be downloaded for free at https://shop.juliacomputing.com/Products/ (registration required). Once you get it, follow the install process specific to your platform. When done you'll have everything needed to begin using Julia productively.
Finally, there's also JuliaBox (https://www.juliabox.com), another free offering from Julia Computing. JuliaBox allows running a Julia Docker container on the fly, in their cloud. It provides access to IJulia Jupyter notebooks (https://github.com/JuliaLang/IJulia.jl), file sync with Google Drive, importing GitHub repositories, and many other features.
If you are not familiar with Jupyter notebooks, you can learn more about them by visiting http://jupyter.org.
An IDE is very important when working with a programming language. A powerful source code editor, code completion, and a good linter and debugger can significantly influence the learning curve and the productivity of using a language. You will be happy to learn that there are some very good IDE and editor options for Julia—and chances are you'll find your favorite one among these.
The IDE choices reflect the pragmatism of the language as a whole. From choosing LLVM as the compiler to providing efficient ways for calling functions from other languages, to using
git and GitHub to power the package manager, the Julia core team takes a don't reinvent the wheel approach. Following the same line of thinking, the Julia community has built powerful IDEs upon existing industry established editors, such as Atom and Visual Studio Code.
Juno (http://junolab.org) is the most advanced Julia IDE and the de facto editor of choice for Julia professionals. It is based on the Atom editor and it can be considered the official development tool, being also distributed with the previously mentioned JuliaPro distribution.
To get it, either download and install JuliaPro from https://juliacomputing.com/products/juliapro.htmlor do a manual install of Atom and the required plugins.
If you choose the manual install, first you need to download Atom from https://atom.io. Once it's up and running, go to the
Settings pane (you can use the shortcut Ctrl/cmd and ,) and then go to the
Install panel. Type
uber-juno into the search box and press Enter. Next, click the
install button on the package with the same name. Atom will pick it up from here, installing all the required Atom and Julia packages.
Once configured, the IDE options will be available in Atom's menu, under
Julia. Various panes can also be enabled from here, to list variables, visualize plots, or search the documentation.
For further information, check out http://junolab.org and https://github.com/JunoLab/uber-juno/blob/master/setup.md.
Visual Studio Code is a cross-platform extendable editor from Microsoft. It is available for all the big three platforms at https://code.visualstudio.com. Once installed, run it and from the menu click
Extensions or use the shortcut Shift and Ctrl/cmd and X. Search for
julia and install the Julia extension from julialang.
The Julia support in Visual Studio Code is not (yet) as powerful as Juno, but if you prefer it, it makes for a great coding experience, providing syntax highlighting, code completion, hover help, evaluation of Julia code, linting, code navigation, and more. Visual Studio Code is also snappier and uses fewer resources than Atom, which makes it an appealing option when running on a less powerful workstation (although Atom has greatly improved in this regard with recent versions).
The extension might need a bit of help figuring out where it can find the Julia binary. If that is the case, you'll get an informative error message, asking you to set the
julia.executablePath configuration option. This should point to the julia binary, and depends on your operating system and the way you installed Julia (see the previous section for details on the installation).
To set the configuration, go to
Settings (Ctrl/cmd and ,) and in the right pane, the one used to overwrite the defaults, add the following:
We already mentioned JuliaBox (https://www.juliabox.com) in the previous section—it allows creating, editing, and running IJulia Jupyter notebooks in the cloud. IJulia can also be installed on the local development machine.
IJulia is a Julia language backend for the Jupyter interactive environment (also used by IPython). It allows us to interact with the Julia language using Jupyter/IPython's powerful graphical notebook, which combines code, formatted text, math, and multimedia in a single document.
Although IJulia/Jupyter is not really an IDE, nor a classical editor, it is a powerful environment for editing and executing Julia scripts, and it's especially popular for data science and scientific computing. Let's take a few moments to set it up.
Start a new Julia REPL and execute the following:
julia> using Pkg julia> Pkg.add("IJulia")
This will install the
IJulia package, while also adding a required minimal Python and Jupyter distribution called Miniconda. This Python distribution is private to Julia (not in your
PATH). Once finished, continue by executing the following:
julia> using IJulia julia> notebook()
This will open the home page of your local Jupyter install in your default browser, at http://localhost:8888/tree. From the toolbar choose
New > Julia 1.0.0 (or whatever version you are currently running) to create a new notebook. You can now create rich documents using embedded executable Julia code.
There's another way of running IJulia as a desktop app, through
Interact. You can download it and give it a try at https://nteract.io/desktop.
If you're new to Jupyter, it's worth learning more about it. Go check it out at http://jupyter.org.
You can also find IJulia notebooks for each chapter in this book in the chapter's support file repository. The notebooks will allow you to go through the code we're writing, step by step. For instance, you can find the code for this chapter at https://github.com/PacktPublishing/Julia-Programming-Projects/blob/master/Chapter01/Chapter%201.ipynb. You can download it on your computer and open it with the local IJulia installation, or upload it to JuliaBox through their Google Drive integration.
vim enthusiasts, there's also
If you prefer Emacs, you'll be pleased to know that Julia supports it as well https://github.com/JuliaEditorSupport/julia-emacs.
If you'd rater go with one of the IDEs provided by JetBrains (like IntelliJ IDEA), you'll be happy to hear that a plugin is available, at https://plugins.jetbrains.com/plugin/10413-julia
Finally, there is also support for Sublime Text, available at https://github.com/JuliaEditorSupport/Julia-sublime. The plugin provides a good Julia editing experience, supporting syntax highlighting, code completion, and jumping to definition, among other things.
If you followed through the first part of the chapter, by now you should have a fully functional local Julia installation, the knowledge to start a Julia REPL session, and have your preferred IDE ready for coding. If that is not the case, please refer to the previous sections. From this point on we're getting down to business—it's time to write some Julia code!
The first thing we need to understand is how to use the powerful REPL. As a Julia developer, you'll spend a significant amount of time doing exploratory programming, interacting with the shell and the filesystem, and managing packages. The REPL will be your trusted sidekick. It's worth getting to know it well, it will save you a lot of time down the line.
The acronym REPL stands for read-eval-print loop. Simply put, it's a language-specific shell, an interactive coding environment that allows inputting expressions, evaluates them, and outputs the result.
REPLs are very useful as they provide a simple way to interact with the language, to try out ideas and prototype, facilitating exploratory programming and debugging. It is especially powerful in the context of data analysis, where one can quickly connect to a data source, load a data sample and then slice and dice, rapidly testing different hypothesis.
Julia provides an excellent REPL experience, with rich functionality that covers quick evaluation of Julia statements, searchable history, tab-completion, syntax highlighting, dedicated help and shell modes, to name just a few.
If you do not have a working Julia installation, please see the Installing Julia section.
You will be greeted with a screen like this one (the Julia version might be different than mine):
Now Julia is waiting for us to input our code, evaluating it line by line. You can confirm that by checking the Terminal prompt, which says
julia>. This is called the julian mode. Let's take it for a spin.
You can follow through the IJulia Jupyter notebook provided with this chapter's support files. If you are not familiar with Jupyter and don't know how to run it locally, you can use Juliabox (juliabox.com). All you have to do is create an account, log in, and then load the notebook from https://github.com/PacktPublishing/Julia-Programming-Projects/blob/master/Chapter01/Chapter%201.ipynb.
Input the following lines, pressing Enter after each one:
julia> 2+2 julia> 2^3
So we can use Julia like a simple calculator. Not very useful, but this is only the beginning and illustrates how powerful this rapid input and feedback cycle can be when we deal with complex computations.
julia> println("Welcome to Julia")
Under each line, you should see the output generated by each expression. Your window should now look like this.
julia> 2+2 4 julia> 2^3 8 julia> println("Welcome to Julia") Welcome to Julia
Let's try some more. The REPL interprets one line at a time, but everything is evaluated in a common scope. This means that we can define variables and refer to them later on, as follows:
julia> greeting = "Hello" "Hello"
This looks great! Let's use the
greeting variable with
julia> println(greting) ERROR: UndefVarError: greting not defined
Oops! A little typo there, and the REPL promptly returned an error. It's not
greeting. This also tells us that Julia does not allow using variables without properly initializing them. It just looked for the
greting variable, unsuccessfully—and it threw an undefined variable error. Let's try that again, this time more carefully:
julia> println(greeting) Hello
OK, that's much better! We can see the output: the
Hello value we stored in the
The REPL provides a few helping features, specific to this interactive environment (they won't be available when executing a Julia script). One of these is the
ans variable, automatically set up and updated by Julia.
If you type
julia> 2^3—unsurprisingly, you'll get
8. Now input
julia> ans—you'll get
8 again! What's up with that?
ans is a special variable that exists only in the REPL and that automatically stores the last returned value. It can prove very useful when working with the REPL, but more importantly, you need to be aware of its existence so that you don't accidentally declare a variable with the same name. Otherwise, you'll run into some very hard to understand bugs with your variable's value constantly overwritten.
The REPL comes with a very powerful feature called prompt pasting. This allows us to copy-paste-execute Julia code and snippets that include both the
julia> prompt and the output of the expression. It activates when pasting text that starts with
julia>. In that case, only expressions starting with
julia> are parsed, and all the others are ignored. This makes it possible to paste a chunk of code that has been copied from another REPL session or from the documentation, without having to scrub away prompts and outputs.
To see this in action, copy and paste the following snippet, as is:
julia> using Dates julia> Dates.now() 2018-09-02T21:13:03.122 julia> ans 2018-09-02T21:13:03.122
If all goes well, both expressions should output your current time, and not the one from the snippet, effectively replacing the results in the snippet with the results in your Julia session.
julia> pri[TAB] primitive type print print_shortest print_with_color println printstyled
It can also be used to substitute LaTeX math symbols with their Unicode equivalents. To do this, type a backslash as the first character, then the first few characters of the symbol, then Tab. This will complete the name of the symbol or will display a list of options if there's more than one matching name. Pressing Tab again on the complete name of the symbol will perform the replacement:
julia> \pi[TAB] julia> π π = 3.1415926535897... julia> \om[TAB] \omega \ominus julia> \ome[TAB] julia> \omega[TAB] julia> ω
Julia does not have the concept of null so you can't really deallocate a variable from memory. If, however, you need to free an expensive resource referenced by a variable, you can replace its value with something like
0 and the previous value will be automatically garbage collected. You can even invoke the garbage collector yourself straight away by calling
The Julia REPL comes with four operational modes—and additional ones can be defined as needed. The currently active mode is indicated by its prompt. In the previous examples we've used the julian mode
julia>, which evaluates the inputted expression. The other three available modes are help,
shell>, and package management,
The active mode can be switched by typing a specific character right at the beginning of the line. The prompt will change in response, to indicate the current mode. The mode will stay active until the current line is evaluated, automatically switching back to julian (with the exception of the
pkg> mode which is sticky—that is, it stays active until explicitly exited by typing backspace at the beginning of the line). The alternative modes can be exited without evaluating the expression by deleting everything on the line until the prompt changes back to
julia>, or by pressing Ctrl + C.
The help mode provides access to documentation without having to get out of the REPL. To access it, simply type
? at the beginning of the line. You should see the
help?> prompt. Now you can input text, and Julia will search the documentation for matching entries, as follows:
julia> ? help?> println search: println printstyled print_with_color print print_shortest sprint isprint println([io::IO], xs...) Print (using print) xs followed by a newline. If io is not supplied, prints to stdout. Examples ≡≡≡≡≡≡≡≡≡≡ julia> println("Hello, world") Hello, world julia> io = IOBuffer(); julia> println(io, "Hello, world") julia> String(take!(io)) "Hello, world\n"
In IJulia, the additional modes are activated by prefixing the input with the desired mode activator. For instance, to access the help for the previous
println function, we need to input
The output supports rich formatting, via Markdown:
julia> using Profile help?> Profile.print
Resulting a rich output as in the following screenshot:
The shell mode is used to switch to a command-line interface similar to the system shell, for directly executing OS commands. To enter it, input a semicolon
; at the very beginning of the julian prompt:
; the prompt changes (in place) to
To enter shell mode in IJulia and execute a shell command, prefix the command with
;, for example
Now we can execute system-wide commands directly, without the need to wrap them in Julia code. This will list the last ten lines of your
repl_history.jl file. This file is used by Julia to keep a history of the commands executed in the REPL, so your output will be different from mine:
julia> using REPL shell> tail -n 10 ~/.julia/logs/repl_history.jl IO # time: 2018-09-02 21:56:47 CEST # mode: julia REPL.find_hist_file() # time: 2018-09-02 21:58:47 CEST # mode: shell tail -n 10 ~/.julia/logs/repl_history.jl
While in REPL mode we can access Julia's API, making this a very powerful combo. For example, in order to programmatically get the path to the REPL history file, we can use the
REPL.find_hist_file() function, as follows:
julia> REPL.find_hist_file() "/Users/adrian/.julia/logs/repl_history.jl"
The path to the file will be different for you.
shell> tail -n 10 $(REPL.find_hist_file()) REPL.find_hist_file() # time: 2018-09-02 21:58:47 CEST # mode: shell tail -n 10 ~/.julia/logs/repl_history.jl # time: 2018-09-02 22:00:03 CEST # mode: shell tail -n 10 $(REPL.find_hist_file())
Similarly to the help mode, the shell mode can be exited without executing any command by pressing backspace at the beginning of the line or typing Ctrl + C.
In IJulia, the command can be executed by prefixing the input with
;, like this:
;tail -n 10 ~/.julia/logs/repl_history.jl
Press the Ctrl key and the R key at the same time in order to initiate a reverse incremental search. The prompt will change to
(reverse-i-search). Start typing your query and the most recent result will show. To find older results, type Ctrl + R again.
The counterpart of Ctrl + R is Ctrl + S, initiating an incremental search. The two may be used in conjunction to move through the previous or next matching results, respectively.
If you want to automatically execute some code every time you run Julia, you can add it to a special file called
startup.jl. This file is not automatically created, so you'll have to add it yourself to your Julia configuration directory. Any code you add to it will be run by Julia each time it starts up. Let's have some fun and do this using Julia—and practice a bit of what we've learned so far.
First, go into shell mode and run these three commands:
shell> mkdir $(dirname(REPL.find_hist_file()))/../config shell> cd $(dirname(REPL.find_hist_file()))/../config /Users/adrian/.julia/config shell> touch startup.jl
Then, in julian mode, execute the following:
julia> write("startup.jl", "println(\"Welcome to Julia!\")") 28
What did we just do? In shell mode, we created a new directory, called
config, just one folder up from where our history file was. Then we
cd into the newly created folder, where we created a new file called
startup.jl. Finally, we asked Julia to add the line
"println(\"Welcome to Julia!\")" to the
startup.jl file. Next time we start the Julia REPL we'll be greeted by this welcome message. Check this out:
It is also possible to define a function that will be automatically called before starting a REPL session. To achieve this, you need to use the
atreplinit(f) function, which registers a one-argument function
f to be called before the REPL interface is initialized in interactive sessions. This function should be called from within the
Let's say that we edit our
startup.jl file so that it now looks like this:
println("Welcome to Julia!") atreplinit() do (f) println("And welcome to you too!") end
Our REPL will now greet us twice:
In order to exit the REPL, you can type
^ D (Ctrl + D). However, that will only work if you're at the beginning of the line (when the text buffer is empty). Otherwise just type
^C (Ctrl + C) to first interrupt (or cancel) and clear the line. You can also run
exit(), which will stop the execution of the current Julia process.
For the complete list of key bindings at the REPL and how to customise them, you can read the official documentation at https://docs.julialang.org/en/v1.0/stdlib/REPL/#Key-bindings-1.
Your Julia installation comes with a powerful package manager called
Pkg. This handles all the expected operations, such as adding and removing packages, resolving dependencies and keeping installed packages up to date, running tests, and even assisting with publishing our own packages.
Packages play a pivotal role by providing a wide range of functionality, seamlessly extending the core language. Let's take a look at the most important package management functions.
In order to be known to
Pkg, the packages must be added to a registry that is available to Julia.
Pkg supports working with multiple registries simultaneously—including private ones hosted behind corporate firewalls. By default,
Pkg is configured to use Julia's General registry, a repository of free and open sources packages maintained by the Julia community.
Pkg is quite a powerful beast and we'll use it extensively throughout the book. Package management is a common task when developing with Julia so we'll have multiple opportunities to progressively dive deeper. We'll take our first steps now as we'll learn how to add packages—and we'll do this by stacking a few powerful new features to our Julia setup.
One of my favourite packages is called
OhMyREPL. It implements a few super productive features for the Julia REPL, most notably syntax highlighting and brackets pairing. It's a great addition that makes the interactive coding experience even more pleasant and efficient.
Pkg is centered around GitHub. The creators distribute the packages as git repos, hosted on GitHub—and even the General registry is a GitHub repository itself.
OhMyREPL is no exception. If you want to learn more about it before installing it—always a good idea when using code from third parties — you can check it out at https://github.com/KristofferC/OhMyREPL.jl
Keep in mind that even if it's part of the General registry, the packages come with no guarantees and they're not necessarily checked, validated or endorsed by the Julia community. However, there are a few common sense indicators which provide insight into the quality of the package, most notably the number of stars, the status of the tests as well as the support for the most recent Julia versions.
The first thing we need to do in order to add a package is to enter the
Pkg REPL-mode. We do this by typing
] at the beginning of the line:
The cursor will change to reflect that we're now ready to manage packages:
IJulia does not (yet) support the
pkg> mode, but we can execute
Pkg commands by wrapping them in
pkg"..." as in
Pkg uses the concept of environments, allowing us to define distinct and independent sets of packages on a per-project basis. This is a very powerful and useful feature, as it eliminates dependency conflicts caused by projects that rely on different versions of the same package (the so-called dependency hell).
Given that we haven't created any project yet,
Pkg will just use the default project,
v1.0, indicated by the value between the parenthesis. This represents the Julia version that you're running on—and it's possible that you'll get a different default project depending on your very own version of Julia.
Now we can just go ahead and
(v1.0) pkg> add OhMyREPL Updating registry at `~/.julia/registries/General` Updating git-repo `https://github.com/JuliaRegistries/General.git` Resolving package versions... Updating `~/.julia/environments/v1.0/Project.toml` [5fb14364] + OhMyREPL v0.3.0 Updating `~/.julia/environments/v1.0/Manifest.toml` [a8cc5b0e] + Crayons v1.0.0 [5fb14364] + OhMyREPL v0.3.0 [0796e94c] + Tokenize v0.5.2 [2a0f44e3] + Base64 [ade2ca70] + Dates [8ba89e20] + Distributed [b77e0a4c] + InteractiveUtils [76f85450] + LibGit2 [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [44cfe95a] + Pkg [de0858da] + Printf [3fa0cd96] + REPL [9a3f8284] + Random [ea8e919c] + SHA [9e88b42a] + Serialization [6462fe0b] + Sockets [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode
pkg> add on a fresh Julia installation,
Pkg will clone Julia's General registry and use it to look up the names of the package we requested. Although we only explicitly asked for
OhMyREPL, most Julia packages have external dependencies that also need to be installed. As we can see, our package has quite a few—but they were promptly installed by
Sometimes we might want to use packages that are not added to the general registry. This is usually the case with packages that are under (early) development—or private packages. For such situations, we can pass
pkg> add the URL of the repository, instead of the package's name:
(v1.0) pkg> add https://github.com/JuliaLang/Example.jl.git Cloning git-repo `https://github.com/JuliaLang/Example.jl.git` Updating git-repo `https://github.com/JuliaLang/Example.jl.git` Resolving package versions... Updating `~/.julia/environments/v1.0/Project.toml` [7876af07] + Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git) Updating `~/.julia/environments/v1.0/Manifest.toml` [7876af07] + Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git)
Another common scenario is when we want to install a certain branch of a package's repository. This can be easily achieved by appending
#name_of_the_branch at the end of the package's name or URL:
(v1.0) pkg> add OhMyREPL#master Cloning git-repo `https://github.com/KristofferC/OhMyREPL.jl.git` Updating git-repo `https://github.com/KristofferC/OhMyREPL.jl.git` Resolving package versions... Installed Crayons ─ v0.5.1 Updating `~/.julia/environments/v1.0/Project.toml` [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0 #master (https://github.com/KristofferC/OhMyREPL.jl.git) Updating `~/.julia/environments/v1.0/Manifest.toml`
[a8cc5b0e] ↓ Crayons v1.0.0 ⇒ v0.5.1 [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0 #master (https://github.com/KristofferC/OhMyREPL.jl.git)
Or, for unregistered packages, use the following:
(v1.0) pkg> add https://github.com/JuliaLang/Example.jl.git#master
If we want to get back to using the published branch, we need to
free the package:
(v1.0) pkg> free OhMyREPL Resolving package versions... Updating `~/.julia/environments/v1.0/Project.toml` [5fb14364] ~ OhMyREPL v0.3.0 #master (https://github.com/KristofferC/OhMyREPL.jl.git) ⇒ v0.3.0 Updating `~/.julia/environments/v1.0/Manifest.toml` [a8cc5b0e] ↑ Crayons v0.5.1 ⇒ v1.0.0 [5fb14364] ~ OhMyREPL v0.3.0 #master (https://github.com/KristofferC/OhMyREPL.jl.git) ⇒ v0.3.0
That was easy, but practice makes perfect. Let's add one more! This time we'll install
Revise, another must-have package that enables a streamlined development workflow by monitoring and detecting changes in your Julia files and automatically reloading the code when needed. Before
Revise it was notoriously difficult to load changes in the current Julia process, developers usually being forced to restart the REPL—a time-consuming and inefficient process.
Revise can eliminate the overhead of restarting, loading packages, and waiting for code to compile.
You can learn more about Revise by reading its docs at https://timholy.github.io/Revise.jl/latest/.
Unsurprisingly, we only have to invoke
add one more time, this time passing in
Revise for the package name:
(v1.0) pkg> add Revise Resolving package versions... Installed Revise ─ v0.7.5 Updating `~/.julia/environments/v1.0/Project.toml` [295af30f] + Revise v0.7.5 Updating `~/.julia/environments/v1.0/Manifest.toml` [bac558e1] + OrderedCollections v0.1.0
[295af30f] + Revise v0.7.5 [7b1f6079] + FileWatching
(v1.0) pkg> status Status `~/.julia/environments/v1.0/Project.toml` [7876af07] Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git) [5fb14364] OhMyREPL v0.3.0 [295af30f] Revise v0.7.5
status command displays all the installed packages, including, from left to right, the short version of the package's id (called the UUID), the name of the package and the version number. Where appropriate, it will also indicate the branch that we're tracking, as in the case of
Example, where we're on the
Once a package has been added, in order to access its functionality we have to bring into scope. That's how we tell Julia that we intend to use it, asking the compiler to make it available for us. For that, first, we need to exit pkg mode. Once we're at the julian prompt, in order to use
OhMyREPL, all we need to do is execute:
julia> using OhMyREPL [ Info: Precompiling OhMyREPL [5fb14364-9ced-5910-84b2-373655c76a03]
That's all it takes—
OhMyREPL is now automatically enhancing the current REPL session. To see it in action, here is what the regular REPL looks like:
And here is the same code, enhanced by
Syntax highlighting and bracket matching make the code more readable, reducing syntax errors. Looks pretty awesome, doesn't it?
OhMyREPL has a few more cool features up its sleeve—you can learn about them by checking the official documentation at https://kristofferc.github.io/OhMyREPL.jl/latest/index.html.
Revise are excellent development tools and it's very useful to have them loaded automatically in all the Julia sessions. This is exactly why the
startup.jl file exists—and now we have the opportunity to put it to good use (not that our heartfelt and welcoming greetings were not impressive enough!).
This will open the file in the default editor. If you haven't yet deleted our previously added welcome messages, feel free to do it now (unless you really like them and in that case, by all means, you can keep them). Now,
Revise needs to be used before any other module that we want to track—so we'll want to have it at the top of the file. As for
OhMyREPL, it can go next. Your
startup.jl file should look like this:
using Revise using OhMyREPL
Save it and close the editor. Next time you start Julia, both
OhMyREPL will be already loaded.
(v1.0) pkg> update
When this command is issued, Julia will first retrieve the latest version of the general repository, where it will check if any of the packages need to be updated.
Beware that issuing the
update command will update all the available packages. As we discussed earlier, when mentioning dependency hell, this might not be the best thing. In the upcoming chapters, we will see how to work with individual projects and manage dependencies per individual application. Until then though, it's important to know that you can cherry pick the packages that you want to update by passing in their names:
(v1.0) pkg> update OhMyREPL Revise
Pkg also exposes a preview mode, which will show what will happen when running a certain command without actually making any of the changes:
(v1.0) pkg> preview update OhMyREPL (v1.0) pkg> preview add HTTP
(v1.0) pkg> pin OhMyREPL Resolving package versions... Updating `~/.julia/environments/v1.0/Project.toml` [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0 Updating `~/.julia/environments/v1.0/Manifest.toml` [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0
Pinned packages are marked with the
⚲ symbol—also present now when checking the status:
(v1.0) pkg> st Status `~/.julia/environments/v1.0/Project.toml` [5fb14364] OhMyREPL v0.3.0 [295af30f] Revise v0.7.5
If we want to unpin a package, we can use
(v1.0) pkg> free OhMyREPL Updating `~/.julia/environments/v1.0/Project.toml` [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0 Updating `~/.julia/environments/v1.0/Manifest.toml` [5fb14364] ~ OhMyREPL v0.3.0 ⇒ v0.3.0 (v1.0) pkg> st Status `~/.julia/environments/v1.0/Project.toml` [5fb14364] OhMyREPL v0.3.0 [295af30f] Revise v0.7.5
(v1.0) pkg> st Status `~/.julia/environments/v1.0/Project.toml` [7876af07] Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git) [5fb14364] OhMyREPL v0.3.0 [295af30f] Revise v0.7.5
We can remove the
Example package with the following code:
(v1.0) pkg> remove Example Updating `~/.julia/environments/v1.0/Project.toml` [7876af07] - Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git) Updating `~/.julia/environments/v1.0/Manifest.toml` [7876af07] - Example v0.5.1+ #master (https://github.com/JuliaLang/Example.jl.git)
Sure enough, it's now gone:
(v1.0) pkg> st Status `~/.julia/environments/v1.0/Project.toml` [5fb14364] OhMyREPL v0.3.0 [295af30f] Revise v0.7.5
Besides the explicit removal of undesired packages,
Pkg also has a built-in auto-cleanup function. As package versions evolve and package dependencies change, some of the installed packages will become obsolete and will no longer be used in any existing project.
Pkg keeps a log of all the projects used so it can go through the log and see exactly which projects are still needing which packages—and thus identify the ones that are no longer necessary. These can be deleted in one swoop with the
pkg> gc command:
(v1.0) pkg> gc Active manifests at: `/Users/adrian/.julia/environments/v1.0/Manifest.toml` `/Users/adrian/.julia/environments/v0.7/Manifest.toml` Deleted /Users/adrian/.julia/packages/Acorn/exWWb: 40.852 KiB Deleted /Users/adrian/.julia/packages/BufferedStreams/hCA7W: 102.235 KiB Deleted /Users/adrian/.julia/packages/Crayons/e1SsX: 49.133 KiB Deleted /Users/adrian/.julia/packages/Example/ljaU2: 4.625 KiB Deleted /Users/adrian/.julia/packages/Genie/XOia2: 2.031 MiB Deleted /Users/adrian/.julia/packages/HTTPClient/ZQR55: 37.669 KiB Deleted /Users/adrian/.julia/packages/Homebrew/l8kUw: 277.296 MiB Deleted /Users/adrian/.julia/packages/LibCURL/Qs5og: 11.599 MiB Deleted /Users/adrian/.julia/packages/LibExpat/6jLDP: 127.247 KiB Deleted /Users/adrian/.julia/packages/LibPQ/N7lDU: 134.734 KiB Deleted /Users/adrian/.julia/packages/Libz/zMAun: 80.744 KiB Deleted /Users/adrian/.julia/packages/Nettle/LMDZh: 50.371 KiB
Deleted /Users/adrian/.julia/packages/OhMyREPL/limOC: 448.493 KiB Deleted /Users/adrian/.julia/packages/WinRPM/rDDZz: 24.925 KiB Deleted 14 package installations : 292.001 MiB
Besides the dedicated
Pkg REPL mode, Julia also provides a powerful API for programmatically managing packages. We won't cover it, but if you want to learn about it, you can check the official documentation at https://docs.julialang.org/en/latest/stdlib/Pkg/#References-1.
Package discovery is not yet as simple as it could be, but there are a few good options. I recommend starting with this list of curated Julia packages: https://github.com/svaksha/Julia.jl. It groups a large collection of packages by domain, covering topics such as AI, Biology, Chemistry, Databases, Graphics, Data Science, Physics, Statistics, Super-Computing and more.
If that is not enough, you can always go to https://discourse.julialang.org where the Julia community discusses a multitude of topics related to the language. You can search and browse the existing threads, especially the package announcements section, hosted at https://discourse.julialang.org/c/community/packages.
Of course you can always ask the community for help—Julians are very friendly and welcoming, and a lot of effort is put towards moderation, in order to keep the discussion civilized and constructive. A free Discourse account is required in order to create new topics and post replies.
Finally, https://juliaobserver.com/packages is a third party website that provides a more polished way to look for packages—and it also performs a GitHub search, thus including unregistered packages too.
Although I already touched upon the topic in the previous paragraphs, I want to close the discussion about
Pkg with a word of caution. The fact that a package is registered does not necessarily mean that it has been vetted in terms of functionality or security. It simply means that the package has been submitted by the creator and that it met certain technical requirements to be added to the general registry. The package sources are available on GitHub, and like with any open source software, make sure that you understand what it does, how it should be used, and that you accept the licensing terms.
This concludes our initial discussion about package management. But as this is one of the most common tasks, we'll come back to it again and again in future chapters, where we'll also see a few scenarios for more advanced usage.
Julia is a new programming language that takes advantage of recent innovations in compiler technology to offer the functionality, ease-of-use, and intuitive syntax of dynamic programming languages at the speed of C. One if its goals is to eliminate the so-called two language problem—when the users code in a high-level language, such as R and Python, but performance-critical parts have to be rewritten in C or C++. Julia feels like a dynamic language and offers all the productivity features associated with these. But at the same time, it eliminates the performance trade-offs, proving to be productive enough for prototyping and exploratory coding, and efficient enough for performance-critical applications.
Its built-in package manager provides access to over 2,000 third-party libraries that seamlessly extend the language with powerful new features—and we've learned how to take advantage of these. And if that is not enough, Julia has the ability to call functions written in other languages, such as C, Fortran, Python, or Java, to name just a few.
Julia is free and open source (MIT licensed) and can be deployed on all the major operating systems, including Windows, the main Linux distributions, and macOS. It also comes with some very good IDE and editor options.
Now that we have successfully set up our development environment, it's time to dive deeper into Julia's syntax. In the next chapter, we'll take a look at some of the basic building blocks of the language—defining variables and constants, manipulating and using
Strings and numeric types, and working with
Arrays. As a testament to Julia's productivity, that's all we'll need (together with some extra packages that we'll add) in order to perform powerful exploratory data analysis on the Iris flowers dataset. Meet you in the next chapter!