How to publish a website that executes R and python (+ OpenAI's API for GPT/DALL-E calls)
Quarto by Posit is my new favorite technical publishing system: it is powerful, flexible and easy. If you want to learn (and share) data science, there's never been a better time to start.
Yesterday I published this demonstration post to my new data blog. I curiously wanted to test two things:
Could I easily prompt GPT and DALL-E via the OpenAI API?
Could I really mix executable R and Python code in a single document (just one of Quarto’s many promoted flexibilities)?
You can see details (including my code blocks) at the link. The answer to both question is: yes, and neither was difficult1. My post illustrates two different API use cases: prompting in the executable code, and co-piloting (via the gptstudio package) in the IDE. Unless you’re already an ace coder, I find AI-assisted coding to be a bona fide superpower. I happen to believe—and my conviction is increasing—that “AI is a really big deal for the future of work.” If you agree with this premise, I’d say the best way to engage with AI is to starting doing self-assigned projects. Just start!
Why I am saying this is my new favorite system? Because I didn’t really encounter any fundamental problems installing the site, posting my entries, and writing the R-to-python-back-to-R test code. If you’ve ever tried stuff like this, I’m so confident you are familiar with the experience of failed package installations, stopped render jobs, error messages … all the little interruptions that seem to be part of the steep learning curve. But this Quarto system just works. I’ve deployed a lot sites using various technologies going way back2 . It’s never been this easy, relatively speaking. You do have to be willing to read their documentation if you get stuck, but the documentation is great (I’m not surprised by that: having attended their conferences, I already knew this is a special company that cares. Posit’s Chief Scientist is Hadley Wickham, who some sincerely consider the best data scientist on the planet).
I thought it might be helpful if I shared the process to publish my Quarto website.
Why do I prefer R?
I prefer R to python where my primary coding tool (aka, IDE) is RStudio. Why do I prefer R? Several reasons. First, the visualization package ggplot2 is beautiful and it's part of the elegant tidyverse. The importance of dataviz is affirmed by Posit’s admirable commitment to the table package gt;
Second, I'm not a developer working in production environments. As an analyst, code is the next step after Excel: not to write applications, but to conduct ad hoc data analysis. Third, R is designed for statistical analysis (where I started my journey teaching financial risk). Once you get past the early, steep learning curve and comfortable with its data structures and native vectorization, R shines in quick ad hoc analysis. I am not qualified to argue pros/cons at the frontiers of machine learning (ML), but among the set of basic techniques, R lacks for nothing. Most of the R-versus-python debates are moot to non-professional developers. Fourth, the R community is awesome.
To deploy my Quarto site (at https://finedtech.io/) required two broad steps.
Install Quarto and Create a Quarto (blog) Site
When new blog posts are added/edited, Render the site (or page) and Publish the rendered site to Netlify via GitHub. Alternatively, you can publish to any of six+ destinations that range from super-easy (e.g. Quarto Pub) to more sophisticated.
1a. Install Posit system: R + RStudio + Quarto
As usual for Posit, the Quarto documentation is excellent. First install R and RStudio Desktop (both here) to your local machine. Then go here to install Quarto. For my R tools, I do not like Windows' deeply nested default locations; I like my key directories at the root where I can see them. For me, these first three installations locate to D:\Rbase, D:\RStudio, and D:\Quarto:
You can see, I also locate my R \library at the D: root. This library is key in R: it contains all of my packages (each as a subdirectory). Most packages are easily installed via the install_packages("package_name") function. They are then retrieved at execution via the library(package_name) function. For example, library(tidyverse) loads the entire tidyverse because it’s a single (meta-) package that contains a brilliant list of useful packages (each of which has a cheatsheet).
I think it's helpful to get comfortable with the PATH variable. When you run a command in a shell, Windows looks to directories listed in the path variable. Just as Windows has a PATH, R has a path (called a library tree) that we get/set with the .libPaths() function. Per my root-biased structure, I appended my windows PATH to include "D:\python"; and analogously my Rprofile.site contains a line—i.e., .libPaths(c("D:/library", .libPaths()))—that appends the root library to the existing .libPath. When learning R, it’s good to understand the steps in its startup sequence. I won't cover the python installation here; you don't need to install python for Quarto.
1b. Create a Quarto website
Now we can create a quarto website per these instructions. Posit has learned how to simplify all this, yet they've also tucked away much flexibility. My own finedtech.io website is locally installed at my D:\github\finedtech2 but the local location of the website is a matter of personal organization.
As the instructions mention, “_quarto.yml” is the config file. My own _quarto.yml file (see below) lazily borrows the built-in sandstone theme. I also added a custom.scss to change the color; the .scss extension refers to Sass which is a superset of CSS.
This _quarto.yml config file is located in the root of my local Quarto website at D:\github\fintech2. I love how Posit has streamlined file organization (see below). My root only contains two actual Quarto documents (index.qmd and about.qmd). The finedtech2.Rproj is an Rstudio project file that makes it easy to switch among different projects; i.e., it stores the settings associated with this working directory.
Each blog post is a folder under the /posts directory, and each post has its own index.qmd file. Consequently, we can view my post via https://finedtech.io/posts/this-quarto-site/index.html or at https://finedtech.io/posts/this-quarto-site/
To add a post, I only need to add a new sub-folder under finedtech2/posts and create a new index.qmd document. Those are the essentials of a local Quarto website.
2a. Render the website
Rendering is the crucial process of translating source documents (i.e., the .qmd files) into their ultimate viewable destination file fomat(s); e.g., pdf, word, html. We can render the page or render the whole site (aka, build the site); both are push-button actions in RStudio. What's important is that the rendered site gets stored in the _site subdirectory; in my case, that's finedtech2/_site. When building the site, what doesn't get rendered? Files with a "." or "_" prefix which are the files the sort to the top in a directory view; in my case, the _freeze, _quarto.yml, _site, .git, .gitignore, .quarto, and .Rproj.user files do not get rendered as they aren't site content. Notice the slim set that does get rendered: about.qmd (my about page) and drh3.jpg (my picture for my about page); styles.css and custom.scss (that I added); index.qmd (the site’s home page), and the posts subdirectory (that itself holds one directory for each post).
2b. Publish the website
It's easier than ever to publish. The instructions are here. Quarto Pub is free and easy but this won't feel like your own site because it’s not; it is popular for homework assignments, I think. For my first test, I did publish my new site to GitHub Pages and it worked just fine. Still, my preference is Netlify per the instructions. Netlify is amazing, and it's free for personal projects.
To publish to my finedtech.io website requires two actions but it’s really only one step for me (after the one-time Netlify configuration):
Push the local repo3 to my github at https://github.com/bionicturtle/finedtech2
Do nothing myself (aka, wait 15 seconds or less) as Netlify automatically deploys the site. I only needed to configure the site once on Netlify: it re-deploys whenever committed changes are pushed.
For the first (and only ongoing!) step, I use Git Bash, but there are GUI alternatives, in particular GitHub Desktop. I'm old school (and old): I worry that if don't practice command prompting, then I might lose the skill (same reason I held out on stick shifts!). In either case, I highly recommend that you acquire basic version control skills. The future of work wants us all to get very comfortable with platforms like GitHub and Hugging Face, and at least a little bit familiar with version control if only because these skills enable collaboration and sharing. Forget about your resume and start thinking about your viewable portfolio. If you are totally new to coding and are looking for a short course(s) to take, git/GitHub/etc would be my first choice.
For the one-time Netlify configuration, the key step is to tell Netlify where to find the rendered site. In my case, that's the _site subdirectory. See below.
Netlify will give you a clunky domain name, e.g., you can also access my site at its originally assigned https://statuesque-marzipan-291f42.netlify.app/. If you are serious, they have a domain registration service that requires payment (but that’s easy: if you find an available domain, you just pay for it, and it’s a seamless “replacement”). In my case, I had already registered finedtech.io at GoDaddy. So I did need to delegate my domain to Netlify, but propagation took less than three hours.
Do this and learn a lot
So that's the process I took: install R, Rstudio and Quarto; minimal edits to out-of-the-box theme; add a few posts via post directory; render pages as desired and render (aka, build) the website; and publish the website. I hope that was helpful!
Thank you Nicola Rennie for this post that showed me r.lemur_data and py$lemur_data_py to share the dataframe
All the way back to Adobe Dreamweaver
You may have noticed that my /finedtech2 root contains a .git subdirectory. This signifies it is a Git repository. As mentioned, because the name begins with a period, Quarto will exclude this .git subdirectory when it renders.