solarchemist.se

Contact

Mon, 28 Apr 2025 18:00:00 +0100

E-mail ✉️

hello[at]solarchemist[dot]se
(I trust you to replace the brackets with the proper symbols).

On the Fediverse

Follow or message me at either of these Mastodon accounts (whichever suits you best).

Signal chat

Message me: solarchemist.03.

Matrix chat

Get in touch with me on the federated Matrix protocol:
@solarchemist:matrix.org

Other services

If none of the above suit you, you could give one of these a try (but please note that I don’t check these services very often):

linkedin.com/in/solarchemist

Mon, 27 Nov 2023 08:00:00 +0100

Taha’s PhD thesis

Nanostructured ZnO and metal chalcogenide films
for solar photocatalysis

Taha Ahmed

PDF (DiVA)   

Source code repository on Codeberg or GitHub .

Publicly defended 2023-12-18 at the Ångström laboratory, faculty of science and technology, Uppsala University.
Errata

Thesis advisors: Tomas Edvinsson and Jiefang Zhu .

Source code and reproducibility

In addition to the PDF of the thesis, I have chosen to also share the code and data that make up my thesis, in the form of a git repository shared under a libre license, making it easy for you to inspect the code, recompile the whole thesis or any parts of it on your own, or adapt all or part of it for your own work.

You are of course more than welcome to reach out to me with comments, criticism, or just to let me know you found some part of the thesis source code useful.

To maintain all the source files and other dependencies of this thesis I used git (what’s git? find out!). The source code repository (note that the Codeberg and Github repositories are identical, use whichever host you prefer) contain the dependencies, the data (larger datasets have been packaged as binary rda archives to save disk space and compilation speed; you could reconstruct the original text files from them using R), and the LuaLaTeX code required to reproduce the thesis as-published on DiVA.

I have cleaned up this repository from zombie code, orphaned files, and other stuff that crept into the work over the years but which is no longer necessary, but no one is perfect, so do let me know if you find any discrepancies between the PDF compiled from the repository (make sure the document was set to final mode) and the PDF on DiVA.

The README in the repository contains more hands-on information on how to download the repository and compile the thesis on your own computer.

Links and notes

I have highlighted some of the features (mostly LaTeX-related) I employed in the thesis to implement some custom functionalities:

Other PhD theses with source code

I am far from the first to share the source code of their PhD thesis. Here are only a few others that have crossed my path:

Most repos on GitHub use the phd-thesis tag, but so far none of the above repos on Codeberg use any tags.

Other thesis templates

It seems every university or college with some self-respect has published a LaTeX-based thesis template. Simply too many to list. Below, a non-representative list:

http://cleanthesis.der-ric.de
https://libguides.ub.uu.se/avhandling/latex XeLaTeX template for Uppsala University
https://github.com/kks32/phd-thesis-template LaTeX/XeLaTeX/LuaLaTeX template for Cambridge University
https://github.com/ismayc/thesisdown R Markdown using bookdown
https://codeberg.org/slumber/phd-latex-template LaTeX template for Université Paris 8 - Vincennes Saint-Denis (poorly documented)
https://codeberg.org/bnass/PhD_shared_scripts scripts for data management, analysis, visualization

Ucar is the author of the awesome errors package. ↩︎

The solar origins of spectroscopy

Tue, 04 Apr 2023 20:00:00 +0100

Spectroscopy is of a foundational importance to physical chemistry, and has been a central fixture in both my own graduate and undergraduate work. In this post I would like to delineate how the birth of spectroscopy stemmed from the study of the solar spectrum, and how sunlight will form the basis of a better global energy conversion system.

Up until widespread electrification of lighting (less than 200 years ago), daylight was the primary light source (not counting gas lamps, candles, fires, and so on). Daylight is sunlight filtered through the atmosphere.¹ If, for the sake of example, we consider the atmosphere to be constant (which it most definitely is not, or see why star’s twinkle), then daylight is simply a function of the Sun’s spectrum attenuated by that constant.

The scientific study of sunlight began only 200 years ago (more on that below), but the question of the Sun’s distance to us, its size and age has probably enagaged all civilizations.

The realisation that the Sun must be very far away was made already in antiquity, most notably by the Hellenic astronomer Posidonius the Syrian who in 90 BC calculated the distance and was only wrong by less than half [1]. The Sun’s mechanism and age,² however, were not correctly determined until the 20th century and the advent of the field of stellar nucleosynthesis. So in a sense, despite living under the same Sun as all our ancestors back to the beginning of humanity, humans have not been in the privileged position of knowing what stuff a star is made of until quite recently (more recent than the invention of the bike, for comparison).

The beginnings of the science of spectroscopy

The observations of “lines” in sunlight, candle-light and electric light seen through single prisms by Wollaston in 1802 [2] can, with hindsight, be considered the beginning of spectroscopy as we know it. But his results might well have been forgotten if Fraunhofer had not made new observations on sunlight 13 years later by adding a telescope to the prism and slit, observing “almost innumerable strong and weak vertical lines which however are darker than the other part of the coloured image; some seem to be almost completely black”³.

Figure 1: This copper-etching was made by Fraunhofer himself. It shows the full visible spectrum including the famous Fraunhofer lines⁴, some of them named with letters. Above it is the curve of spectral response of the eye to daylight illumination. Wikimedia, CC-BY-SA.

Thanks to Kirchhoff and Bunsen (1859) the relationship⁵ between dark and bright lines of the same wavelength (absorption vs emission, in modern terms) were understood, and also the unique correspondence of line wavelength to chemical element, thus making the spectrum a fixed frame of reference and seeding the ground for the budding field of astrophysics, a field which bloomed after the Doppler effect was described in 1842.

A few decades later Anders J. Ångström made an important contribution to spectroscopy by mapping one thousand absorption lines in the solar spectrum (and was the first to label the spectrum in units of 10^-10 m, later named after him) [3] which was improved on by Rowland who carefully measured 21 thousand lines in 1897. Modern measurements of the solar spectrum have shown that at least 62 elements are certainly present in the Sun’s atmosphere. Considering that by mass 98.4 % of the sun consists of just hydrogen and helium,⁶ that’s a large variation of heavier elements.

These discoveries all concerned the visible part of the spectrum. But after the development of the optical prism (by the 19th century) scientists began to expand the spectrum beyond visible light: Herschel discovered the infrared part of the solar spectrum in 1800, and Ritter discovered the ultraviolet one a year later. Both spectral regions were found to contain more absorption lines.

The method of detecting and recording one’s observed spectrum improved over the years and was tightly interwoven with the development of photography.

Today we have the line spectra of all naturally occuring elements available at our fingertips from a variety of tools, here for example the LaTeX pgf-spectra package [5] which builds on spectral data from NIST:

Figure 2: Emission lines of sodium in the visible spectrum.

Sunlight above and below the atmosphere

Whereas the spectral composition of daylight varies dramatically with season, location and time of day, the amount of sunlight that reaches the top of Earth’s atmosphere (i.e., the total solar irradiance, TSI, previously known as the solar constant) has been experimentally shown to be practically constant over time, and any variations are certainly not enough to account for the rise in mean global surface temperature observed since the industrial revolution.

The first experimental observations in the modern sense of the solar constant (i.e., TSI) were made in the early 19th century by Herschel and Poillet using their newly invented actinometer and pyrheliometer, respectively. They were soon followed by John Ericsson, who invented what he called a “solar calorimeter” that improved on the earlier devices by taking the atmospheric column above the measurement into better account and arrived at a value of 7.11 BTU⁷ per minute per square foot of surface at the boundary of the atmosphere [6], which is equivalent to $\SI{1332}{\watt\per\square\metre}$ and very close to the modern value.

Figure 3: Solar irradiation at the top of the atmosphere (AM0, yellow area) and at sea level (AM1.5G, red area), as well as the solar irradiance modelled as a black body (black curve). Also shown is the total irradiance for each of the UV, Vis, and NIR spectral regions for the AM0 and AM1.5G spectra.

Since 1978, solar irradiance above the Earth’s atmosphere is continuously measured by satellite-based radiometers, which is then averaged over the year and corrected to a distance of exactly 1 AU to yield the TSI. Indeed, one of the reasons the “solar constant” was renamed to TSI is that it is not really constant, but changes depending on solar output, which varies mainly with the solar sunspot cycle (which goes from minimum to maximum magnetic activity in 11-year cycles). The last solar minimum was in 2022, and TSI during the preceding minimum in 2011 was $\SI{1360.8\pm0.5}{\watt\per\square\metre}$ [7]. But note that whereas the solar irradiance at the top of Earth’s atmosphere varies quite a lot depending on several periodic phenonema of varying timescales, TSI varies very little. For example, TSI shows less than 0.2 % variation due to the sunspot cycle [8].

Below the atmosphere, that is, on the ground we (that is, in the photovoltaic field) almost always resort to the ASTM G173-03 solar reference spectra, which give us a full solar spectrum (250 nm–4000 nm) depending primarily on the air mass, which is a convenient way to express latitude. Of course, more detailed models exist, but for the purpose of scientific reports such international standards are really all that is needed.

Notes and links

If you are interested in the physics of daylight and the history of its study I can recommend the treatise by Henderson [1] (a book which might be hard to find, though).

Spectroscopy, Britannica. A remarkably comprehensive article.
What is the Mössbauer effect?, Skulls in the Stars (2023). Great explainer of absorption/emission in atoms.
Invisible light: prism experiments through the centuries, Wallifaction (2014)
A brief (incomplete) history of light and spectra
Electrification, Wikipedia
The historical cost of light
Introduction to spectroscopy, NASA
UV-visible spectroscopy, LibreTexts Chemistry
Actinometer, Wikipedia
Pyrheliometer, Wikipedia
Ångström, Wikipedia
Spectroscopy, Wikipedia
History of spectroscopy, Wikipedia
Air mass (solar energy), Wikipedia
Spectrometer, spectroscope, and spectrograph, excerpt from Field Guide to Spectroscopy [9]
Spectrophotometry, Wikipedia
Spectrophotometry, LibreTexts Chemistry
Atomic spectroscopic data, NIST
Total solar irradiance data, Laboratory for Atmospheric and Space Physics, University of Colorado Boulder (2020)
Using spectroscopy for solar irradiance and UV measurements, AZO Materials (2017)
Solar irradiance, ed. Rob Garner, NASA (2017)
Solar radiation measurements, presentation slides (PDF) from workshop for the National Association of State Universities and Land Grant Colleges, Tom Stoffel and Steve Wilcox, NREL (2004)
Enhancements enable solar simulators to shed light on new photovoltaic designs, Jay Jeong (2007)
What is an optical spectrometer?, West and Harrell, Oxford Instruments (2022)

Session info

## Linux 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64

## R version 4.1.3 (2022-03-10)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.3 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=sv_SE.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] bandgaps_0.1.1.9000 photoec_0.4.2.9000  common_0.1.2        stringr_1.4.0      
##  [5] dplyr_1.0.10        tibble_3.1.8        tidyr_1.2.1         ggplot2_3.3.6      
##  [9] knitr_1.39          git2r_0.30.1        here_1.0.1          conflicted_1.1.0   
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_1.2.0 xfun_0.31        bslib_0.4.0      purrr_0.3.5     
##  [5] colorspace_2.0-3 vctrs_0.5.1      generics_0.1.3   htmltools_0.5.3 
##  [9] yaml_2.3.5       utf8_1.2.2       rlang_1.0.6      jquerylib_0.1.4 
## [13] pillar_1.8.1     glue_1.6.2       withr_2.5.0      DBI_1.1.3       
## [17] lifecycle_1.0.3  munsell_0.5.0    blogdown_1.10    gtable_0.3.0    
## [21] memoise_2.0.1    evaluate_0.15    labeling_0.4.2   fastmap_1.1.0   
## [25] fansi_1.0.3      highr_0.9        scales_1.2.0     cachem_1.0.6    
## [29] jsonlite_1.8.0   farver_2.1.1     digest_0.6.29    stringi_1.7.8   
## [33] bookdown_0.27    grid_4.1.3       rprojroot_2.0.3  cli_3.4.1       
## [37] tools_4.1.3      magrittr_2.0.3   sass_0.4.2       crayon_1.5.1    
## [41] pkgconfig_2.0.3  ellipsis_0.3.2   assertthat_0.2.1 rmarkdown_2.16  
## [45] R6_2.5.1         compiler_4.1.3

## Commit:  ec42cbcbc2cd1968feedcfe4b13f4f15d76c32e9
## Author:  taha@luxor <taha@chepec.se>
## When:    2023-10-09 20:58:44 GMT
## 
##      Added a link.
##      
## 1 file changed, 2 insertions, 1 deletions
## index.Rmd | -1 +2  in 1 hunk

## working directory clean

References

[1] Henderson, S.T., Daylight and its spectrum, New York, Elsevier, 1970.

[2] Wollaston, W.H., XII. A method of examining refractive and dispersive powers, by prismatic reflection, Philosophical Transactions of the Royal Society of London, v. 92, pp. 365–380, Jan. 1802.

[3] Ångström, A.J., Recherches sur le spectre solaire, Uppsala, W. Schultz, 1868.

[4] Murdin, P. (ed), Encyclopedia of astronomy & astrophysics, Boca Raton, CRC Press, 2001.

[5] Gomes, H., “Pgf-spectra: Draw continuous or discrete spectra using PGF/TikZ,” 27-Apr-2023.

[6] McVeigh, J.C., Sun power: An introduction to the applications of solar energy, 2nd ed., Oxford, Pergamon, 1983.

[7] Kopp, G., Lean, J.L., A new, lower value of total solar irradiance: Evidence and climate significance, Geophysical Research Letters, v. 38, n. 1, Jan. 2011.

[8] Kopp, G., “5.02 - Earth’s incoming energy: The total solar irradiance,” In: Liang, S. (ed), Comprehensive Remote Sensing, Oxford, Elsevier, pp. 32–66, 2018.

[9] Ball, D.W., Field guide to spectroscopy, v. FG08, Bellingham, Washington USA, SPIE Press, 2006.

In the UV to NIR spectral range the atmosphere can be considered a combination of multiple band-stop filters, where different compounds block specific wavelengths (see atmospheric window).↩︎
The realisation that the Sun must be very far away from us was much easier to grasp than the mechanism that powered it, and by extension, its age. Famously, even Lord Kelvin held firm to the view that the Sun was at most a few hundred million years old, despite many of his contemporaries arguing that it was necessarily much older (in fairness, fusion was unknown at the time).↩︎
The quote is from [1, p. 9]. Wollaston’s paper (in the original German) can be read at the Internet Archive. For a more accessible source, see the notes in the Wikipedia article.↩︎
Fraunhofer lines, i.e., the black spectral lines seen when sunlight is split in a spectrometer, formed due to absorption by various atom species in the gaseous state in the Sun’s atmosphere.↩︎
Stokes independently discovered this principle but never claimed any credit [1]. And Foucault had observed both emission and absorption lines of sodium in a flame (the latter by placing a bright arc behind the flame) but never pursued the subject further.↩︎
This was only realised in 1925 by Cecilia Payne-Gaposchkin in her doctoral thesis, which primarily dealt with how the large variations in absorption line strength (from star to star) in stellar spectra was due to differing temperatures and not chemical composition [4]. Incidentally, her data also demonstrated a clear abundance of hydrogen and helium in the sun, but she did not quite believe her own analysis on that point, writing “the enormous abundance derived for these elements in the stellar atmosphere is almost certainly not real”.↩︎
BTU, British thermal unit, is a non-SI unit that is no longer used in science but still finds use in the fossil fuel sector. Converting from it to kWh is not straightforward (depends on whether the kWh was produced from combustible or non-combustible generation method). The International Energy Agency recently updated its recommended conversion factor. ↩︎

About

Mon, 20 Feb 2023 18:00:00 +0100

Welcome to my corner of the web! My name is Taha Ahmed and I’m the author of this site.

I like the idea of giving back to the community by building on/with free software and free knowledge in general, and as such I take every opportunity to promote open science, open data, and open source.¹ All of which is made possible by the open web.²

I maintain many self-hosted services, here are some I would like to mention:

my linkblog where I share interesting tidbits I’ve encountered while browsing the web (sometimes with commentary). You can subscribe to it using its RSS feed.
my self-hosted Gitea instance.
my public Shiny server. If you work with UV-Vis spectroscopy of semiconductors, check out my Tauc fit evaluator. I have written a how-to install Shiny, but that post should really be updated…
a few RStudio Server instances (primarily for my own use). I have an old write-up on that setup (they are now run in LXC containers managed entirely via Ansible playbook, so the config described in that post is woefully out-dated…).
a JupyterHub instance for my own use, which I’ve setup with Python, Julia and R kernels.

The site header contains links to some of my web profiles (I try to use only libre services, but it’s a work in progress :-), and here are some more of them that did not make it into the header:

The source code of this blog (including that of the blog posts themselves) is hosted on codeberg.org.

I have chosen not to use a regular comment system for this blog. Instead I’d like to encourage you, dear reader, to use hypothes.is which effectively provides an annotation layer for the web. You’ll see a sidebar to the right — open it to annotate! Please note that I have no control over the content of these annotations, it’s up to you and other web users whether to enable this annotation layer or not. It’s possible to receive all annotations made on this blog by subscribing to this customised hypothes.is stream (Atom feed).

You may of course also send me comments via email or any of the channels listed above.

See, for example: Alan Jacobs, Tending the Digital Commons: A Small Ethics toward the Future. The Hedgehog Review: Vol. 20 No. 1 (Spring 2018). ↩︎
Interested in the subject? A couple of excellent essays, both by Parimal Satyal: Rediscovering the small web, and Against an increasingly user-hostile web. ↩︎

Notes on Kelvin probe microscopy

Fri, 04 Nov 2022 20:00:00 +0100

I was fortunate to get to familiarise myself with our research group’s state-of-the-art Kelvin probe for a graduate research project I was part of. On a very broad level, The Kelvin probe (KP) is a method that allows the experimenter to correlate the Fermi level of the sample to the absolute reference electrode scale, which is a useful thing for solar cells and many other kinds of materials.

In this blog post, which is a distillate of a longer note I wrote in my electronic labnotebook a while ago, I will give a short introduction to KP from a beginning experimenter’s perspective. Note that this post is limited to Kelvin probe microscopy, and does not discuss Kelvin probe force microscopy (except perhaps in passing).

At the end of this post we have collected some links to other KP resources on the web.

History of the Kelvin probe technique

The Kelvin probe method was first demonstrated in 1898 by Lord Kelvin [1]. The two plates forming a capacitor were repeatedly moved by hand. Kelvin and his contemporaries used the instrument mainly to study the “contact electrification” of metals.

The next major improvement was in 1928 when Zisman developed a way to automatically vibrate one of the metal plates (“for hours”) by way of an air blast pointed at a piano wire [2]. The work was part of Zisman’s masters thesis at MIT, and published a few years later.

I think the study of the operation of the forebears of scientific instruments can be very helpful in understanding their modern counterparts. So, here I will devote a few lines to go through the operation of an early version of the Kelvin probe (mainly based on Zeisman’s paper).

Figure 1: Electric circuit diagram I made of a manually operated Kelvin probe based on a sketch in [2]. The “tip” (labelled A) and the specimen (B) form a capacitor. The capacitor is connected to a potentiometer (a variable resistor and battery) via switch S₂, and to an electrometer via switch S₁. This diagram represents the most common way of measuring the contact potential difference (CPD) before the introduction of Zisman’s vibrating plate. We can use this diagram to better understand the operation of a modern Kelvin probe.

The experimenter can monitor the charge on A using the electrometer (given that S₁ is closed, of course). If electrode A is moved away from B (i.e., increasing the distance), the capacity of the capacitor AB will decrease and an electric charge will be transferred to the electrometer causing its readout to change.

Now let’s assume that we keep S₁ open, but close S₂ and set the potentiometer so that the CPD is precisely counter-acted. We would then have neutralised the charge between A and B, and if we now open S₂ and close S₁ again, the electrometer should no longer display a changing readout when we move the two plates away from each other.

So by gradually increasing the distance between A and B, while determining the CPD and electrometer readout for each distance (an admittedly laborious process), we could note at which applied potential difference the charge displays a minimum (ideally it becomes zero) and that’s our CPD. Of course, happening to distance the plates precisely at the zero-charge point is quite unlikely, so a slightly better method is to plot the electrometer readout against the applied potential difference – the intercept on the axis along which we plot the potential difference will then be the CPD.

The major achievement of Zisman’s vibrating setup was that it made away with the whole process of moving the plates by hand, by exploiting the fact that periodically (and uniformly) vibrating capacitor plates will give rise to a steady AC current whose amplitude will depend on the capacitance (i.e., charge) on said capacitor, with both the capacity and the AC current being zero at the point where the applied potential difference balances the CPD.

Zisman’s setup also replaced the electrometer readout with speakers, that emitted a sound until the applied potential difference balanced the CPD, at which point they went silent. This way a measurement merely consisted of balancing the resistance on the potentiometer until no sound was heard, which could be accomplished by the experimenter in a matter of seconds.

The contact potential difference (CPD)

The existence of a contact potential difference between two dissimilar metals was discovered by Volta in 1801 (“Volta’s electromotive force of contact”, according to [1]), you will therefore sometimes find authors preferring to call it the Volta potential difference.

Figure 2: Left: two different metals in proximity, no electrical circuit between them. Center: now with an external electrical circuit connecting them. The two surfaces become equally and oppositely charged. Right: introducing a variable electromotive force (EMF) in the external circuit allows us to nullify the electric field, effectively measuring it. Image reproduced from a KP Technology manual.

The left panel of figure 2 shows two different metals in close proximity, but isolated (without electrical contact between them). Dissimilar metals will have different work functions ($\phi_1$, $\phi_2$) and their Fermi levels are thus not aligned ($\varepsilon_1$, $\varepsilon_2$). This system acts like a parallel plate capacitor, where the capacitance is equal to the ratio of the amount of charge to the potential difference,

\[\begin{equation} C = \frac{Q}{\Delta V} \end{equation}\]

and the capacitance is also inversely proportional to the distance between the two surfaces

\[\begin{equation} C = \varepsilon_\text{r}\varepsilon_\text{0}\frac{A}{d} \end{equation}\]

In the center panel of figure 2 the metals are now connected via an external circuit, causing electrons to flow from the metal with the smaller work function to that with the larger work function. This causes the smaller work function metal to charge positively and the larger work function metal to charge negatively. This charge-separation across the metal-metal interface is driven by the work function difference, and the charge-separation buildup stops once the created electric field (due to the charge-separation) compensates for the work function difference. At this equilibrated state, the Fermi levels of both metals are equal ($\varepsilon_1 = \varepsilon_2$), and the so-called contact potential, $V_\text{c}$, between the electrodes exactly equals the work function difference.

The right panel of figure 2 shows that by introducing a variable EMF in the external circuit, we can find the potential at which the contact potential is nulled (i.e., where $V_\text{b} = -V_\text{c}$). This is called the backing potential (or sometimes the counter potential) and forms the basis for the operation of a Kelvin probe.

The work function

As we have already seen, Lord Kelvin demonstrated that a potential is generated between two conductors when they are brought into contact (electrons in the material with the lower work function flow to the one with the higher work function). This contact potential difference (CPD) depends on the work function of the two materials in contact:

\[\begin{equation} V_\text{CPD} = \frac{\phi_\text{tip} - \phi_\text{sample}}{e} = \frac{\Delta\phi}{e} \end{equation}\]

where $\phi_\text{tip}$ is the work function of the tip of the Kelvin probe, and $\phi_\text{sample}$ is the work function of the sample.

So, what is the work function? In short, the work function is the amount of energy needed to release electrons from the material. More stringently, the work function is defined as the minimum work that is required to extract an electron from within the sample to a position just outside the sample (carrying zero kinetic energy).

“Just outside the sample” is far enough from the sample surface that the electron is outside the surface dipole layer (which translates to a few nanometers). For an uncharged sample the energy of “just outside the sample” and very far from it (i.e., “at infinity”) are the same, and equivalent to the vacuum level, $E_\text{VAC}$. Note that there is a possible source of confusion here, as some authors denote the “just outside the sample” position as the “vacuum level” and the “at infinity” position as the “vacuum level at infinity” (this latter being the vacuum level intended when comparing against other reference electrodes). In the following, I intend to call the position “just outside the sample” for just that, and the “at infinity” position for the “vacuum level”, i.e., $E_\text{VAC}$ (please correct me if you discover any inconsistencies!). As we have noted, for an uncharged sample there is effectively no difference in energy between these positions. But most samples will be charge (here charged does not mean that the whole material carries an effective charge, which is a physical impossibility) but rather that the surface in contact with the probe is charged, which is the case for many semiconductors, e.g., crystals with a non-centrosymmetric symmetry will have one direction that is more polar than the others (cf. piezoelectric crystals). Anyway, the point is that for most real-world materials, the point at which the vacuum level is defined actually matters.

The work function for an uncharged sample is comprised of the chemical work and the electrostatic work necessary for transporting the charged electron through the dipole layer at the surface: [3]

\[\begin{equation} \phi = -\mu_\text{e} + e\chi = -(\mu_\text{e} - e\chi) \end{equation}\]

where $\mu_\text{e}$ is the chemical potential (i.e., the chemical work to transfer the electron from infinity into the sample, hence the minus sign), $e$ is the electron charge, and $\chi$ is the so-called dipole or surface potential (this is the potential drop between just inside the sample to just outside of it). Note that I have chosen to use lower-case phi ($\phi$) to symbolise the work function rather than upper-case Phi ($\Phi$).

If the sample is charged an additional work component is necessary to transfer the electron to a position infinity far away, namely $e\psi$, where $\psi$ is the Volta potential and is due to surface charges [3].

In a semiconductor, the work function can be expressed as a function of the position of $E_\text{VAC}$ and the Fermi level of the material, $E_\text{F}$, of which the latter in turn depends on the material’s density of states, temperature, carrier density and doping concentration: [4]

\[\begin{equation} \phi = E_\text{VAC} - E_\text{F} \end{equation}\]

[The work function, $\psi$,] is measured as the combination of the two components, which cannot be experimentally separated. The dominant one, i.e., the bulk component, corresponds to the chemical potential that derives from the electronic density and density of states in the solid. Its calculation involves the determination of the difference in energy between the $N$ and $N-1$ electron system. The surface component, also known as surface dipole component, corresponds to an additional potential step (positive or negative) that originates with a redistribution of charges at the surface of the solid. This component is inherent to the surface of the solid and exists only at the solid–vacuum interface. — [4]

The surface of the solid thus represents an energy barrier that the electron at the Fermi level must overcome to reach the vacuum level (free space). This also means that $\phi$ and $E_\text{VAC}$ are meaningful only in relation to the surface of the solid material.

The redistribution of charges at the surface of a material, which gives rise to the surface dipole, takes on several possible mechanisms. The generic mechanism for metal surfaces is the “spill out” of electrons into vacuum. Deep in the metal, the electron density along the $z$ direction perpendicular to the surface is constant (at least in a simple jellium-type model), but as the lattice abruptly terminates at the surface, electrons tunnel out of the solid over some small distance (Ångströms), creating a negative sheet of charges outside the solid and leaving a positive sheet of uncompensated metal ions in the surface and sub-surface atomic planes. This double sheet of charges creates a potential step that increases the electron potential just outside the surface, effectively raising $E_\text{VAC}$ and [$\psi$]. This electronic redistribution and potential step depends sensitively on the atomic arrangement at the surface of the solid […] — [4]

This makes Kelvin probe microscopy an exceptionally sensitive indicator of surface conditions and affected by absorbed or evaporated layers, surface charging, oxide layer imperfections, surface contamination, and more.

Principle of operation of a Kelvin probe

It appears to me that most contemporary Kelvin probe’s use the so-called Baikie system (so named after its inventor, professor Iain Baikie, the founder of KP Technology), where the CPD is determined using what is called an “off-null height-regulated” method where the signal generated by the vibrating electrode is measured far from the electric balance point, meaning the generated current amplitude depends linearly on the backing potential, and thus the point where the current amplitude is equal to zero determines the CPD [5]. This significantly reduces noise compared to the traditional method of determining CPD.

The Kelvin probe can be considered a natural reference electrode, and the only absolute reference electrode available to us. Still, to obtain the work function of a sample versus the absolute vacuum level the probe must first be calibrated against a reference material with a known work function.

When the probe is vibrated, a varying capacitance is produced: \[\begin{equation} C_\mathrm{K}(t) = \varepsilon_\mathrm{r} \varepsilon_\mathrm{0} \frac{A}{d(t)} \end{equation}\] where $C_\mathrm{K}(t)$ is the time-varying Kelvin capacitance, $\varepsilon_\mathrm{0}$ is the permittivity of free space, $\varepsilon_\mathrm{r}$ is the relative permittivity, $A$ is the surface area of the capacitor plate and $d(t)$ is the time-dependent separation between the capacitor plates.

Assuming the tip–sample separation can be represented by a periodic sinusoidal displacement $d(t) = d_\mathrm{0} + d_\mathrm{1}\sin(\omega t)$, then the varying capacitance becomes \[\begin{equation} C_\mathrm{K}(t) = \frac{C_\mathrm{0}}{1 + \frac{d_\mathrm{1}}{d_\mathrm{0}} \sin(\omega t)} \end{equation}\] where $C_\mathrm{0}$ is the mean capacity, $\omega$ is the angular frequency of vibrations in radians per seconds, $d_\mathrm{0}$ is the mean distance between tip and probe, and $d_\mathrm{1}$ is the amplitude of tip motion (the ratio $d_\mathrm{1}/d_\mathrm{0}$ is the so-called modulation index). I am going to skip how this ties into the electronic circuit design of the Kelvin probe, and directly proceed to tell you that the instantaneous total surface charge on the tip of the probe evaluates to: \[\begin{equation} Q_\mathrm{S} = C_\mathrm{K} (V_\mathrm{b} + V_\mathrm{c}) \end{equation}\] I am going to skip some more steps in this derivation, and straight away note that this leads to the output signal produced by the Kelvin probe using the “off-null height-regulated” method, namely the peak-to-peak voltage $V_\mathrm{ptp}$. This is then plotted against the backing potential and the intercept of the resultant straight line on the ordinate is the point where $V_\text{b} + V_\text{c} = 0$. Et voilà, we have our CPD, and assuming we can determine or know the work function of the probe tip (the other half of the capacitor), calculating the work function of our sample is only a subtraction away.

When discussing the determination of absolute work function of materials we would be amiss to not mention another method that can do the job, namely photoemission spectroscopy (PES). The main advantage of KP over PES with regards to semiconductor materials is that KP can be performed in the dark, which gives the experimenter a way to avoid the confounding effect of band bending that may occur under illumination [4]. Another advantage is that KP can be done in air, whereas PES requires vacuum (this may or may not be considered an advantage, depending on your vantage point). Work function measurements made in air are affected by the adsorption of atmospheric gasses, such as oxygen [5], which is why modern KP chambers allow purging with nitrogen and humidity control.

If you have read this far you might be interested in the simple R package kptech I wrote which helps you import and plot data from a KPTech Kelvin probe.

Kelvin probe force microscopy (KPFM)

Kelvin probe force microscopy was first proposed by [6], and allows the experimenter to obtain both topographic and potential images of the sample’s surface. It is based on atomic force microscopy (AFM).

The CPD is determined as in a regular KP, and the cantilever deflection (due to electrostatic repulsion, it is not touching the sample’s surface) is normally detected by a photodetector.

Notes and links

Tutorial on Kelvin probe measurements, Rudy Schlaf (University South Florida)
Translation guide for discussing electron energy concepts, Steve Byrnes (University of California, Berkeley)
KP Technology

There’s a bunch of questions on StackExchange related to the work function:

Youtube gives a few hits for “kelvin probe”:

Apparently surface potential of bacteria can be measured using Kelvin probe force microscopy, Journal of Visualized Experiments.
Product demo of a scanning Kelvin probe by BioLogic.
A recorded seminar on Kelvin probe force microscopy of field-effect transistors by Dr Mukherjee, Israel Institute of Technology.
A tutorial on fundamental theory and application of KPFM.
An animation of the operation of the probe in Kelvin probe force microscopy.
An animated presentation on Kelvin probe force microscopy from a course on Nanoscale imaging at Rice University.
A demonstration of a Kelvin probe at the Instytut Fotonowy in Cracow.

References

[1]

W.T. Kelvin, V. Contact electricity of metals, Philosophical Magazine Series 5. 46 (1898) 82–120. https://doi.org/10.1080/14786449808621172.

[2]

W.A. Zisman, A new method of measuring contact potential differences in metals, Review of Scientific Instruments. 3 (1932) 367. https://doi.org/10.1063/1.1748947.

[3]

M. Rohwerder, The kelvin probe technique as reference electrode for application on thin and ultrathin electrolyte films, in: Handbook of Reference Electrodes, 2013: pp. 333–340. https://doi.org/10.1007/978-3-642-36188-3_15.

[4]

A. Kahn, Fermi level, work function and vacuum level, Materials Horizons. 1 (2015) 96. https://doi.org/10.1039/c3mh00043e.

[5]

G. Halek, I.D. Baikie, H. Teterycz, Surface investigation by using a scanning Kelvin probe, in: 2010 International Students and Young Scientists Workshop "Photonics and Microsystems", IEEE, 2010: pp. 24–27. https://doi.org/10.1109/STYSW.2010.5714161.

[6]

M. Nonnenmacher, M. O’Boyle, H.K. Wickramasinghe, Surface investigations with a Kelvin probe force microscope, Ultramicroscopy. 42–44 (1992) 268–273. https://doi.org/10.1016/0304-3991(92)90278-R.

Observing the global energy transition (from over here)

Sat, 01 Oct 2022 18:00:00 +0100

In this post I have collected news and tidbits, some with my commentary, on planned or installed projects from around the world that further the transition from our carbon-based energy system to a photon-based. This includes projects on renewable chemical feedstocks (hydrogen, ammonia, etc.) as well as their generation by renewable methods and the infrastructure developments underpinning such projects.

These notes are made from my personal vantage point, which is necessarily limited, and without any claim to be authoritative or even objective. The notes are not presented in any particular order.

Even though each tidbit can certainly feel small and inconsequential in the moment, every small step towards weening ourselves off the carbon-based economy counts. And judging by most outlooks, the pace of the energy transition will only accelerate even more during the remainder of this decade.

This is necessary if we are to even stay within reach of the 1.5℃ climate target, according to the World Energy Transitions Outlook 2023 report from IRENA. Most worryingly, the pace is not enough, as the gap between what has been achieved and what is required continues to grow.

Specifically, by 2050 the production of electrolysers needs to grow 6000-fold to achieve the 1.5℃ climate target, from 600 MW today (2021) to 3670 GW [1]. This is a faster scale-up than we achieved for solar PV, but it is possible.

Below, please find my collected news clippings organised in the following categories: solar-based power plants, electrolysers (planned or in operation), manufacture of electrolysers, fuel cell vehicles (battery electric vehicles are quickly becoming mainstream, which is great), manufacture of fuel cells, photoelectrochemical plants, and in not so good news.

Solar power plants

Electrolysers

Fuel cell vehicles

Photoelectrochemical plants

A photoelectrochemical pilot plant using concentrated PV at EPFL, described in an academic paper by the Haussener laboratory (DOI: 10.1038/s41560-023-01247-2).

Other renewables

These are not my focus, but if a project catches my eye I might include here.

Masdar (UAE) and NRA (Egypt) agreed at the sidelines of COP27 (the UN climate change conference 2022, in Sharm El Sheikh) to build what would be a massive wind turbine park producing 48 TWh of electricity per year.

Manufacturing of solar PV, electrolysers, wind turbines, batteries, and heat pumps

Global manufacturing capacity for renewable technologies is expanding rapidly. In its report “The State of Clean Technology Manufacturing” the International Energy Agency points to solar PV, electrolysers, wind turbines, batteries, and heat pumps as critical technologies for the energy transition.

ThyssenKrupp’s Chemical & Process Technologies business unit (ThyssenKrupp Uhde) is ramping up its production facilities in order to produce electrolysis cells with a combined rated power reaching $\SI{1}{\GW}$ per year. The cells use coatings developed by De Nora that allow system efficiency up to 80%. According to ThyssenKrupp, over $\SI{10}{\GW}$ of its electrolysers are already operating worldwide across 600 projects.

Hyundai Motor Group (South Korea) intends to invest about $6 billion in total until 2030 to develop a factory to produce 700 000 fuel cell stacks annually starting from that year (2018-12).

And perhaps not so good news

Background

We have, over the last century, built ourselves a global energy conversion system (and a global economy) entirely predicated on the ever-more greedy extraction of fossil fuels. Frankly then, it is no surprise that we would sooner or later run afoul of our biosphere’s physical boundaries.

The energy transition will require a fundamental realignment of the chemical industry, an industry that so far the has relied primarily on oil, coal and gas as for both its energy and its raw material feeds. And the same can be said for the world’s power grids, who might also need to change from their current centralised design to one that allows more distributed power generation.

Hyping the energy transition, Quillette, R. Bryce (2022)

Whatever path of transition that we decide to take, we must be careful not to invest in flukes, because inevitably the energy required to fuel this transition will be mostly from non-renewable sources.

A short primer on the greenhouse effect and climate change

The basic physics of the greenhouse effect is simple. The planet’s surface and atmosphere can be considered a system in steady state (incoming energy equals outgoing energy), with space being the ultimate heatsink.¹ The incident solar light peaks in the visible region (500 nm) of the electromagnetic spectrum, and is nearly balanced by infrared radiation emitted from high-altitude atmospheric gases (H₂O, CO₂, etc.). The atmosphere essentially acts like multiple bandstop filters that blocks most of the electromagnetic spectrum from reaching the surface, except for visible light and some ultraviolet and near-infrared, as well as certain bands in the IR range.

Greenhouse gases (GHG) in the upper troposphere form a spectral window that allows the planet surface and lower layers of the atmosphere to radiate excess energy with wavelengths in the IR into space. This requires emissions from the surface characteristic of temperatures roughly 10℃ to 15℃ warmer than the so-called grey body equilibrium surface temperature² [2]. An increase of GHG makes the atmosphere more opaque at IR wavelengths, thus reducing the planet’s heat radiation to space, and causing the planet to warm until the equilibrium of incident solar light to planetary heat radiation is restored (now at a higher equilibrium surface temperature) [3]. The importance of the atmosphere and the greenhouse effect to life on Earth cannot be understated. The greenhouse effect is the main reason why our surface temperature is not similar to that of the Moon (had it had an atmosphere).

Careful measurements of ocean heat content over six years (2005–2010) showed that the Earth is absorbing more energy from the Sun than it is radiating back to space, with an inferred mean planetary energy imbalance over that six-year period of (0.58±0.15) W m⁻². It is inferred, because of the two dominant causes of changes to Earth’s energy imbalance, greenhouse gases and changes of atmospheric aerosols, the latter remains unmeasured (on the other hand, the former is measured very precisely) [3].

The long lifetime of CO₂ in the atmosphere³ (300 years) means that even if we deploy large-scale carbon capture and sequestration (which I believe is unfeasible), and even if we manage to arrest or even reverse that increase (which is not happening so far), the benefits will not be felt immediately. Our atmosphere, hydrosphere and biosphere are large and intricately interconnected systems, and the momentum of past emissions means that a global mean temperature rise is now assured. But we can still limit how large that rise will eventually be.

Zero emissions isn’t enough. We need climate repair., Jacobin Magazine, Hudson (2022)

Solar energy collection and conversion

The challenge with energy conversion has always been a game of optimising collection and conversion. For coal, gas and oil collecting it was fairly straight-forward: just identify (and assert control of) the underground or undersea basins where it was naturally amassed. And once the internal combustion engine (ICE) was developed (at least 150 years ago), converting that fossil fuel to work was simple, but with large negative externalities (poor conversion efficency, pollution, noise) that everyone ignored because the fuel was so easy to collect and thus immensely profitable. Before the age of fossil fuels collection was often arduous, back-breaking manual work, and conversion was often even less efficient than with the ICE.

For solar power, collection and conversion works a little differently. The fuel (in the collection step) is no longer materia, but light. And the material itself is no longer broken down in a chemical reaction to produce heat, but rather facilitates (i.e., catalyses) the conversion of photons to electrons while itself remaining inert for decades. And since our primary source of light is the Sun, and sunlight does not arrive in concentrated beams but is rather evenly spread out everywhere, collecting sunlight requires installing photovoltaic (PV) panels on large areas of land⁴ or buildings.

Atmospheric carbon capture: inefficient, but may be unavoidable

We will most likely need to invest in the atmospheric capture of carbon dioxide, [5] not because it is particularly economical or efficient, but simply because it has now become imperative to quickly decrease the atmosphere’s CO₂ concentration.

This does open up interesting avenues for chemists to take that carbon up the value chain to carbohydrates and all kinds of compounds. But making any sort of fuel out of this captured carbon would defeat the whole point of the exercise, in my opinion.

Capturing carbon dioxide on a large scale from the atmosphere is technically possible, but would require massive amounts of renewable power; power which we would perhaps more wisely be using to supplant non-renewable power generation (there is an inherent conflict of priorities here) [5].

A more limited approach is to simply get the CO₂ out of the atmosphere and store it in bedrock or something, which is known as carbon capture and storage (CCS). Some authors have suggested methods that would combine the splitting of water with CO₂(aq) capture using ocean chemistry [6], and others have shown that carbon dioxide emissions could potentially provide 1570 TWh worth of electricity annually [7]. It is probably premature to pin any hope to such schemes.

We should rather commit to reduce and then eliminate our carbon emissions.

Can sucking CO₂ out of the atmosphere really work? MIT Technology Review (2014)

Notes and assorted links

A worldwide database of hydrogen projects by IEA. Unfortunately requires sign-up with IEA account for access.
https://www.iea.org/energy-system/low-emission-fuels/hydrogen
Surface temperature of the Earth, Physics in a nutshell
https://climate.nasa.gov/vital-signs/global-temperature
Global surface temperature, Wikipedia
Clean power is shaking up the global geopolitics of energy, The Economist (Special report “The new power superpowers”, 2018)
Guide to hydrogen energy law and regulation, CMS Law (2021)
Did a 1912 newspaper article predict global warming?, Snopes (2016)
A 1912 news article ominously forecasted the catastrophic effects of fossil fuels on climate change, Quartz (2016)
Försvinnande energitillgång, Medborgarutredningen, Christerson et al. (2017)
https://www.esa.int/Enabling_Support/Space_Engineering_Technology/SOLARIS This is pretty cool, space-based solar! Of course, it is still far from being implemented, but the fact that the EU, USA and China are all investigating it is either promising or just politics. Time will tell.

References

[1] Odenweller, A., Ueckerdt, F., Nemet, G.F., et al., Probabilistic feasibility space of scaling up green hydrogen supply, Nature Energy, v. 7, n. 9, pp. 854–865, Sep. 2022.

[2] Kopp, G., “5.02 - Earth’s incoming energy: The total solar irradiance,” In: Liang, S. (ed), Comprehensive Remote Sensing, Oxford, Elsevier, pp. 32–66, 2018.

[3] Hansen, J., Sato, M., Kharecha, P., et al., Earth’s energy imbalance and implications, Atmospheric Chemistry and Physics, v. 11, n. 24, pp. 13421–13449, 2011.

[4] Lee, N., Grunwald, U., Rosenlieb, E., et al., Hybrid floating solar photovoltaics-hydropower systems: Benefits and global assessment of technical potential, Renewable Energy, v. 162, pp. 1415–1427, Dec. 2020.

[5] Keith, D.W., Holmes, G., Angelo, D.S., et al., A process for capturing CO$_2$ from the atmosphere, Joule, v. 2, n. 8, pp. 1573–1594, Aug. 2018.

[6] Rau, G.H., Willauer, H.D., Ren, Z.J., The global potential for converting renewable electricity to negative-CO$_2$-emissions hydrogen, Nature Climate Change, v. 8, n. 7, pp. 621–625, Jul. 2018.

[7] Hamelers, H.V.M., Schaetzle, O., Paz-García, J.M., et al., Harvesting energy from CO$_2$ emissions, Environmental Science & Technology Letters, v. 1, n. 1, pp. 31–35, Jan. 2014.

Although net energy is zero, net entropy is not zero and is the driving force.↩︎
The black body temperature of the Earth can be calculated, and at an average distance from the Sun of 1 AU it yields a globally averaged equilibrium temperature (at the top of Earth’s atmosphere) of 278 K (5℃). But due to GHG in the atmosphere, the Earth cannot be considered a black body, but rather a grey body, and the actual globally averaged equilibrium temperature becomes 290 K (17℃).↩︎
Apart from having a long lifetime, CO₂ in the atmosphere works on a time delay. It takes about 10 years to achieve its full warming effect.↩︎
Surface land or surface water, really. There are several benefits of putting PV on natural or man-made dams and lakes, in particular when combined with existing hydropower [4], in that the panels are cooled by the water and simultaneously reduce water loss via evaporation. PV can also be combined with other land uses, such as agriculture.↩︎

Spectrochemical data of the dyes EBT, MB, MO and RB5

Sat, 06 Aug 2022 18:00:00 +0000

In this post we share some experimentally recorded UV-Vis-NIR spectra and absorption coefficient ¹ curves of the dyes eriochrome black T (EBT), methylene blue (MB), methyl orange (MO), and reactive black 5 (RB5) in aqueous solution. For EBT, MB and MO we also report the absorption coefficient in the presence of 10% EtOH (by volume).

This post also includes a small study on pH and conductivity at room temperature of the same dyes.

This post is likely of interest for research into photocatalytic dye degradation, and the dataset is shared in this blog post’s repo on Codeberg (as R dataset files).

Spectral absorption coefficients

To make the absorption coefficient curves (figure 2) less noisy, we will start by calculating the average absorbance value inside a wavelength band $\SI{10}{\nm}$ wide centered on the main absorbance band of each dye. This band is shown as a shaded yellow area in figure 1 below.

Unfortunately, I’ve lost the original data for RB5 in water, but we still have the plot of those measurements, which is shown as an inset. However, the shape of the absorption coefficient curve of RB5 in only water can be assumed to very closely follow that of the dye in the presence of alcohol (figure 1).

Figure 1: UV-Vis spectra of EBT, MB, MO, and RB5 at varying concentrations in aqueous solution with or without the addition of a small amount of alcohol.

EBT is the only dye that shows isosbestic points between the spectra of the different solvents. With the exception of MB, the other dyes show only a slight overall intensity decrease when introducing alcohol into the solvent. MB demonstrates a very obvious, yet subtle, intensity change in its main shoulder when adding EtOH to the solvent, which I have discussed my paper [1].

EBT is also the only dye (of the four tested) that shows spectral changes dependent on concentration, which indicates formation of new molecular species at higher concentrations. In contrast, the other dyes are all well-behaved as the concentration changes, in the sense that the position and shape of their peaks are independent of concentration, which indicates that the dye solution contains the same molecular components irrespective of concentration (within the studied range).

From the absorption spectra of a series of concentrations and by applying the Beer-Lambert law we can calculate the absorption coefficients for the measured wavelength range.

\[\begin{equation} A = \epsilon lc \;\Longleftrightarrow\; \epsilon = \frac{A}{lc} \end{equation}\]

Given that absorbance is unit-less, and with $[l]=\si{\cm}$ and $[c]=\si{\mol\per\litre}$, then the unit of the molar absorption coefficient is $[\epsilon]=\si{\litre\per\mole\per\cm}$ which is how it is usually given.

We will calculate the absorption coefficient $\epsilon$ for each wavelength step, by dye and solvent.Effectively we are calculating one linear fit (abs vs conc) for each wavelength step.

# this chunk calculates linear fits for every single wavelength step, which is time-consuming
if (file.exists(here::here("assets/abscoeff.rda"))) {
   abs.coeff <- LoadRData2Variable(here::here("assets/abscoeff.rda"))
} else {
   # replace both MeOH and EtOH strings with the more generic ROH for easier looping
   df.dye %<>%
      mutate(solution = ifelse(solution == "H2O+MeOH", "H2O+ROH", solution)) %>%
      mutate(solution = ifelse(solution == "H2O+EtOH", "H2O+ROH", solution))
   # convert df.dye$conc from factor to numeric
   df.dye$conc <-
      df.dye$conc %>% as.character() %>% as.numeric()
   # loop over each spectra by dye and solution
   dyes      <- unique(df.dye$dye)
   solutions <- unique(df.dye$solution)
   wl.steps  <- unique(df.dye$wavelength)
   # hold the results of the looping in a new dataframe
   abs.coeff <-
      data.frame(
         wavelength = rep(wl.steps, length(dyes) * length(solutions)),
         dye = "",
         solution = "",
         k = NA,
         m = NA,
         rsq = NA,
         adj.rsq = NA,
         concentrations = "") # uM
   i <- 0
   # NOTE: this loop is time-consuming, so I've intentionally made it chatty
   for (d in 1:length(dyes)) {
      message("Dye: ", dyes[d])
      if (dyes[d] == "RB5") {
         # we only have H2O+ROH spectrum for RB5, so we need to reset solutions so the next inner loop doesn't    fail
         solutions <- "H2O+ROH"
      }
      for (s in 1:length(solutions)) {
         message("Dye: ", dyes[d], " :: Solution: ", solutions[s])
         # varying conc of a particular dye and solution
         for (w in 1:length(wl.steps)) {
            # to keep track of the current row in abs.coeff, we will use a counter var
            i <- i + 1
            message("[", i, "] ", dyes[d], " :: ", solutions[s], " :: ", wl.steps[w], " nm")
            # temporary dataframe which we will use to calculate linear fits (abs coeff) for each wl step
            intensity.by.wl <-
               df.dye %>%
               filter(dye == dyes[d]) %>%
               filter(solution == solutions[s]) %>%
               filter(wavelength == wl.steps[w]) %>%
               select(sampleid, wavelength, intensity, conc, dye, solution)
            # now we can calculate abs coeff for this solution and dye at the current wavelength
            this.fit <- lm(data = intensity.by.wl, formula = intensity ~ conc)
            # save the concentrations for reference
            abs.coeff$concentrations[i] <-
               paste(sort(intensity.by.wl$conc), collapse = ", ")
            # save R-squared and adjusted R-squared
            abs.coeff$rsq[i] <- summary(this.fit)$r.squared
            abs.coeff$adj.rsq[i] <- summary(this.fit)$adj.r.squared
            # save coefficients of linear fit
            abs.coeff$m[i] <- this.fit$coefficients[1]
            abs.coeff$k[i] <- this.fit$coefficients[2]
            # solution and dye
            abs.coeff$solution[i] <- unique(intensity.by.wl$solution)
            abs.coeff$dye[i] <- unique(intensity.by.wl$dye)
         }
      }
   }
   # adjust units of abs coeffs to L mol-1 cm-1 (from L umol-1 cm-1)
   abs.coeff$k <- 1E6 * abs.coeff$k
   abs.coeff$m <- 1E6 * abs.coeff$m
   # and last, because we are missing H2O-only spectrum for RB5, remove the empty rows in abs.coeff
   # (faster to clean-up in this way than to rewrite this entire loop)
   abs.coeff <- abs.coeff %>% filter(!is.na(k))
   # save abs.coeff
   save(abs.coeff, file = here::here("assets/abscoeff.rda"))
}

Figure 2: Spectral absorption coefficients in aqueous solution for Eriochrome Black T, methylene blue, methyl orange, and reactive black 5, as calculated from the absorbance–concentration series shown in figure 1.

Experimental notes

Preparation of dye stock solutions

Figure 3: Chemical structures of the dyes, as commonly given.

Weighed up EBT and MB powder. Used a Mettler AT261 DeltaRange balance at the XPS room (Ångström bldg 3, level 3). The balance has a readability of $\SI{0.01}{\mg}$ (cf. the balance’s technical specs), but to make use of it the operator has to press a button (otherwise you get the lower, but still respectable, $\SI{0.1}{\mg}$ readibility), but I only realised this after I had weighed the EBT powder. But at least the MB measurement used the full sensitivity range of the balance.

Measured weight of EBT was $\SI{46.3}{\mg}$ (error with respect to calculated weight +0.35%). Measured weight of MB was $\SI{32.01}{\mg}$ (error with respect to calculated weight +0.08%). Measured weight of MO was $\SI{32.84}{\mg}$ (error with respect to calculated weight +0.33%)

All measurements were done using disposable polysterene weighing boats. The MB powder showed some static charge, so I used an anti-static gun on a weighing boat which decreased the “jumpiness” of the particles.

The EBT was likely stored a long time, but the MB and MO were from new, sealed containers.

Both EBT and MB were easily dissolved in water. EBT caused a lot of air bubbles at the solution-air interface at the highest concentrations. MO required ultra-sonication for $\SI{10}{\minute}$ to dissolve all particles.

Mixing alcohol into the aqueous dye solutions

For EBT and MB, added ca $\SI{2}{\milli\liter}$ of dye solution using disposable plastic pipette, then added $\SI{200}{\micro\liter}$ of EtOH with autopipette and agitated the solution until completely mixed.

For MO, added precisely $\SI{1.8}{\milli\liter}$ of dye solution using an electronic auto-pipette, then added $\SI{200}{\micro\liter}$ of EtOH with auto-pipette and mixed the solution.

For RB5, added ca $\SI{2}{\milli\liter}$ of dye solution using disposable plastic pipette, then added $\SI{200}{\micro\liter}$ of MeOH with auto-pipette and mixed the solution.

In all cases, the mixing was done directly in PMMA macro cuvettes ($SI{1}{\cm}$ path length).

Configuration of the light-source and spectrometer

Spectrophotometer: OceanOptics HR2000 spectrometer. Light source with both deuterium and halogen lamps. Light transmitted from lamp to cuvette and from cuvette to spectrometer using OceanOptics UV-Vis fibre-optic cables.

For all experiments reported here we used both the deuterium and tungsten lamps of the OOHR2000 spectrometer, giving us an effective wavelength range approximately $\SIrange{260}{1000}{\nm}$ (the spectrum of this light is shown below).

Figure 4: Spectrum of deuterium (contributes UV) and tungsten (Vis) lamp measured after transmission through approximately of fibre-optic cable and a decimetre of air. The sharp dip at $\SI{950}{\nm}$ is caused by absorption in the fibre-optic cable.

Figure 5: UV-Vis transmittance spectrum of our fibre-optic cable, from calibration experiments courtesy of OceanOptics.

Small study into the pH and conductivity of aqueous dye solutions

I measured pH and conductivity simultaneously for all dye stock solutions, after allowing the solutions to equilibrate with air for at least two hours. The pH and conductivity measurements were done using a handheld Mettler Toledo SevenGo Duo pro™ SG78 pH/ion/conductivity meter.

sampleid	solute	C/µM	V/mL	pH	T(pH)/°C	σ/(µS/cm)	T(σ)/°C
H01AA	EBT	100.0	1000	5.881	19.0	34.60	19.1
H01AB	EBT	50.0	500	5.908	19.1	18.56	19.4
H01AC	EBT	20.0	250	5.901	19.0	9.37	19.3
H01AD	EBT	10.0	500	6.254	18.9	6.17	18.9
H01AE	EBT	5.0	200	6.096	19.0	5.34	19.2
H01AF	EBT	2.0	250	6.169	18.9	3.44	19.2
H02AA	MB	100.0	1000	5.993	18.7	10.12	19.2
H02AB	MB	50.0	500	5.847	18.9	7.31	19.0
H02AC	MB	20.0	250	6.078	19.1	4.52	19.2
H02AD	MB	10.0	500	5.938	18.9	3.67	19.0
H02AE	MB	5.0	200	5.866	18.9	3.08	19.0
H02AF	MB	2.0	250	5.812	18.9	2.46	19.3
H02AG	MB	1.0	250	5.964	18.7	1.72	19.4
H02AH	MB	0.5	200	6.145	18.9	1.86	19.0
H03AA	MO	100.0	1000	NA	NA	10.45	22.0
H03AB	MO	50.0	500	NA	NA	6.99	22.2
H03AC	MO	20.0	250	NA	NA	4.49	22.3
H03AD	MO	10.0	500	NA	NA	4.53	22.1
H03AE	MO	5.0	200	NA	NA	4.63	21.7
H03AF	MO	2.0	250	NA	NA	3.22	22.7

The measured pH of deionised (DI) water at the same experimental conditions was $\num{6.024}$ and its conductivity was $\SI{2.35}{\micro\siemens\per\cm}$. Deionised water in contact with air will eventually reach an equilibrium pH 5.7 (the time it takes to reach equilibrium depends on the volume, the shape of the vessel, stirring or not, etc., but is generally on the order of a few hours).

The observed pH values for the different solute concentrations hints at a markedly different acid-base chemistry for EBT and MB, respectively. The following plot shows the change in electrolytic conductivity with concentration for EBT, MB, and MO (the dashed line is the measured pH of DI water, which for the purposes of this visualisation lacks a defined concentration):

Links and notes

Isosbestic point, Wikipedia
Equilibrium pH of deionised water in contact with air, Chemistry SE
pH and composition of carbonic acid solutions, Wikipedia
Difference between distilled and deionised water, ResearchGate
Methylene blue, NIH
Electrolytic conductivity, Wikipedia

Session info

## Linux 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64

## R version 4.1.3 (2022-03-10)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.3 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=sv_SE.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] oceanoptics_0.0.0.9004 common_0.1.2           here_1.0.1            
##  [4] ggrepel_0.9.1          magick_2.7.3           cowplot_1.1.1         
##  [7] ggplot2_3.3.6          knitr_1.39             dplyr_1.0.10          
## [10] git2r_0.30.1           conflicted_1.1.0      
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.9       highr_0.9        bslib_0.4.0      compiler_4.1.3  
##  [5] pillar_1.8.1     jquerylib_0.1.4  tools_4.1.3      digest_0.6.29   
##  [9] jsonlite_1.8.0   evaluate_0.15    memoise_2.0.1    lifecycle_1.0.3 
## [13] tibble_3.1.8     gtable_0.3.0     pkgconfig_2.0.3  rlang_1.0.6     
## [17] cli_3.4.1        DBI_1.1.3        yaml_2.3.5       blogdown_1.10   
## [21] xfun_0.31        fastmap_1.1.0    withr_2.5.0      stringr_1.4.0   
## [25] generics_0.1.3   vctrs_0.5.1      sass_0.4.2       rprojroot_2.0.3 
## [29] grid_4.1.3       tidyselect_1.2.0 glue_1.6.2       R6_2.5.1        
## [33] fansi_1.0.3      rmarkdown_2.16   bookdown_0.27    farver_2.1.1    
## [37] magrittr_2.0.3   scales_1.2.0     htmltools_0.5.3  assertthat_0.2.1
## [41] colorspace_2.0-3 labeling_0.4.2   utf8_1.2.2       stringi_1.7.8   
## [45] munsell_0.5.0    cachem_1.0.6     crayon_1.5.1

## Commit:  5ea2dc4e89159b2773568a88901ec0310004dd71
## Author:  taha@luxor <taha@chepec.se>
## When:    2023-10-14 19:08:20 GMT
## 
##      Published!
##      
## 10 files changed, 782 insertions, 8 deletions
## README                     | -8 +  0  in 1 hunk
## assets/EBT.rda             | -0 +  0  in 0 hunk (binary file)
## assets/MB.rda              | -0 +  0  in 0 hunk (binary file)
## assets/MO.rda              | -0 +  0  in 0 hunk (binary file)
## assets/RB5.rda             | -0 +  0  in 0 hunk (binary file)
## assets/abscoeff.rda        | -0 +  0  in 0 hunk (binary file)
## assets/dyes.rda            | -0 +  0  in 0 hunk (binary file)
## assets/pH-conductivity.csv | -0 + 20  in 1 hunk
## assets/solutions.csv       | -0 + 13  in 1 hunk
## index.Rmd                  | -0 +749  in 1 hunk

## Untracked files:
##  Untracked:  index.Rmd.lock~

References

[1] Ahmed, T., Edvinsson, T., Optical quantum confinement in ultrasmall ZnO and the effect of size on their photocatalytic activity, The Journal of Physical Chemistry C, v. 124, n. 11, pp. 6395–6404, Mar. 2020.

Also known by the outdated term attenuation coefficient.↩︎

Sources and license

Mon, 13 Jun 2022 18:00:00 +0100

Unless otherwise noted, text and figures on this site are licensed under Creative Commons BY-SA v4.0.

Code is licensed under GPL v3, with the source code of this site available at Codeberg. Each blog post has its own git repository (using so-called git submodules) inside that main repository.

Feel free to comment on this blog using hypothes.is (available from the right sidebar). If you have a more technical comment, please feel free to open an issue in either the main blog repo or in a specific blog post’s repo.

Using Python and RSS to play Youtube channels on the desktop

Fri, 20 May 2022 20:00:00 +0100

For a long time I was never attracted to the concept of youtube “channels”, being put off by the required buy-in to the youtube ecosystem and by their awful web player, unskippable ads, and more.

Until one day I realised that I could subscribe to youtube channels using my own RSS reader, and without the need for any google account. With that, my interest in youtube channels experienced a resurgence, and within a few months, my RSS reader listed over a 100 video feeds.

Unfortunately, although most RSS readers are excellent for following mostly text-based content, the experience of watching videos leaves something to be said about ease of use. This is how a typical session could look like in TinyTinyRSS, with all video feeds organised into a single category, i.e., a single list):

Your browser does not support embedded webm video, but you can download the video here

In this post I will describe the simple tool that I have unimaginatively named ttrss-video-mpv and which I glued together from the TinyTinyRSS API, a Python library to access said API, and a few lines of bash to tie it into my desktop environment of choice, i3wm.

RSS is awesome!

But first, a short primer on why RSS is awesome. RSS is just XML (that is, structured data, no styling) which makes it easily machine-readable and easy to let your reader/client software apply whatever styling/layout suits you best. This is valuable for anyone who wants to control their viewing/reading experience to their own needs or likes.

RSS is a de facto standard syndication system for websites (it has been around since before social media).

All of this makes RSS accessible and useful. I don’t recall where I read the following, so I hope I am not mischaracterising here, but in essence, an RSS feed is just another way to publish a website:

HTML (plus CSS, JS, etc.) + HTTP → website
XML + HTTP → RSS feed

RSS is an umbrella term, and encompasses several different versions of both the RSS and Atom formats. In this post I’ll simply use the label “RSS”, and by that mean both Atom and RSS.

All websites should publish RSS feeds!

Let’s round off this ode to RSS with the unimitable words of Felicia Day:

RSS is a way to consume a lot of information very quickly, and store it in nice categories if you miss it. So I can catch up with a small blog’s output at the end of the week and, if I so choose, read every article easily in one sitting. You think on Friday I’m gonna go browse that same site’s Twitter feed on their page (digging through all the messy @ replies) and see what they did that week?! Or go to their Facebook page that is littered with contests? No way dude, I’m too busy for that!

ttrss-video-mpv

If you just want to see the code, it’s available on codeberg.

And here is how it looks in action, when used with the rofi application launcher on the i3 window manager:

Your browser does not support embedded webm video, but you can download the video here

This way, following your youtube channels becomes almost as easy as following any other stream.

Note that this tool only works for the TinyTinyRSS feed reader, and assumes you have installed bash, python, mpv and youtube-dl.

Folllowing non-youtube video channels is straight-forward, assuming they offer an RSS feed, thanks to the massive list of websites supported by youtube-dl.

Ideas for future development?

Could we achieve similar functionality on Android devices? Today, on Android, I watch videos by browsing the TTRSS app and opening videos in NewPipe.

But creating a functionally similar playback of all videos in Android would require developing our own app for starters (and I’ve no experience with that) and a way to access the TinyTinyRSS API plus some way to use youtube-dl.

Notes and links

I created the screencast GIFs using peek and screenkey. My linkblog has a list of screen capture software as well as a short note on how to subscribe to Youtube channels using RSS.

To just watch a random collection of recent Youtube videos, there’s Astronaut, code repo here and NY Magazine article here.

Use RSS to follow new content

RSS makes the web tolerable again, Jamie Adams, 2019
How to read RSS in 2020, Laura Kalbag, 2020
Consuming News, Amolith, 2021
Turn to RSS Feeds to Regain Control of the World Wide Web, Roy Schestowitz, 2012. HackerNews thread
Reports of RSS’s demise have been greatly exaggerated

Self-host your RSS reader

Do you self-host an RSS reader? r/selfhosted

Watch youtube using non-google clients (e.g., RSS readers)

Replacing YouTube & Invidious, Amolith, 2021
Youtube RSS, Egg Freckles, 2020
Vivaldi browser comes with Youtube channel RSS subscription built-in
Introduction to web feeds and how to subscribe to Youtube channels using RSS
Consuming YouTube through RSS: Reclaim control of your subscriptions
Drew DeVault uses RSS, an RSS-to-email program along with some filters in his email inbox, and a custom keybinding in his mail client (mutt) that identifies the youtube URL and plays it with mpv.
Listen to any Youtube/Vimeo channels in podcast format

Watch youtube videos via RSS in mpv or youtube-dl

Elfeed + MPV + YouTube, r/emacs
Youtube subscriptions using elfeed & mpv, r/emacs
Use Newsboat rss reader and mpv to subscribe and play bsdnow youtube feed, FreeBSD forums
Youtube without (personalized) recommendations, The Privacy Captain
https://reddit.com/r/rss/comments/j3qn9k/opening_youtube_videos_with_mpv_in_liferea/

What is RSS

Credits

Mon, 09 Aug 2021 18:00:00 +0100

I could not have created this website without the prior work of many others, who generously chose to publish their work on the web and to license it freely.

These are some of the major building blocks of this website:

Hugo, a very fast static site generator.
My own fork of the Hugo theme Ghostwriter.
The R packages blogdown and knitr.
The extremely versatile document conversion tool pandoc.
MathJax, a client-side JavaScript implementation of LaTeX.
Icons from Fork Awesome and Academicons.

The posts on this blog are written in R Markdown, with source code for each post available.

Thanks to pandoc and MathJax, the R Markdown format allows us to use familiar LaTeX notation and a bunch of vital academic-blogging features such as citations, quotes, and footnotes in our blog posts.

Creating a composite gif with multiple gganimate panels

Mon, 02 Aug 2021 20:00:00 +0100

The topic for this blog post is how to create a composite gif made up of multiple gganimate objects, overlaid on top of one another.

This is not fundamentally novel, as these abilities are hinted at in the documentation of both gganimate and magick, but to my knowledge this is the first actual demonstration of a multi-panel composite of overlaid gif animations using gganimate and magick.

My original reason for attempting this animation was to make an eye-catching centerpiece for a poster we presented at the ISF 2021 conference, and since the conference was entirely web-based (thanks COVID-19!) and the figure I was basing it on was created with ggplot2, the idea of creating a gif using gganimate was not far-fetched.

The original, static, multi-panel plot

In the work described in our paper published last year (see citation in figure caption) we tracked a chemical reaction of a colloidal solution of zinc oxide nanoparticles (ZnO NPs) in the presence of an common dye (methylene blue, MB) under simulated solar illumination using UV/Vis absorbance spectroscopy.

Figure 1: Reproduced from J. Phys. Chem. C 2020, 124, 11, 6395.

Since we measured a full spectrum every minute, and the reaction usually ran for over 100 minutes, it was natural to display the data as a time evolution series, showing how the spectral bands change over the course of the reaction.

From the band edge, we deduced the optical band gap of the ZnO NPs, from which we could calculate the average diameter of the NPs. From the MB absorption band we got the absorbance of MB, which decreased over time as the photocatalyst degraded the dye.

So, not the most polished plot ever, but I tend to value technicality over aesthestics most days of the week. The plot was created exclusively using ggplot2, and in so doing I supposed we lived up to the sentiment expressed by Hadley Wickham in this 2020 interview by Will Chase:

I hope that more people continue to express visualizations in a programming language, rather than a point and click tool. That’s my hope for the next 20 years, that it perchance becomes a given that code is the language of data science and data visualization.

The source code generating the static plot is published as part of the git repository of the paper.

I should note, that we use a few technical tricks (for lack of a better word) in this plot, for example, the plotted absorbance values are squared and at the same time the ordinate is shown using a square-root transform. This allows us to visually magnify the MB absorbance bands, which would otherwise be dwarfed by the prominent band edge absorbance. We use a similar trick for the abscissa, by plotting it against the energy scale instead of the wavelength scale, which has the effect of expanding the band edge region horizontally, making it much easier to follow the shift of the band edge over time.

Finally, to make the plot more information-dense (chemists just hate to leave unused whitespace in plots), we made use of the empty space created in the plot by our stretching and squeezing of the axes by plotting the derived quantities from the band edge (band gap and particle diameter) and the fitted absorbance as inset plots (we used grid::viewport() to position the insets).

Now, let’s see how we went about “translating” this into a multi-panel animation.

Animated plot with multiple overlaid panels

Here’s the animated plot:

For the intrepid reader who wants to see it all, the full source code of this blog post is available. For everyone else, the following chunks demonstrate just the ggplot2::, gganimate:: and magick:: functions we used to build the animated plot above.

nth_spectra <- 1
this_data <-
   pc_abs_small_N04H_nostir %>%
   filter(time_abs %in% 
      seq(0, length(unique(pc_abs_small_N04H_nostir$time_abs)), 
         by = nth_spectra))
this_condensed <- condensed_abs_small_N04H_nostir
this_ceiling <- ceiling_pc_abs_small_N04H_nostir
this_fityk <- 
   fityk_peaks_small_N04H_nostir_MB_max %>%
   # rename to conform with the name used in this_data and this_ceiling
   rename(time_exp = spectra)
mb_yoffset <- 220
cg_xoffset <- 500
cg_width <- 576
cg_height <- 324
# this height adjustment makes the plot panels of both insets equally sized
# necessary because only the bottom inset includes x-axis title and labels
mb_height <- cg_height - 45
# note, mb_yoffset actually pertains to the edge/diam plots, and vice versa
edge_yoffset <- mb_yoffset + mb_height
diam_yoffset <- mb_yoffset

For convenience, I rename the dataframes that will make up the different plots, and define coordinates and dimensions (in pixels) of the inset plots.

p <-
   ggplot() +
   geom_line(
      data = this_data,
      colour = "#5682a7",
      aes(x = eV, y = sqabs, group = sampleid)) +
   ## Tauc linear fits
   geom_smooth(
      data = subset(this_ceiling, fitted == TRUE),
      aes(x = eV, y = sqabs, group = time_exp),
      method = "lm",
      formula = "y ~ x",
      se = FALSE,
      size = 0.2,
      colour = alpha("#D5B450", 0.75), 
      linetype = 1,
      fullrange = TRUE) +
   # text box "ZnO band edge"
   annotate(
      "label", 
      x = 3.35, y = 4.5, 
      label = "ZnO band edge:\ngrowing NPs",
      size = 5, label.padding = unit(0.4, "lines")) +
   # text box "MB abs band"
   annotate(
      "label", 
      x = wavelength2energy(MB_band), y = 1.5,
      label = "MB abs band:\ndegradation",
      size = 5, label.padding = unit(0.4, "lines")) +
   # axis labels
   labs(
      x = "Energy/eV", y = "(Abs)²",
      title = 
         paste(
            "Tracking the growth of a ZnO photocatalyst and its effectiveness", 
            "at photodegradation over time, *t*/min = {round(frame_along, 0)}"),
      subtitle = 
         paste(
            "by *in-situ* UV/Vis absorption spectroscopy", 
            "under simulated solar irradiation")) +
   # secondary x-axis (wavelength)
   scale_x_continuous(
      breaks = seq(1, 5, 0.5),
      sec.axis = sec_axis(~ 1239.842 / ., name = "Wavelength/nm")) +
   scale_y_sqrt(expand = c(0, 0.05), breaks = seq(0, 8)) +
   coord_cartesian(xlim = c(1.5, 4.2), ylim = c(0, 7.5)) +
   theme_bw() +
   theme(
      legend.position = "none",
      axis.title = element_text(size = 14),
      axis.text = element_text(size = 12),
      plot.title = element_markdown(size = 14),
      plot.subtitle = element_markdown(size = 12))
# create animated plot
p_anim <- p + transition_reveal(along = time_exp)
p_anim_gif <-
   animate(
      plot = p_anim, 
      renderer = gifski_renderer(),
      width = 1920, height = 1200, units = "px",
      res = 150, # larger res makes all plot elements appear bigger
      # one frame per UV/Vis spectrum
      nframes = max(this_data$time_exp))

It is definitely nice to be able to use Markdown in the title, subtitle and text annotations, and the functions of the ggtext:: package allows us to do that.

We used the gifski renderer, which is the default, but I think it requires the gifski package to be installed on the system. I created an ansible role to take care of that.

p_mb <- 
   ggplot(data = this_fityk %>% filter(peakno %% 2 == 1)) +
   geom_errorbar(
      colour = alpha("#0B096C", 0.4),
      size = 0.65,
      aes(
         x = time_exp, 
         group = seq_along(time_exp),
         ymin = errors_min(height),
         ymax = errors_max(height))) +
   geom_point(
      colour = "black",
      size = 0.5,
      aes(
         x = time_exp, 
         group = seq_along(time_exp),
         y = height)) +
   labs(x = "*t*/min", y = "Abs (fitted height)") +
   scale_y_continuous(
      # easiest way to make certain this panel matches width of diam
      # is to give both the same sec_axis
      sec.axis = 
         sec_axis(
            trans = ~., 
            name = "Particle diameter/nm", 
            breaks = seq(0, 0.4, 0.2), 
            labels = c(3,4,5))) +
   theme_bw() +
   theme(
      legend.position = "none",
      axis.title.x = element_markdown(),
      axis.title.y.right = element_text(colour = NA),
      axis.text.y.right = element_text(colour = NA),
      axis.ticks.y.right = element_blank(),
      # make area outside panel transparent
      plot.background = element_rect(fill = NA, colour = NA))
# create animated plot
p_mb_anim <- p_mb + transition_reveal(along = time_exp)
p_mb_anim_gif <-
   animate(
      plot = p_mb_anim,
      renderer = gifski_renderer(),
      width = cg_width, height = cg_height, units = "px",
      res = 150,
      nframes = max(this_data$time_exp))

The MB inset plot is pretty straight-forward. In all the insets, we continue to transition_reveal along the same variable, and we use the same number of frames as in the main plot.

Note the use of seq_along() in the group aesthestics, which is necessary for the visual effect of the data points sequentially revealing themselves (with previously shown points remaining visible).

main_gif <- image_read(p_anim_gif)
mb_gif <- image_read(p_mb_anim_gif)
cg_offset <- paste0("+", cg_xoffset, "+", edge_yoffset)
cg_gravity <- "NorthWest"
main_mb_gif <- 
   image_composite(
      image = main_gif[1], 
      composite_image = mb_gif[1], 
      offset = cg_offset,  
      gravity = cg_gravity)
for (i in 2:max(this_data$time_exp)) {
   combined <- 
      image_composite(
         image = main_gif[i],
         composite_image = mb_gif[i],
         offset = cg_offset,  
         gravity = cg_gravity)
   main_mb_gif <- c(main_mb_gif, combined)
}

With both the main plot and the first inset created (as gganimate objects), it is time to create the first composite animation. The loop is a little awkward, and can probably be coded more efficiently, but hey, it works.

This gave us a new gganimate object, main_mb_gif, consisting of the main plot and one inset.

p_edge <- 
   ggplot(data = this_condensed) +
   geom_errorbar(
      size = 0.65,
      colour = alpha("#4387BF", 0.2),
      aes(
         x = time_exp, 
         group = seq_along(time_exp),
         ymin = errors_min(fit_Eg),
         ymax = errors_max(fit_Eg))) +
   geom_point(
      shape = 21,
      size = 0.65,
      fill = "#D5B450",
      colour = "#4387BF",
      aes(
         x = time_exp, 
         group = seq_along(time_exp),
         y = fit_Eg)) +
   labs(x = "", y = "Band gap/eV") +
   scale_x_continuous() +
   scale_y_continuous(
      sec.axis = 
         sec_axis(
            trans = ~., 
            name = "Particle diameter/nm", 
            breaks = seq(3.5, 3.8, 0.1), 
            labels = c(3,4,5,6))) +
   theme_bw() +
   theme(
      legend.position = "none",
      axis.title.x = element_blank(),
      axis.text.x = element_blank(),
      axis.title.y = element_text(colour = "#4387BF"),
      axis.text.y = element_text(colour = "#4387BF"),
      axis.title.y.right = element_text(colour = NA),
      axis.text.y.right = element_text(colour = NA),
      axis.ticks.y.right = element_blank(),
      # make area outside panel transparent
      plot.background = element_rect(fill = NA, colour = NA))
# create animated plot
p_edge_anim <- p_edge + transition_reveal(along = time_exp)
p_edge_anim_gif <- 
   animate(
      plot = p_edge_anim,
      renderer = gifski_renderer(),
      width = cg_width, height = mb_height, units = "px", 
      res = 150,
      nframes = max(this_data$time_exp))
edge_gif <- image_read(p_edge_anim_gif)

For the bottom one of the two stacked insets (p_edge and p_diam will be drawn on top of each other), we need to start getting a little messy with our code.

In order to precisely match the dimensions of the two stacked plots, and since the bottom one will have a left y-axis whereas the top one needs a right-side y-axis, both plots get dual y-axes, where the bottom one is drawn invisibly by setting its theme elements to colour=NA. We just need to make sure that both y-axes of both plots occupy the same amount of horizontal space, which we do by setting the labels of each so they are identical.

cg_offset <- paste0("+", cg_xoffset, "+", mb_yoffset)
cg_gravity <- "NorthWest"
main_mb_edge_gif <- 
   image_composite(
      image = main_mb_gif[1],
      composite_image = edge_gif[1],
      offset = cg_offset,  
      gravity = cg_gravity)
for (i in 2:max(this_data$time_exp)) {
   combined <- 
      image_composite(
         image = main_mb_gif[i],
         composite_image = edge_gif[i],
         offset = cg_offset,  
         gravity = cg_gravity)
   main_mb_edge_gif <- c(main_mb_edge_gif, combined)
}

Now we simply composite the first composite gif (which consists of the main plot with one inset) with the new inset, resulting in a new composite consisting of the main plot with two insets.

p_diam <- 
   ggplot(
      data = 
         this_condensed %>% 
         filter(spectra <= subset(cutoffs, sample == "N04H-small-nostir")$diameter)) +
   geom_errorbar(
      size = 0.35,
      colour = alpha("#D5B450", 0.3),
      aes(
         x = time_exp,
         group = seq_along(time_exp),
         ymin = errors_min(fit_np_diam),
         ymax = errors_max(fit_np_diam))) +
   geom_point(
      size = 0.45,
      colour = "#D5B450",
      aes(
         x = time_exp,
         group = seq_along(time_exp),
         y = fit_np_diam)) +
   labs(y = "Band gap/eV", x = "") +
   # manual adjustments to make sure the diam panel matches the edge panel's size
   scale_y_continuous(
      # invisible labels on left-hand y-axis, just to occupy space
      breaks = seq(3,6), labels = c("3.5", "3.6", "3.7", "3.8"),
      sec.axis = sec_axis(~., name = "Particle diameter/nm")) +
   scale_x_continuous() +
   coord_cartesian(xlim = c(0, max(this_condensed$time_exp))) +
   theme_bw() +
   theme(
      legend.position = "none",
      panel.grid = element_blank(),
      axis.title.y = element_text(colour = NA),
      axis.text.y = element_text(colour = NA),
      axis.ticks.y.left = element_blank(),
      axis.title.y.right = element_text(colour = "#D5B450"),
      axis.text.y.right = element_text(colour = "#D5B450"),
      axis.title.x = element_blank(),
      axis.text.x = element_blank(),
      axis.ticks.x = element_blank(),
      # make plot background transparent
      panel.background = element_rect(fill = NA, colour = NA),
      # make area outside panel transparent
      plot.background = element_rect(fill = NA, colour = NA))
# create animated plot
p_diam_anim <- p_diam + transition_reveal(along = time_exp)
p_diam_anim_gif <-
   animate(
      plot = p_diam_anim,
      renderer = gifski_renderer(),
      bg = "transparent", # https://github.com/thomasp85/gganimate/issues/174
      width = cg_width, height = mb_height, units = "px", 
      res = 150,
      nframes = max(this_data$time_exp))
diam_gif <- image_read(p_diam_anim_gif)

Here, for the top inset, we re-use the same techniques as for the bottom inset, with the addition that we need to make sure both panel and plot background is transparent. Also, we need to make the gganimate transparent, which is achieved with the bg='transparent' option.

cg_offset <- paste0("+", cg_xoffset, "+", diam_yoffset)
cg_gravity <- "NorthWest"
main_mb_edge_diam_gif <-
   image_composite(
      image = main_mb_edge_gif[1],
      composite_image = diam_gif[1],
      offset = cg_offset,  
      gravity = cg_gravity)
for (i in 2:max(this_data$time_exp)) {
   combined <- 
      image_composite(
         image = main_mb_edge_gif[i],
         composite_image = diam_gif[i],
         offset = cg_offset,  
         gravity = cg_gravity)
   main_mb_edge_diam_gif <- c(main_mb_edge_diam_gif, combined)
}

Finally, we composite again.

image_write_gif(
   image = main_mb_edge_diam_gif, 
   loop = 10,
   path = here::here("assets/animation-composite.gif"))

The final composite is then saved as a gif file using magick::image_write_gif(). At this point, we also set loop= to some number so that the gif does not loop forever (just a matter of taste).

Encountered issues and how to handle them

If you use knitr together with gganimate, be very mindful about knitr’s chunk option dev! I usually set opts_chunk$set(dev='svg') globally in my blog posts, and it turns out gganimate::animate(device=..) inherits this chunk option. Obviously the SVG device makes no sense for a gif, and you will get not-very-useful device-related errors. So if you are using knitr, make sure to set device='png' inside each call to animate(), or else in the relevant chunk’s options.
magick::image_read() failed with the error unable to get registry ID cache:hosts @ error/registry.c/GetImageRegistry/202 for no apparent reason. Fixed by increasing ImageMagick’s resource limits.
magick::image_write_gif() failed with the same error, despite image_read() working. Fixed by increasing the resource limits even more (see my imagemagick ansible role).
The end_pause argument of gganimate::animate() has no effect for gifs composited with magick::image_write_gif(). And unfortunately, image_write_gif() has no similar argument. I thought I could create and end pause manually by repeating the last frame of the gif a number of times, but I could not figure out how to achieve that using the magick functions. In the end, I gave up on adding an end pause and used the loop argument instead to set a max number of loops, since my point was to allow the reader some time to read the plot without everything moving around.

sessionInfo()

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] conflicted_1.1.0       photoec_0.2.2.9000     oceanoptics_0.0.0.9004
##  [4] common_0.0.2           tibble_3.1.6           tidyr_1.1.4           
##  [7] dplyr_1.0.8            magrittr_2.0.2         errors_0.3.6          
## [10] knitr_1.37             here_1.0.1             ggtext_0.1.1          
## [13] magick_2.7.3           gganimate_1.0.7        ggplot2_3.3.5         
## 
## loaded via a namespace (and not attached):
##  [1] progress_1.2.2    tidyselect_1.1.2  xfun_0.29         bslib_0.3.1      
##  [5] purrr_0.3.4       colorspace_2.0-2  vctrs_0.3.8       generics_0.1.2   
##  [9] htmltools_0.5.2   yaml_2.2.2        utf8_1.2.2        rlang_1.0.1      
## [13] gridtext_0.1.4    jquerylib_0.1.4   pillar_1.7.0      glue_1.6.1       
## [17] withr_2.4.3       DBI_1.1.2         tweenr_1.0.2      lifecycle_1.0.1  
## [21] stringr_1.4.0     munsell_0.5.0     blogdown_1.7      gtable_0.3.0     
## [25] memoise_2.0.1     evaluate_0.14     fastmap_1.1.0     fansi_1.0.2      
## [29] gifski_1.4.3-1    highr_0.9         Rcpp_1.0.8        scales_1.1.1     
## [33] cachem_1.0.6      jsonlite_1.7.3    farver_2.1.0      png_0.1-7        
## [37] hms_1.1.1         digest_0.6.29     stringi_1.7.6     bookdown_0.24    
## [41] rprojroot_2.0.2   grid_4.0.5        cli_3.2.0         tools_4.0.5      
## [45] sass_0.4.0        crayon_1.5.0      pkgconfig_2.0.3   ellipsis_0.3.2   
## [49] xml2_1.3.3        prettyunits_1.1.1 assertthat_0.2.1  rmarkdown_2.11   
## [53] R6_2.5.1          compiler_4.0.5

Measurement errors and dimensional analysis in R

Tue, 17 Nov 2020 18:00:00 +0100

I wrote this blog post in preparation for a hands-on web presentation I gave on Nov 13, 2020. It’s published mostly as-is in the hopes it might benefit others. Comments or questions on the text or regarding the code are most welcome!

Today I would like to demonstrate how to get automatic error propagation for your data analysis in R using the errors package by Iñaki Ucar et al. [1] I would also like to demonstrate how to combine that with dimensional analysis (with automatic unit conversion) using the units package by Pebesma et al. [2] This is usually known as quantity calculus.

To make it easy for you, dear reader, to follow along, we will take it from the start and assume all you have is a Windows 10 computer. If you work on MacOS¹ or Linux, you can skip right ahead to “Working with measurement errors” (but you may have to adapt some of the steps, as they were written with Windows 10 in mind).

Most of the fundamental tools used for reproducible scientific work, such as R, knitr, git, Python, and not to be forgotten, the whole chain of document generation tools such as LaTeX, Markdown, and pandoc, “just work” on any Linux distribution, but can be a bit of a hassle on Windows.

It is not hard to see why that is. All these tools are available for free and with libre software licences. Windows is neither free nor libre, and has a long history of working against this academic model of software sharing. These days, Microsoft, Google and other tech behemoths embrace open source software, because it allows them to monetise work without paying for it. Although, to be fair, they are also contributing back to varying degrees to this free/libre pool of software. For an academic, open source tooling offers many tangible and intangible benefits, prime among them by making the entire work reproducible. And it is the best way to future-proof your work.

So, let’s get to it. We will make use of the fact that Microsoft has added something called the Windows subsystem for Linux (WSL) on Windows 10, which allows you to run a (nearly feature-complete) Linux shell. In fact, recently Microsoft released WSL version 2 (WSL2), which attempts to bring the Linux shell natively to the Windows 10 desktop.

In order to work with R, the next sections will show you how to make use of WSL2 to install the latest Ubuntu LTS release, and then we will walk through installing R and RStudio Server using WSL2 Ubuntu. You will then be able to run R (in RStudio) from a browser window on your Windows desktop. Pretty cool, right?

With that in place, we will explore how to use R and the suite of r-quantities packages to seamlessly manipulate vectors that have associated measurement uncertainties and units through the steps of your data analysis, by way of an example using a subset of data from a real experiment in the Edvinsson lab.

Install WSL2 on Windows 10

The rest of this section, not including the subsections, is now of out of date! Microsoft has in fact published a guide on how to download and install Linux, which you should read. It boils down to 3 steps:

Open Powershell in administrator mode.
Install WSL2 by typing: wsl --install
Restart your computer.

In the near future, installing Ubuntu on Windows will be as simple as wsl --install (it’s in the development version of Windows 10), but for now, we have to follow the steps below to achieve the same.

Open Powershell as administrator, then enable WSL1:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

To enable WSL2, we must first enable the Windows virtual machine (VM) feature:

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Now you have to restart Windows.

Next, download the WSL2 Linux kernel update package for x64 (or ARM64, depending on your machine), and install it.

Then open Powershell as administrator again, and set WSL2 as default for all future VMs:

wsl --set-default-version 2

Now it is time to install our Linux distribution of choice (i.e., Ubuntu) from the Microsoft Store. Open the Microsoft Store, find Ubuntu, and install it. (And also, by the way, the Microsoft Store is awful. Such nasty underhanded nudging by Microsoft to create a wholly unnecessary “account”).

Anyway, you just installed Ubuntu on your Windows machine, congratulations! Now you get to create your Linux username and password: simply start the app Ubuntu from the Start menu.

Parenthesis: if your Windows 10 is itself a virtual machine

This is unlikely to apply to 99% of my readers, so just skip ahead to the next section.

Who does this section apply to? Well, if you, like me, run Windows 10 as a VM on top of your regular OS, you will need to make sure that a) your host machine (layer 0) supports nested virtualisation, and b) that your hypervisor has nested virtualisation enabled.

Details on how for the KVM hypervisor on a Linux host in footnote.²

Parenthesis: managing and rebooting your WSL2 virtual machines

Closing the Ubuntu shell in Windows does not restart the Ubuntu VM, and sometimes it can be useful to effect a reboot. So how do we do that?

It’s not obvious, but the wsl executable has two useful attributes,

wsl --list --verbose
wsl --terminate Ubuntu-20.04

that allows you to list/view all installed VMs (running or not), and then you simply --terminate to shut down, and wsl -d Ubuntu-20.04 to start again. Starting the VM can also be achieved from the app link in the Start menu, assuming the VM has one.

Install R and RStudio server on WSL2 Ubuntu

With a complete Ubuntu shell in place, we can now go about our business and install R. In your Ubuntu shell, add the public key of the CRAN repository, then add the Ubuntu 20.04 CRAN repo of R version 4.x to your package manager’s list of sources:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
echo "deb https://cran.uni-muenster.de/bin/linux/ubuntu focal-cran40/" | sudo tee /etc/apt/sources.list.d/r-project.list

Note that you can change the domain to any one of the CRAN mirrors if you want. At this point, remember to tell apt about the new source by:

sudo apt update

Install R and its dependencies using the aptitude package manager (we will also install libudunits2-dev, which is a C library and utility for handling physical units that the R package units depends on).

sudo apt install r-base r-base-core r-recommended r-base-dev gdebi-core build-essential libcurl4-gnutls-dev libxml2-dev libssl-dev libudunits2-dev

This installs about 400 MB of software in a jiffy. That’s R!

Now download the latest RStudio Server package file for Ubuntu 18+ and install it:

wget https://rstudio.org/download/latest/stable/server/bionic/rstudio-server-latest-amd64.deb
sudo gdebi rstudio-server-latest-amd64.deb

Great! Now all you got to do is launch your RStudio Server:

sudo rstudio-server start

With that, you can open your browser (Chromium/Chrome recommended) at http://localhost:8787 and enjoy your very own instance of RStudio Server. (You sign in with the same username and password as in your Ubuntu Linux VM).

And with that, all the preparatory steps are done.³ We can now start working in our data analysis environment in R.

Working with quantities and measurement errors

The quantities package (which ties together the capabilities of the units and errors packages) has an excellent vignette on quantity calculus in R.

The errors package is available on CRAN, and its git repo is published on Github. The package is also presented by its authors in an R Journal article from 2018, and there is also an informative blog post by the main author for the very first release of the package.

The units package handles physical quantities (values with associated units) in R, and builds on UNIDATA’s udunits-2 C library of over 3000 supported units, making it a very robust implementation while avoiding the sin of re-inventing the wheel. It does lead to a dependency on the system-level package udunits2 however, making it very hard to install and work with units on Windows.

I think that these packages have finally brought R up to par with Python with regards to quantity calculus, which is a crucial aspect for any scientist or engineer handling physical measurement values. Although it should be noted that the uncertainties package in Python is more mature (v1.0 was released in 2010 whereas the errors package in R still has not achieved that milestone), the errors package is very usable, indeed I used it for practically all the analysis steps of the latest paper I co-authored (I even managed to cite the package). [3]

Let’s take the errors and units packages out for a spin! Since this is a clean R installation, first we will need to install these packages and any other packages we would like to have available.

Install necessary R packages

Let’s install all the r-quantities packages, and a subset of the tidyverse packages (feel free to install any other packages you may like).

install.packages(c("errors", "units", "constants", "quantities"))
install.packages(c("dplyr", "magrittr", "ggplot2", "ggforce", "tibble"))
install.packages("remotes")
remotes::install_github("solarchemist/R-common")
remotes::install_github("solarchemist/oceanoptics")

(This operation may take a minute).

Now let’s load all the packages we will need and get to work on some real-life data.

library(errors)
library(constants)
library(units)
library(quantities)
library(common)
library(oceanoptics)
library(dplyr)
library(magrittr)
library(ggplot2)
library(ggforce)

Back to work

For the sake of keeping this post short, I will spare you the legwork of reading the individual spectra files into a dataframe. I have done that and saved to resultant dataframe to an rda-file which has been uploaded to a publicly accessible URL. Let’s import the data to a variable, and have a quick look at the dataframe:

# read the MB abs spectra from the public URL
df.MB <-
   common::LoadRData2Variable(url="https://public.solarchemist.se/data/uvvis-abs-MB-conc-series.rda")
df.MB %>% glimpse()
## Rows: 28,672
## Columns: 13
## $ sampleid           <chr> "H02AB", "H02AB", "H02AB", "H02AB", "H02AB", "H02AB", "…
## $ range              <chr> "001", "001", "001", "001", "001", "001", "001", "001",…
## $ wavelength         <dbl> 188.76, 189.23, 189.70, 190.17, 190.64, 191.12, 191.59,…
## $ intensity          <dbl> 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000,…
## $ IntegrationTime    <chr> "1000", "1000", "1000", "1000", "1000", "1000", "1000",…
## $ n_Averaged         <chr> "10", "10", "10", "10", "10", "10", "10", "10", "10", "…
## $ Boxcar             <chr> "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", …
## $ CorrElectricDark   <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ StrobeLampEnabled  <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ CorrDetectorNonLin <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ CorrStrayLight     <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ n_Pixels           <chr> "2048", "2048", "2048", "2048", "2048", "2048", "2048",…
## $ conc               <dbl> 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50, 50,…

As you can see, it is raw data from the spectrometer, and it comes with neither measurement errors nor units. (Unfortunately, it is still all-to-common for lab instruments to export data without such attributes).

It is perfectly acceptable to add estimated standard errors to this raw data (we will add units at the same time using the set_quantities() function of the quantities package).

# specify unit and error for the measured concentration values
df.MB$conc <-
   set_quantities(
      df.MB$conc * 1e-6,
      "mol/L",
      # set relative error 2% (estimate)
      df.MB$conc * 1e-6 * 0.02)
# specify unit and error for wavelength values to instrumental resolution
df.MB$wavelength <-
   set_quantities(df.MB$wavelength, "nm", 1.5)
# specify error for absorbance values to 1% rel error (estimate)
df.MB$intensity <-
   # absorbance is unitless, or more precisely, has a unit of "1"
   set_quantities(df.MB$intensity, "1", df.MB$intensity * 0.01)
# add a column describing the solvent (H2O or H2O+EtOH)
df.MB$solvent <- "H2O"
df.MB$solvent[grep("+EtOH$", df.MB$sampleid)] <- "H2O+EtOH"

For plotting, we will use the package ggplot2, which is like a cousin to the dplyr data manipulation package. Both come with a sort of “coding grammar”, which is very comfortable to use once you get used to it. In my opinion it is much easier to express chains of manipulations using this grammar (you can find plenty of examples in the code below).

We should note that ggplotting two columns that both are units objects will fail with Error in Ops.units(): both operands of the expression should be "units" objects, due to the units package (rightfully) determining that they don’t have the same units (or units that can be converted into each other). The (very elegant) solution to this finicky behaviour is provided by the ggforce package. All we have to do is make sure we load library(ggforce), and then we can get back to using ggplot2 as usual (mostly as usual, you will notice that we now use a scale_*_unit() function instead of the usual scale_*_continuous()).

(ref:uvvis-MB-conc-series) Recorded UV-Vis spectra of MB in water and MB in water with 10 vol% EtOH (with the estimated measurement error of the absorbances shown as vertical error bars in red).

ggplot(
   df.MB %>%
      filter(wavelength %>% as.numeric() >= 250) %>%
      filter(wavelength %>% as.numeric() <= 850)) +
   facet_wrap(~solvent) +
   geom_errorbar(
      colour = "red",
      aes(x = wavelength,
          ymin = errors_min(intensity),
          ymax = errors_max(intensity))) +
   geom_line(aes(x = wavelength, y = intensity, group = sampleid)) +
   scale_x_unit(
      breaks = seq(300, 800, 100),
      expand = expansion(0.03, 0)) +
   scale_y_unit(expand = expansion(0.03, 0)) +
   labs(x = "Wavelength", y = "Abs") +
   theme(legend.position = "none")

Figure 1: (ref:uvvis-MB-conc-series)

To check whether a unit already exists in the udunits2 library, I like to use the udunits2::ud.is.parseable() function. If you know of a better way, please let me know!

Now, as you have surely already noticed, this dataset is simply a series of UV-Vis absorbance spectra at varying concentrations (of methylene blue in water and water with some ethanol, in this case).

Here’s how the data looks at this point (after we set errors and/or units for some columns):

df.MB %>% glimpse()
## Rows: 28,672
## Columns: 14
## $ sampleid           <chr> "H02AB", "H02AB", "H02AB", "H02AB", "H02AB", "H02AB", "…
## $ range              <chr> "001", "001", "001", "001", "001", "001", "001", "001",…
## $ wavelength         (err) /nm 189(2) /nm, 189(2) /nm, 190(2) /nm, 190(2) /nm, 191…
## $ intensity          (err) /1 0(0) /1, 0(0) /1, 0(0) /1, 0(0) /1, 0(0) /1, 0(0) /1…
## $ IntegrationTime    <chr> "1000", "1000", "1000", "1000", "1000", "1000", "1000",…
## $ n_Averaged         <chr> "10", "10", "10", "10", "10", "10", "10", "10", "10", "…
## $ Boxcar             <chr> "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", …
## $ CorrElectricDark   <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ StrobeLampEnabled  <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ CorrDetectorNonLin <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ CorrStrayLight     <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "…
## $ n_Pixels           <chr> "2048", "2048", "2048", "2048", "2048", "2048", "2048",…
## $ conc               (err) /mol*L^-1 5.0(1)e-5 /mol*L^-1, 5.0(1)e-5 /mol*L^-1, 5.0…
## $ solvent            <chr> "H2O", "H2O", "H2O", "H2O", "H2O", "H2O", "H2O", "H2O",…

Most columns are of type “character”, but note how $wavelength (and two other columns) say something else in that field. We can take a closer look at the structure of the dataframe using the str() function:

df.MB %>% str()
## 'data.frame':    28672 obs. of  14 variables:
##  $ sampleid          : chr  "H02AB" "H02AB" "H02AB" "H02AB" ...
##  $ range             : chr  "001" "001" "001" "001" ...
##  $ wavelength        : Units+Errors: /nm num  189(2) 189(2) 190(2) 190(2) 191(2) ...  $ intensity         : Units+Errors: /1 num  0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) 0(0) ...  $ IntegrationTime   : chr  "1000" "1000" "1000" "1000" ...
##  $ n_Averaged        : chr  "10" "10" "10" "10" ...
##  $ Boxcar            : chr  "0" "0" "0" "0" ...
##  $ CorrElectricDark  : chr  "No" "No" "No" "No" ...
##  $ StrobeLampEnabled : chr  "No" "No" "No" "No" ...
##  $ CorrDetectorNonLin: chr  "No" "No" "No" "No" ...
##  $ CorrStrayLight    : chr  "No" "No" "No" "No" ...
##  $ n_Pixels          : chr  "2048" "2048" "2048" "2048" ...
##  $ conc              : Units+Errors: /mol*L^-1 num  5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 5.0(1)e-5 ...  $ solvent           : chr  "H2O" "H2O" "H2O" "H2O" ...

Having thus set suitable estimates (or perhaps even properly calculated) errors for the measured quantities wavelength, absorbance, and concentration, we will use this data to calculate the spectral absorption coefficient of MB in the two solvents we used.

As an introductory exercise, let’s define the wavelength of the strongest absorption band along with its unit and a synthetic standard error value:

wl.max.abs <- set_quantities(665, "nm", 5)

We can change the notation of the error from parenthesis to plus-minus when printing a quantity:

print(wl.max.abs, notation="plus-minus", digits=1)
## 665 ± 5 /nm

By default, errors uses parenthesis notation (which is more compact) and a single significant digit for errors.

We can also print more digits for the error part by setting digits. Both of these settings can also be set globally (for our entire R session) by using the options() function, like this:

options(errors.notation="plus-minus", errors.digits=2)

Before we get into calculating the spectral absorption coefficients, recall that the Beer-Lambert law

\[\begin{equation} A = \epsilon lc \;\Longleftrightarrow\; \epsilon = \frac{A}{lc} \end{equation}\]

includes the optical path length of the cuvette, $l=\SI{1}{\cm}$ in our case. Let’s set that as a variable so we can use it when calculating $\epsilon$:

# I don't know the tolerances of the cuvette dimensions, (quite the oversight, I know)
# but since these cuvettes are mass-produced I am assuming they are pretty tight and
# so estimating the error at 1/10 mm
optical.length <- set_quantities(1, "cm", 0.01)

With a series of UV-Vis absorbance spectra at varying concentrations, we can of course plot the linear relationship between $A \propto c$ at a particular wavelength (such as the main absorbance peak), but for the sake of this exercise we will take the calculation one step further and calculate $\epsilon(\lambda, A, c)$, i.e., the absorption coefficient for all wavelengths. It is not pretty code, but it gets the job done. Here goes:

# loop over each spectrum by solvent
solvents <- df.MB %>% pull(solvent) %>% unique()
# beware, unique() silently drops all quantity attributes
wl.steps <- df.MB %>% filter(sampleid == unique(df.MB$sampleid)[1]) %>% pull(wavelength)
# hold the results of the loop in an empty dataframe, pre-made in the right dimensions
# complication! quantities cols must use the same (or convertable) units from the start
abs.coeff <-
   data.frame(
      wavelength = rep(wl.steps, length(solvents)),
      solvent = "",
      # slope of linear fit
      k = rep(set_quantities(1, "L/mol", 1), length(solvents) * length(wl.steps)),
      # intercept of linear fit
      m = rep(set_quantities(1, "1", 1), length(solvents) * length(wl.steps)),
      # calculated abs coeff
      coeff = rep(set_quantities(1, "L*cm^-1*mol^-1", 1), length(solvents) * length(wl.steps)),
      rsq = NA, # R-squared of linear fit
      adj.rsq = NA) # adj R-squared of linear fit
i <- 0
# Fair warning: this loop is horribly slow, and may take several minutes to complete!
for (s in 1:length(solvents)) {
   message("Solvent: ", solvents[s])
   # varying conc of a particular solvent
   for (w in 1:length(wl.steps)) {
      # to keep track of the current row in abs.coeff, we will use a counter i
      i <- i + 1
      message("[", i, "] ", solvents[s], " :: ", wl.steps[w], " nm")
      # temporary df which we will use to calculate linear fits (abs.coeff) for each wl.step
      intensity.by.wl <-
         df.MB %>%
         filter(solvent == unique(df.MB$solvent)[s]) %>%
         # even though we drop the errors when filtering, it does not affect the returned rows
         filter(as.numeric(wavelength) == unique(as.numeric(df.MB$wavelength))[w]) %>%
         select(sampleid, wavelength, intensity, conc, solvent)
      # now we can calculate abs coeff for this solvent at the current wavelength
      this.fit <-
         # now this is the really tricky one
         # https://github.com/r-quantities/quantities/issues/10
         qlm(data = intensity.by.wl, formula = intensity ~ conc)
       # save R-squared and adjusted R-squared
      abs.coeff$rsq[i] <- summary(this.fit)$r.squared
      abs.coeff$adj.rsq[i] <- summary(this.fit)$adj.r.squared
      # save coefficients of linear fit
      abs.coeff$m[i] <- coef(this.fit)[[1]]
      abs.coeff$k[i] <- coef(this.fit)[[2]]
      abs.coeff$coeff[i] <- coef(this.fit)[[2]] / optical.length
      # solvent
      abs.coeff$solvent[i] <- unique(intensity.by.wl$solvent)
   }
}
save(
   abs.coeff,
   file = "/media/bay/taha/sites/hugo/solarchemist/static/assets/data/uvvis-abscoeff-MB-conc-series.rda")

# read the calculated MB abs coeff from the public URL
abs.coeff <-
   LoadRData2Variable(url="https://public.solarchemist.se/data/uvvis-abscoeff-MB-conc-series.rda")

Well, that was quite a workout!⁴ But now that we have saved the calculated absorption coefficients to an rda-file on disk, we won’t have to repeat that calculation (for a while at least). But the proof is in eating the pudding, so let’s plot the absorption coefficients we got.

(ref:abscoeff-MB-conc-series) Calculated spectral absorption coefficient for MB in water and MB in water with 10 vol% EtOH. With the estimated standard error due to propagated measurement errors of the underlying quantities (red errorbars).

ggplot(
   abs.coeff %>%
      filter(wavelength %>% as.numeric() >= 250) %>%
      filter(wavelength %>% as.numeric() <= 850)) +
   geom_errorbar(
      colour = "red",
      aes(x = wavelength,
          ymin = errors_min(coeff),
          ymax = errors_max(coeff))) +
   geom_line(aes(x = wavelength, y = coeff, group = solvent)) +
   scale_x_unit(
      breaks = seq(300, 800, 100),
      expand = expansion(0.03, 0)) +
   scale_y_unit(expand = expansion(0.03, 0)) +
   labs(x = "Wavelength", y = "Abs. coeff.") +
   theme(legend.position = "none")

Figure 2: (ref:abscoeff-MB-conc-series)

That looks good. Note how the vertical errorbars of the two curves (different solvents) overlap completely at certain spectral regions, notably at the main absorbance peak.

And not only did our entire analysis automatically propagate the measurement errors, we also managed to do the same for the units, doing automatic dimensional analysis at the same time (note how the units in the plot axes labels is automatically derived from the data).

That’s quantity analysis, in R, with real-life data! I hope you learned something new, and that this will inspire you to use quantity analysis for your future work.

But what about Python?

I feel it would be bad form to end this post without even mentioning Python’s support for error propagation and dimensional analysis, which far predates R’s.

Python has a whole array of packages to choose from for working with physical units, but as far as I can tell none of them build on the UNIDATA database.

However, the pint package appears to be the most popular by far (with 1200 stars on Github, it’s actually an order of magnitude more popular than all the related R packages), and it integrates with both NumPy and the uncertainties package and Pandas.

Support for errors and error propagation has been around since 2010 courtesy of the uncertainties package by Eric O. Lebigot.

I am not aware of any package for integrating between the uncertainties and pint packages, but perhaps that is not necessary in Python. I’m not sure on that point. is a little lacking.

In any case, getting up and running with Python is fairly easy. In fact, Python 3.x comes pre-installed on the Ubuntu shell we just configured. You should be aware, that you could just write and execute your Python code directly from RStudio Server (or R in general) using the reticulate package for R.

But if you prefer working with Python in its own IDE, you can actually install Jupyter Notebook and work from a browser, just like for RStudio Server. Just follow these steps:

In the WSL Ubuntu shell, run the following commands to install Jupyter Notebook:

sudo apt install python3-pip python3-dev
sudo -H pip3
sudo -H pip3 install --upgrade pip
sudo -H pip3 install jupyter

then run jupyter notebook and copy-paste the provided URL into your browser.

Note: for serious work, you should look into installing JupyterLab and/or JupyterHub instead of Jupyter Notebook.

This template for a student lab report I co-authored way back when uses both uncertainties and numpy for its calculations, and may be of interest.

Wrapper function redefining `lm()` to support units/errors

# wrap lm() to drop quantities and get around error when calling summary(lm())
# https://www.r-spatial.org/r/2018/08/31/quantities-final.html
qlm <- function(formula, data, ...) {
   # get units info, then drop quantities
   row <- data[1, ]
   for (var in colnames(data)) if (inherits(data[[var]], "quantities")) {
      data[[var]] <- drop_quantities(data[[var]])
   }

   # fit linear model and add units info for later use
   fit <- lm(formula, data, ...)
   fit$units <- lapply(eval(attr(fit$terms, "variables"), row), units)
   class(fit) <- c("qlm", class(fit))
   fit
}
# get quantities attributes back for the coef method
# (which is the only class we care about in this demonstration)
coef.qlm <- function(object, ...) {
  # compute coefficients' units
  coef.units <- lapply(object$units, as_units)
  for (i in seq_len(length(coef.units) - 1) + 1)
    coef.units[[i]] <- coef.units[[1]] / coef.units[[i]]
  coef.units <- lapply(coef.units, units)

  # use units above and vcov diagonal to set quantities
  coef <- mapply(set_quantities, NextMethod(), coef.units,
                 sqrt(diag(vcov(object))), mode="symbolic", SIMPLIFY=FALSE)

  # use the rest of the vcov to set correlations
  p <- combn(names(coef), 2)
  for (i in seq_len(ncol(p)))
    covar(coef[[p[1, i]]], coef[[p[2, i]]]) <- vcov(object)[p[1, i], p[2, i]]

  coef
}

sessionInfo()

sessionInfo()
## R version 4.1.3 (2022-03-10)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 22.04.3 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.10.0
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=sv_SE.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] ggforce_0.3.3          ggplot2_3.3.6          magrittr_2.0.3        
##  [4] dplyr_1.0.10           oceanoptics_0.0.0.9004 common_0.1.2          
##  [7] quantities_0.1.6       units_0.8-0            constants_1.0.1       
## [10] errors_0.3.6           knitr_1.39            
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.9       highr_0.9        bslib_0.4.0      compiler_4.1.3  
##  [5] pillar_1.8.1     jquerylib_0.1.4  tools_4.1.3      digest_0.6.29   
##  [9] gtable_0.3.0     jsonlite_1.8.0   evaluate_0.15    lifecycle_1.0.3 
## [13] tibble_3.1.8     pkgconfig_2.0.3  rlang_1.0.6      DBI_1.1.3       
## [17] cli_3.4.1        yaml_2.3.5       blogdown_1.10    xfun_0.31       
## [21] fastmap_1.1.0    withr_2.5.0      stringr_1.4.0    generics_0.1.3  
## [25] sass_0.4.2       vctrs_0.5.1      grid_4.1.3       tidyselect_1.2.0
## [29] glue_1.6.2       R6_2.5.1         fansi_1.0.3      rmarkdown_2.16  
## [33] bookdown_0.27    polyclip_1.10-0  farver_2.1.1     tweenr_1.0.2    
## [37] MASS_7.3-55      scales_1.2.0     htmltools_0.5.3  assertthat_0.2.1
## [41] colorspace_2.0-3 labeling_0.4.2   utf8_1.2.2       stringi_1.7.8   
## [45] munsell_0.5.0    cachem_1.0.6

References

[1] Ucar, I., Pebesma, E., Azcorra, A., Measurement errors in R, The R Journal, v. 10, n. 2, pp. 549–557, 2018.

[2] Pebesma, E., Mailund, T., Hiebert, J., Measurement units in R, The R Journal, v. 8, n. 2, pp. 486–494, 2016.

[3] Ahmed, T., Edvinsson, T., Optical quantum confinement in ultrasmall ZnO and the effect of size on their photocatalytic activity, The Journal of Physical Chemistry C, v. 124, n. 11, pp. 6395–6404, Mar. 2020.

If you run MacOS, you probably want to install Homebrew, a Linux-like package manager.↩︎
Check that the bare-metal Linux host supports “Nested virtualisation” by inspecting either one of /etc/modprobe.d/qemu-system-x86.conf or /sys/module/kvm_intel/parameters/nested. The first file should contain nested=1, and the second should contain Y or 1. If not, set them accordingly. With the Windows VM shut down, edit the KVM XML config file so that the <cpu> tag includes mode=host-passthrough, like this: <cpu mode='host-passthrough'>. Note that these settings must be in place before installing WSL2 Ubuntu on Windows. Also note that WSL2 supports nested virtualisation, but not WSL1.↩︎
We should mention here that there is a known limitation to WSL2 virtualisation regarding having services (such as RStudio Server in this case) persist across reboots of the VM or even across reboots of the Windows machine itself. In short, WSL2 does not support systemd nor init, so any services on the Linux VM will not autostart when the VM is rebooted. Even so, the web is full of hacks attempting to circumvent this limitation. On the other hand, this might not be such a big problem for the kind of experimentation that having an otherwise complete Linux shell available on the Windows desktop allows.↩︎
Note that we had to redefine the lm() function using our own wrapper that takes care of unsetting and resetting the units and errors. See above. The code was copied verbatim from Ucar’s r-spatial article.↩︎

Multiple Rstudio Server instances using a single R/LaTeX install with KVM

Sun, 20 Jan 2019 18:00:00 +0100

Introduction

Rstudio Server Open Source Edition (OSE) is offered with some key limitations compared to the Pro Edition. A few of these limitations are easy to circumvent using basic Linux sysadmin skills (such as encrypting traffic by using a reverse proxy), but most of RStudio Server OSE’s limitations are not so easy to work with, in my opinion.

The ability to use Rstudio server in the browser has been great and a major time-saver, but the one thing that caused me the most grief was the lack of support for multiple concurrent sessions (this is acutely felt when working within Rstudio projects). Now, Rstudio server Pro has this functionality and I’d be happy to support the Rstudio team, but the Pro license is firmly out of reach for a single user like myself (at 10000 USD/yr it’s clearly meant for institutions or large teams only).

Concurrent multiple sessions outside Pro is a feature that many others have asked for or tired to implement, e.g., see “Run multiple instances of RStudio in a web browser”, StackOverflow, “Unable to run multiple RStudio Server sessions at the same time”, Batch Connect - OSC RStudio Server”, “Parallel processes for Rstudio-server, much like Jupyter Notebooks”, RStudio Community, or “Multiple Rstudio server sessions from one user account”, RStudio Support.

Several work-arounds or solutions have been offered by the community, for example by running two Rstudio server processes on different IP addresses, or by assigning different Linux user/group ownerships to the Rstudio server processes, but unfortunately Rstudio has also been locking down the free Rstudio server in what can only be interpreted as an attempt to negate these efforts, such as by hard-coding the path to the .rstudio/ directory, or by making Rstudio project folders read-only for all users except the owner (and overriding the machine’s system-wide permissions settings in the process), or by making the browser-side of the session logout even across incognito mode. In short, none of the work-arounds actually achieve multiple sessions in the same browser.

In this post I will demonstrate how to setup multiple Rstudio Server OSE instances using a single, shared R and LaTeX installation on an Ubuntu host with KVM virtualisation. We will go through the setup of the host and the first virtual machine (VM) in some detail, and then we will see how more Rstudio Server instances can be (manually) created by cloning the existing VM.

The issue at hand

I wanted to have the ability to run multiple concurrent browser sessions of RStudio Server. In particular, my workflow required that I have different Rstudio projects, or even the same project, open in different browser windows/tabs at the same time. I also like to keep a particular RStudio project open, while working on other Rstudio projects in other tabs.

So, in short, we are trying to circumvent (without breaking any rules) the “multiple sessions” limitation that RStudio has put on its free-tier server offering. If you run an Rstudio server with a single user account, have some CPU/RAM to spare and want to run more than one concurrent RStudio Server session and don’t mind using virtual machines, read on.

The main advantage of this approach over simply provisioning an arbitrary number of VMs in the cloud (a perfectly valid solution to achieve “multiple sessions”) is that by keeping the VMs on a single host we can centralise the maintenance of the R and LaTeX installations. A disadvantage is that we have to compile R from source (but I will walk you through that, it’s not at all complicated).

The benefit of using full virtualisation is that as far as RStudio Server is concerned, we are running only one instance of it. Also, increasing the number of RStudio Server instances is trivial once the first VM is configured.

Requirements

You will need a server with sufficient CPU and RAM to run a hypervisor and the number of virtual machines of your choice. Still, nothing too fancy is required. For example, my own server runs Ubuntu 16.04 on an Intel Xeon E3-1200 series processor with 32 GB of RAM, and that’s been more than plenty to run 3 Rstudio Server VMs along with a multitude of other services.

If you want to use your Rstudio Server from outside your LAN (without having to fuss with virtual private networks) you will need to assign a (sub)domain to each Rstudio Server instance. If your router does not get a static IP address from your ISP, you will additionally have to setup dynamic DNS (DDNS). I will link to some guides on how to that below.

Of course, you will need superuser permission on the server in question (it’s your server, right?). Also, I’m assuming your server already runs RStudio Server, so by following this project you will gain at least one more instance (if you setup just one virtual machine).

Outline of setup

For the purposes of this project, we will assume that your server sits behind your router. The server is reachable externally at yourhost.yourdomain.se, and the router uses DDNS to keep that domain pointed at it. The server runs Ubuntu Server.

We will install a a virtual machine hypervisor (KVM) and a single virtual machine which we will then install RStudio Server Open Source Edition on, and we will demonstrate how you can clone this VM to setup as many RStudio Server instances as you like (and the server has resources for). Using full virtualisation will of course incur a slight performance penalty in terms of CPU and RAM overhead. If resources are scarce, I guess using Docker or LXD may offer another another way to achieve a similar result.

In KVM terminology, the host is the system that runs the VM hypervisor, and the guest is the virtual machine (VM). One host, multiple guests. The guests are controlled (started, stopped, cloned, destroyed) via the VM hypervisor on the host.

The guest (the virtual machine itself) will also run headless Ubuntu server 16.04 with RStudio Server.

NOTE: it’s actually very important that both the host and the guests run the same version of the same OS, since R will be compiled on the host and absolutely needs the same environment on the guest. Even using Ubuntu 16.04 on the host and Ubuntu 18.04 will fail to run R properly on the guest (I’ve tried).

In terms of security, the setup described here assumes that 1) the server sits behind a router with a firewall, and 2) that the local network can be considered safe (the RStudio Server instances on the guests send their traffic to/from the host over unencrypted connections). But this is not so bad, since any setup that uses a proxy that terminates HTTPS connections will need to pass on the traffic unencrypted over part of the network. I guess if you distrust the LAN you could configure a separate VLAN for this traffic, or otherwise encrypt it somehow.

The central idea of this project is that we will install and maintain the R and LaTeX installations only on the host. We can expose the R and LaTeX trees to the guest by using KVM filesystem passthrough. For this to work, it’s critical that we use the same OS with the same version on host and guest.

The table below outlines the software components needed to complete our setup.

Installed on host	Installed on VM	Configured on VM
Apache web server	No	No
Certbot/Let’s Encrypt	No	No
LaTeX/TeXLive	No	Yes (added to `$PATH`)
R (compiled from source)	No	Yes (symlinks)
KVM	No	No
Fonts	No	No
Rstudio server	YES	Yes

As you can see, only RStudio Server has to be installed on both the host and the guest. The rest of the software stack can be almost exclusively managed on just the host. This is what makes this endeavour workable, in my opinion.

Apache is used as a reverse proxy to encrypt all traffic to/from each RStudio Server. Certbot is used to download the Let’s Encrypt certificates. Additionally, we will setup a CRON job or a Systemd timer to automatically renew those certificates.

We will use TeXLive and install a default LaTeX installation. LaTeX, as installed by TeXLive, lives entirely inside a single directory (except for a few files) and so is easy to passthrough to the virtual machine. R, as installed from the Debian repos, is less well-behaved in that regard. Actually, it’s TeXLive that’s unusual and not very Debian-like in this regard, but since we are looking to passthrough the entire installation so that we can “trick” the guest into thinking the installation is local, having it all inside a single directory tree is ideal. And I’ve found no way to passthrough R, as installed from the repos, to a VM. My solution is to instead compile R from source. This allows us to fit all of it inside the /opt/R/ tree, and allows us to passthrough just that directory to the guest.

And we will of course need to install the VM hypervisor, KVM in this case, as well as libvirt which is a commandline tool to manage KVM and its virtual machines. With all that in place, we will install RStudio Server on the guest (this will be instance no. 2, assuming you already have it installed on the host itself). Once the guest setup is complete, we can clone it to create additional RStudio Server instances.

For all of this to work, we will also need to learn a little about network interfaces and bridges so that we can provision each VM with its own IP address on our LAN. Each instance will get its own Apache virtualhost file which will simply setup a proxy pointing to its IP address on the LAN, redirecting all traffic from the subdomain we setup to it.

A note on naming. I will use the placeholders <yourhost> and <guestname> in this text. You should of course replace those with the hostname of your host machine and virtual machine, respectively. Also, the sooner you decide on names for your virtual machines, the better. My recommendation is that you use the same hostname in the subdomains you setup to point at each VM, but that’s optional. I also recommend you to setup the DNS records for the virtual machines early in the process, since DNS propagation may take some time, and you will need those records in place to install the Let’s Encrypt certificates.

And that’s about the gist of it. Still interested? Let’s get into the nitty-gritty!

Setup the host

The server used as an example in this post runs Ubuntu 16.04. The instructions here should be easily adapted to work for any Debian-based Linux distribution.

Install Apache, if you haven’t already. We will not go through that here. Plenty of guides describe the process. I should mention that my server runs Apache v2.4.18 (the syntax used in the virtualhost config files changed slightly with v2.2, so if your server happens to use an older Apache version, beware). We will get back to the web server configuration once the VM is up and running with its own IP address. Now, let’s move on to installing LaTeX and compiling R.

Install LaTeX (TeXLive)

If you plan to install LaTeX, you should do so before compiling R since the latter will look for it during compilation.

$ wget http://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz
$ tar -xvf install-tl-unx.tar.gz
$ cd install-tl-20190119/
$ ./install-tl

Depending on your preferences, you might want to make the directory /usr/local/texlive owned by your user account instead of root to simplify the maintenance of TeXLive. Note that downloading a full TeXLive installation can take quite a while.

To add LaTeX to the system-wide PATH, I added the following line to /etc/profile:

PATH="/usr/local/texlive/2018/bin/x86_64-linux:$PATH"

This will make the LaTeX executables findable by R and other programs on the host (you might have to logout and log back in for the change to take effect).

Compile R from source

I have found that compiling R from source (on the host) aids considerably in achieving a working R session on each guest, because whereas compiled R lives entirely inside the /opt/R/ directory, the R installated from the Debian repos gets parts of it installed all over the filesystem (/usr/lib/R/, /usr/local/lib/R/, among other places) which makes it unworkable to setup KVM filesystem passthrough.

For removing an existing R installed by the apt package manager, see the appendix. But please don’t remove your existing R installation before reading through the installation instructions below carefully.

Download and extract the R source tarball (at the time I compiled R, version 3.5.1 was available).

chepec@yourhost:~/Downloads
$ wget https://cran.r-project.org/src/base/R-3/R-3.5.1.tar.gz
$ tar -xzvf R-3.5.1.tar.gz
$ cd R-3.5.1

Depending on which R packages you plan to install, you might need to install certain dependencies before compiling R. How to figure out which dependencies you need may not be entirely straight-forward. You should at least have a look at the capabilities() of your current R installation (before removing it):

> capabilities()
       jpeg         png        tiff       tcltk         X11        aqua
       TRUE        TRUE        TRUE        TRUE       FALSE       FALSE
   http/ftp     sockets      libxml        fifo      cledit       iconv
       TRUE        TRUE        TRUE        TRUE        TRUE        TRUE
        NLS     profmem       cairo         ICU long.double     libcurl
       TRUE        TRUE        TRUE       FALSE        TRUE        TRUE

Some of these capabilities/dependencies appear to be packaged together with apt’s R installation, so when we remove it, we also uninstall the dependencies. Anyway, I found that, on my particular system, tcltk was missing and had to be installed prior to compiling R for a whole bunch of R packages to work later. So we need to install the tcl and tk development packages:

chepec@yourhost:~
$ sudo apt install tcl8.6-dev tk8.6-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libxft-dev libxss-dev x11proto-scrnsaver-dev
Suggested packages:
  tcl8.6-doc tk8.6-doc
The following NEW packages will be installed:
  libxft-dev libxss-dev tcl8.6-dev tk8.6-dev x11proto-scrnsaver-dev
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.

Now let’s build R from the source tarball we downloaded (note the build flags, and adapt to your needs).

chepec@yourhost:~/Downloads/R-3.5.1
$ ./configure --prefix=/opt/R/$(cat VERSION) --enable-R-shlib --with-blas --with-lapack --with-tcltk

This should produce lots of output, ending with a block looking like this:

R is now configured for x86_64-pc-linux-gnu

  Source directory:          .
  Installation directory:    /opt/R/3.5.1

  C compiler:                gcc  -g -O2
  Fortran 77 compiler:       f95  -g -O2

  Default C++ compiler:      g++   -g -O2
  C++98 compiler:            g++ -std=gnu++98 -g -O2
  C++11 compiler:            g++ -std=gnu++11 -g -O2
  C++14 compiler:            g++ -std=gnu++14 -g -O2
  C++17 compiler:
  Fortran 90/95 compiler:    gfortran -g -O2
  Obj-C compiler:

  Interfaces supported:      X11, tcltk
  External libraries:        readline, BLAS(generic), LAPACK(generic), curl
  Additional capabilities:   PNG, JPEG, TIFF, NLS, cairo, ICU
  Options enabled:           shared R library, R profiling

  Capabilities skipped:
  Options not enabled:       shared BLAS, memory profiling

  Recommended packages:      yes

To figure out if anything was amiss during the compilation, you really should spend some time scrolling through config.log in the compilation directory. It might save you a bunch of time later.

Now if you’re happy with the build, let’s make and install.

$ make
$ sudo make install

That’s it. Now either put the opt/R/<version-number>/bin/ directory into the system PATH, or symlink the two R executables into somewhere already on the system PATH. I prefer the latter approach, so:

$ sudo ln -s /opt/R/3.5.1/bin/R /usr/bin/R
$ sudo ln -s /opt/R/3.5.1/bin/Rscript /usr/bin/Rscript

When we create our first VM we will create the same symlinks on there. And don’t forget that these symlinks should be changed once you have upgraded to a newer R version.

NOTE: at this point, it’s perhaps worth underscoring that going forward, everything R-related should be kept inside this /opt/R/ tree. That includes config files (by default placed in /opt/R/3.5.1/lib/R/etc/) as well as installed system and user packages. Anything you place outside this tree will not be visible on the VM which would defeat the purpose of this whole project.

Install KVM and the virtual machine OS

Install the VM hypervisor (KVM/libvirt):

$ sudo apt install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker

Verify the KVM installation by running kvm-ok. Also check that the libvirtd service is running: systemctl status libvirtd.service. If it’s not, start it: sudo systemctl start libvirtd.service.

For networking, I decided to setup the server’s physical network interface card (NIC) as a bridge adapter. This way, each VM will act as its own NIC visavi the router, and I can simply let the router assign IP addresses as usual (and it allows me to use my router’s DHCP server to set a static IP address for each VM). This was simpler to setup than iptables redirects on the host or other alternatives (there are many ways to setup networking for VMs). And if you want SSH access to the VM, just setup port forwarding of the SSH port on the router, as per usual.

To set up your NIC as a bridge adapter, first make sure the bridge-utils package is installed on the host. We will edit /etc/network/interfaces, so make sure to take a backup copy of it first: sudo cp /etc/network/interfaces /etc/network/interfaces.backup. For good measure, you might want to backup /etc/resolv.conf as well.

This is how my /etc/network/interfaces looked before any changes:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth1
iface eth1 inet dhcp

and this is the same file now configured for bridging:

# The loopback network interface
auto lo
iface lo inet loopback

# The physical network interface
auto eth1
iface eth1 inet manual

auto br0
iface br0 inet dhcp
bridge_ports eth1
bridge_stp off
bridge_fd 0
bridge maxwait 0

This sets up the new br0 interface, and effectively makes eth1 just a physical transport layer to it. Bridging means the VM will be exposed to whatever the bridged NIC is connected to, in our case to our LAN and the router. As such it will get an IP address from the router’s DHCP server just like any other networked device on the LAN.

Note that by default, KVM creates its own bridge at 192.168.122.1. This device acts as a virtual router for your VMs. (We will not use this virtual router for anything in this project).

With the edits in place, restart the networking service (but be warned: if your connection to the host is over SSH, at this point you will lose the connection! I found it useful to apply these networking changes while physically connected to the host).

$ sudo systemctl restart networking.service

With KVM installed and with the host’s network configured, we are now ready to install our first virtual machine. The installation is done over VNC.

All virtual machine files and other related files will be stored under /var/lib/libvirt/. The default path of saved ISO images is /var/lib/libvirt/boot/.

We will go ahead and download the latest netinst Ubuntu Server 16.04 64-bit image, and save it in /var/lib/libvirt/boot/. Installing is fairly straight-forward once you figure out the virt-install command parameters (lots of guides online, see the Bibliography). I used the following command with success:

$ sudo virt-install --virt-type=kvm --name <guestname> --ram 4096 --vcpus=2 \
  --os-variant=ubuntu16.04 --hvm \
  --cdrom=/var/lib/libvirt/boot/ubuntu-16.04.3-amd64-netinst.iso \
  --network=bridge=br0,model=virtio --graphics vnc \
  --disk path=/var/lib/libvirt/images/<guestname>.qcow2,size=20,bus=virtio,format=qcow2

Here we allocate a maximum of 4 GB RAM, 2 CPU cores and 20 GB disk space to the new virtual machine. KVM is pretty smart and won’t use up those resources until usage demands it, so most of the time the load on the host from this VM will be much less than this. As for the disk space, 20 GB should be plenty. In fact, if you don’t plan to keep your work files on the VM itself, even half that would be plenty). Anyway, it’s fairly easy to expand a qcow2 disk at a later date, but slightly complicated to shrink one.

The above command will simply tell you that installation has started - you will need to connect to the OS installation using VNC to actually proceed with the installation. KVM gives every guest it launches a new VNC instance with its own port (starting with port 5900). So we’ll check the current guest’s VNC port number, and then tunnel in via ssh to proceed with the OS installation:

# find the VNC port number of the VM
$ sudo virsh dumpxml <guestname> | grep vnc
# sample output:
<graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>

Use a VNC client (such as xtightvncviewer) to connect to the port used by the libvirt installer on the server. For security, we will route the connection through an SSH tunnel:

# Create an ssh tunnel from your workstation to the host
$ ssh <username>@yourhost.yourdomain.se -L 5900:localhost:<localport>
# on your workstation, use a local VNC client to connect to the tunnel
$ xtightvncviewer localhost:<localport>

Et voilà! Now proceed with the OS installation as usual. If the VNC connection is dropped during the installation (happens sometimes), reconnect by repeating the last chunk and continue from where you were dropped off.

When the OS installation is done, it’s a good time to setup static DHCP for this VM and port forwarding for the SSH port. Both of these settings are done on your router. (There are some links in the Bibliography to help get you started). You probably also want to setup passwordless SSH access to the VM (since you already have a server, I’m going to assume you know how to do that). Make sure to set the VM’s hostname in both /etc/hostname and /etc/hosts.

With those things in place, we will also setup a new subdomain for accessing this Rstudio server instance from anywhere. If you configured SSH port forwarding earlier, you will also be able to SSH from anywhere. I cannot go into how to do this in detail since it depends on your registrar (Loopia, GoDaddy, Amazon Route 53, etc.). But in summary, go to your registrar, and create a CNAME record that points <name-for-VM>.yourdomain.se to the already existing (sub)domain of your host server (yourhost.yourdomain.se). Once the new DNS settings have propagated across the web, you should be able to reach your new virtual machine using your chosen URL.

While still on the host, make sure that the virtual machine is running:

$ virsh list --all
$ virsh start <guestname>

Share directories on the host to the guests

Shutdown the virtual machine and edit its configuration XML file:

$ virsh shutdown <guestname>
$ virsh edit <guestname>

and add these blocks inside the devices tree:

<filesystem type='mount' accessmode='passthrough'>
   <source dir='/opt/R'/>
   <target dir='R'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
   <source dir='/usr/local/texlive'/>
   <target dir='texlive'/>
</filesystem>
<filesystem type='mount' accessmode='passthrough'>
   <source dir='/usr/share/fonts'/>
   <target dir='fonts'/>
</filesystem>

The string in target dir is just a label, which we will have use for in the fstab (on the guest) to mount these shares. Make the label informative.

Restart the VM.

$ virsh start <guestname>

And with that, let’s move on to mounting the passed-through directories on the VM.

Setup the guest (virtual machine)

Filesystem passthrough for R, TeXLive and fonts

loop
virtio
9p
9pnet
9pnet_virtio

then load those modules (make sure to do this step as root):

$ sudo service kmod start

Create the mountpoints for the shares on the guest. The mountpoints for R and TeXLive should not exist on the guest, so simple create those directories (use sudo if necessary):

$ mkdir /opt/R
$ mkdir /usr/local/texlive

The fonts directory probably exists already, so let’s delete it and recreate an empty one:

$ sudo rm -rf /usr/share/fonts
$ sudo mkdir /usr/share/fonts

Now you should be ready to mount the passed-through directories (here we make use the label we set in target dir above):

$ sudo mount R /opt/R -t 9p -o trans=virtio
$ sudo mount texlive /usr/local/texlive -t 9p -o trans=virtio
$ sudo mount fonts /usr/share/fonts -t 9p -o trans=virtio

Now make sure everything works as expected on the guest. You should be able to read from these directories, but not write. If everything works as expected, go ahead and add these shares to /etc/fstab for automatic mounting at boot:

R        /opt/R               9p    trans=virtio   0   0
texlive  /usr/local/texlive   9p    trans=virtio   0   0
fonts    /usr/share/fonts     9p    trans=virtio   0   0

Note: by default, the security profile of KVM will not allow the guest to write to any directories mounted using the passthrough method. And this works just fine for our purposes, since all maintenance of the R, LaTeX and fonts directories should be done on the host machine anyway.

Make R and TeXLive executables accessible

For R and LaTeX to work as expected on the guest, it’s not enough to make the /opt/R and /usr/local/texlive trees available. We should also expose the executables where other programs expect to find them, namely somewhere on the system’s PATH.

Additionally, there are a few crucial TeX utilities that we need to get from the host to the guest.

$ ls -1 /usr/bin/tex*
/usr/bin/texi2any
/usr/bin/texi2dvi
/usr/bin/texi2pdf
/usr/bin/texindex

These are actual files and not links. They are not available on the guest, but need to be for TeXLive to work properly in conjunction with R.

There’s no way to passthrough single files from host to guest, and we definitely do not want to passthrough the entire /usr/bin/ directory. So although inelegant, the best solution appears to be to simply copy these files from host to guest. Note that you will need to redo this if these files should be updated by tlmgr on the host in the future. Assuming you have setup SSH access from host to guest, you can copy them like this:

chepec@yourhost:~
$ scp -p /usr/bin/texi* <guestname>:/home/chepec/Downloads/
$ ssh <guestname>
chepec@<guestname>:~
$ sudo cp -a /home/chepec/Downloads/texi* /usr/bin/
$ sudo chown root:root /usr/bin/texi*

Create symlinks in /usr/bin/ for the R executables:

$ sudo ln -s /opt/R/3.5.1/bin/R /usr/bin/R
$ sudo ln -s /opt/R/3.5.1/bin/Rscript /usr/bin/Rscript

And add the TexLive path to the system PATH. In /etc/profile, append

PATH="/usr/local/texlive/2018/bin/x86_64-linux:$PATH"

Why not add the R/bin directory to PATH as well, you ask? Well, you may well want to do that, especially if you have more than just two executables in there. Both methods (symlinking or editing $PATH) achieve the same result.

NOTE: if you have a network share or similar where you keep your work files, you might want to mount that too on the guest. Since you most definitely want to read and write to this directory from the VM, you might want to use NFS (if it’s inside the LAN) or SSHFS (otherwise) to mount it instead of KVM passthrough. If you do, having your regular user account on the VM have the same username, uid and gid as that on the share will most certainly make your life as an admin easier.

Install Rstudio Server

Browse to https://www.rstudio.com/products/rstudio/download-server/ and follow the instructions to download and install the latest RStudio Server Open Source Edition.

The installation will automatically set up a systemd startup script (or upstart, depending on your system) for Rstudio server. Neat!

By default, RStudio Server listens on port 8787 on all interfaces, 0.0.0.0:8787. At this point, you should be able to verify that RStudio Server is accessible via your browser by entering http://<ip-address-guest>:8787.

Back to the host

Configure Apache reverse proxy and Let’s Encrypt certificates

The Apache web server on our host machine will act as a reverse proxy, and we will also install a free TLS (HTTPS) certificate from Let’s Encrypt for our RStudio Server instance.

Let’s say you named your virtual machine callisto and created a subdomain rstudio.callisto.yourhost.se for it. We would thus create a new Apache virtualhost file /etc/apache2/sites-available/rstudio.callisto.yourhost.se.conf with the following contents:

<VirtualHost *:80>
   ServerAdmin webmaster@yourhost.se
   ServerName rstudio.callisto.yourhost.se

   Redirect permanent / https://rstudio.callisto.yourhost.se/

   ErrorLog ${APACHE_LOG_DIR}/rstudio.callisto.yourhost.se_error.log
   CustomLog ${APACHE_LOG_DIR}/rstudio.callisto.yourhost.se_access.log combined
</VirtualHost>

<VirtualHost *:443>
   ServerAdmin webmaster@yourhost.se
   ServerName rstudio.callisto.yourhost.se

   SSLEngine on
   SSLCertificateKeyFile   /etc/letsencrypt/live/rstudio.callisto.yourhost.se/privkey.pem
   SSLCertificateFile      /etc/letsencrypt/live/rstudio.callisto.yourhost.se/cert.pem
   SSLCertificateChainFile /etc/letsencrypt/live/rstudio.callisto.yourhost.se/chain.pem

   <Proxy *>
      Allow from localhost
   </Proxy>

   RewriteEngine on
   RewriteCond %{HTTP:Upgrade} =websocket
   RewriteRule /(.*) ws://192.168.100.110:8787/$1 [P,L]
   RewriteCond %{HTTP:Upgrade} !=websocket
   RewriteRule /(.*) http://192.168.100.110:8787/$1 [P,L]
   ProxyPass / http://192.168.100.110:8787/
   ProxyPassReverse / http://192.168.100.110:8787/
   ProxyRequests Off

   ErrorLog ${APACHE_LOG_DIR}/rstudio.callisto.yourhost.se_error.log
   CustomLog ${APACHE_LOG_DIR}/rstudio.callisto.yourhost.se_access.log combined
</VirtualHost>

You should of course edit domain name, email address and IP address above to fit your setup.

With DNS in place, now is the time to install the Let’s Encrypt certificate. This will be managed by certbot. EFF has excellent instructions on how to install the certbot client. With certbot installed, fetch the certificate (I like to manage the integration with Apache myself, so I use the certonly option):

$ sudo certbot certonly --authenticator standalone \
  --pre-hook "systemctl stop apache2.service" \
  --post-hook "systemctl start apache2.service" \
  --email webmaster@yourhost.se -d rstudio.callisto.yourhost.se

You should really setup a CRON job or a systemd timer for automatic renewals your Let’s Encrypt certificates. Check the Certbot docs and the web for info on how to accomplish that.

With the TLS certificate installed, we are now ready to enable our virtualhost and reload Apache:

$ sudo a2ensite rstudio.callisto.yourhost.se.conf
$ sudo systemctl reload apache2.service

If no errors were thrown, you should new be able to reach your RStudio Server instance running on your VM from your browser using the URL http://<rstudio.callisto.yourhost.se> which should automatically redirect to https.

Create additional VMs (KVM cloning)

Shutdown the VM before cloning.

The cloning operation is straight-forward (should be done as root):

$ virt-clone --original callisto --name ganymede --file /var/lib/libvirt/images/ganymede.qcow2
Allocating 'ganymede.qcow2'                  |  20 GB  00:00:39
Clone 'ganymede' created successfully.

This clones callisto and sets the new VM’s name to ganymede. Note that this naming only affects the VM hypervisor, we still have to login to the new guest and edit the hostname (in /etc/hosts and /etc/hostname).

The bridged networking we setup above means that the new VM will get its own IP address assigned by the router (which you might want to assign statically by the way). We need to create a new subdomain that this instance will be reached on, and once that’s in place we will create a new Apache vhost and download a Let’s Encrypt certificate.

Create /etc/apache2/sites-available/rstudio.ganymede.yourhost.se.conf with the following contents:

<VirtualHost *:80>
   ServerAdmin webmaster@yourhost.se
   ServerName rstudio.ganymede.yourhost.se

   Redirect permanent / https://rstudio.ganymede.yourhost.se/

   ErrorLog ${APACHE_LOG_DIR}/rstudio.ganymede.yourhost.se_error.log
   CustomLog ${APACHE_LOG_DIR}/rstudio.ganymede.yourhost.se_access.log combined
</VirtualHost>

<VirtualHost *:443>
   ServerAdmin webmaster@yourhost.se
   ServerName rstudio.ganymede.yourhost.se

   SSLEngine on
   SSLCertificateKeyFile   /etc/letsencrypt/live/rstudio.ganymede.yourhost.se/privkey.pem
   SSLCertificateFile      /etc/letsencrypt/live/rstudio.ganymede.yourhost.se/cert.pem
   SSLCertificateChainFile /etc/letsencrypt/live/rstudio.ganymede.yourhost.se/chain.pem

   <Proxy *>
      Allow from localhost
   </Proxy>

   RewriteEngine on
   RewriteCond %{HTTP:Upgrade} =websocket
   RewriteRule /(.*) ws://192.168.100.111:8787/$1 [P,L]
   RewriteCond %{HTTP:Upgrade} !=websocket
   RewriteRule /(.*) http://192.168.100.111:8787/$1 [P,L]
   ProxyPass / http://192.168.100.111:8787/
   ProxyPassReverse / http://192.168.100.111:8787/
   ProxyRequests Off

   ErrorLog ${APACHE_LOG_DIR}/rstudio.ganymede.yourhost.se_error.log
   CustomLog ${APACHE_LOG_DIR}/rstudio.ganymede.yourhost.se_access.log combined
</VirtualHost>

Note the different IP address and server name (edit to fit your setup).

And install the Let’s Encrypt certificate:

$ sudo certbot certonly --authenticator standalone \
  --pre-hook "systemctl stop apache2.service" \
  --post-hook "systemctl start apache2.service" \
  --email webmaster@yourhost.se -d rstudio.ganymede.yourhost.se

Enable the site and reload the web server:

$ sudo a2ensite rstudio.callisto.yourhost.se.conf
$ sudo systemctl reload apache2.service

And that should be it.

Autostart the virtual machines

Virtual machines created with KVM do not autostart by default when the host reboots. But for our application, it might be desirable to have them autostart. To achieve that, issue the following for each VM.

$ virsh autostart <guestname>

You can confirm the new setting by:

$ virsh list --all --autostart
$ virsh list --all --no-autostart

Conclusion

This is admittedly a resource-demanding solution to the “single-session per browser” limitation put on RStudio Server OSE by RStudio. Also, it requires the admin to setup multiple VMs and the user to remember multiple URLs.

Still, it’s the most reliable and resource-efficient way to achieve actual multi-sessions without any restrictions in the browser I have seen to date, outside of the Pro offering of course. And the single R/LaTeX installation at its core makes it quite maintainable and means adding more VMs (i.e., more RStudio Server instances) marginally cost-free in terms of setup.

I have been running this setup myself for about a year now, and I decided to write-up this project in the hopes that others (single researchers like me, or groups/departments) might find it useful. And if you have comments or questions, please feel free to ask them on my Twitter or Mastodon accounts (and sorry for the lack of comments on this site – I’m still figuring out how to implement that with Hugo outside of Disqus).

Appendix

Remove an existing R aptitude installation

Here’s how to completely remove an existing R installation that was installed from the Debian repos. Please be careful to record any configuration you want to move over to your new installation. This will remove everything, including config files.

The installed R version was 3.4.4 (2018-03-15). List all of the installed R packages:

chepec@yourhost:~
$ dpkg -l | grep ^ii | awk '$2 ~ /^r-/ { print $2 }'
r-base
r-base-core
r-base-dev
r-base-html
r-cran-boot
r-cran-class
r-cran-cluster
r-cran-codetools
r-cran-crayon
r-cran-digest
r-cran-evaluate
r-cran-foreign
r-cran-irdisplay
r-cran-jsonlite
r-cran-kernsmooth
r-cran-lattice
r-cran-magrittr
r-cran-mass
r-cran-matrix
r-cran-memoise
r-cran-mgcv
r-cran-nlme
r-cran-nnet
r-cran-pbdzmq
r-cran-r6
r-cran-repr
r-cran-rgl
r-cran-rpart
r-cran-spatial
r-cran-stringi
r-cran-stringr
r-cran-survival
r-cran-uuid
r-doc-html
r-recommended

Uninstall all the R packages identified above in one fell swoop:

chepec@yourhost:~
$ dpkg -l | grep ^ii | awk '$2 ~ /^r-/ { print $2 }' | xargs sudo apt remove --purge -y

Clean up any remaining directories (adapt this to catch all of them):

$ sudo rm -rf /usr/lib/R
$ sudo rm -rf /etc/R

Jupyter, an alternative to RStudio Server?

One drastic solution is to abandon RStudio Server altogether and look for alternatives.

The most obvious being Jupyter notebooks, which should offer a comparable web-based interface in particular with the JupyterLab/JupyterHub module. The advantage is that Jupyter supports not only R but other kernels too. The disadvantage lies mainly in my own unfamiliarity with the JupyterHub setup.

sessionInfo()

sessionInfo()
## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] ggplot2_3.3.5  dplyr_1.0.8    magrittr_2.0.2 common_0.0.2   knitr_1.37    
## 
## loaded via a namespace (and not attached):
##  [1] pillar_1.7.0     bslib_0.3.1      compiler_4.0.5   jquerylib_0.1.4 
##  [5] tools_4.0.5      digest_0.6.29    jsonlite_1.7.3   evaluate_0.14   
##  [9] lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0     pkgconfig_2.0.3 
## [13] rlang_1.0.1      cli_3.2.0        DBI_1.1.2        yaml_2.2.2      
## [17] blogdown_1.7     xfun_0.29        fastmap_1.1.0    withr_2.4.3     
## [21] stringr_1.4.0    generics_0.1.2   vctrs_0.3.8      sass_0.4.0      
## [25] grid_4.0.5       tidyselect_1.1.2 glue_1.6.1       R6_2.5.1        
## [29] fansi_1.0.2      rmarkdown_2.11   bookdown_0.24    purrr_0.3.4     
## [33] scales_1.1.1     htmltools_0.5.2  ellipsis_0.3.2   assertthat_0.2.1
## [37] colorspace_2.0-2 utf8_1.2.2       stringi_1.7.6    munsell_0.5.0   
## [41] crayon_1.5.0

Self-hosted Gitea server

Tue, 24 Jul 2018 16:00:00 +0000

It is always a good idea to keep your projects under version control. Once you get into the habit of using Git, you might warm to the idea of pushing your repository to a web-hosted service such as Github or Gitlab. They are undoubtedly useful for collaboration, sharing and much more.

But for some projects, whether personal or business, using a hosted service is not acceptable (remember, “the cloud” is just someone else’s computer). Even so, you can still get a nice web-based front-end for your Git repos. You will just have to self-host it. Ideally on hardware that you own, but in lieu of that a VPS will do.

My own motivation for self-hosting a Git server is that I would like to have access to the capabilities such a service offers while I work on my thesis, which I’m not comfortable hosting in the cloud (not until it is done, anyway).

Currently, your choice of front-ends boils down to Gitlab CE, Gogs or Gitea.¹ Gitea is a fork of Gogs. Both of them are very light on system resources. Gitlab Community Edition is a monster in comparison. For a non-professional coder like myself, Gitea will be just fine.

The rest of this post describes how I setup Gitea on a server of mine. You can have a look at the finished product at https://git.solarchemist.se (it’s not much to look at though, as most repos are private). Before jumping off, I should note that there are already several “how to setup Gitea” guides published by others, so this will not be a step-by-step instruction. For that, I can recommend any of the following:

“How to setup Gitea on an Ubuntu Server” by Ryan Gilbert. This is a well-written, easy-to-follow step-by-step instruction. It is also the most recent.
“Setting Up a Self-Hosted Github Clone with Gitea” by Charlesreid1. Also well-written and with lots of screenshots and even an illustration.
“Set up Gitea with Nginx” by Rune Olsen. This tutorial concentrates on the settings for the web server, using Nginx as a reverse proxy for Gitea.
“How To Start your own Git Server in less than 30 Minutes” by CHEF-KOCH. Has a section with enlightening discussion on Github alternatives. Also goes through Gitea setup in some detail, including setup of PostgreSQL database.
And of course, the Gitea Docs themselves.

MySQL database for Gitea

# install mysql from ubuntu repos
sudo apt install mysql-server mysql-client libmysqlclient-dev
# check the installed mysql version
mysql --version
# secure the mysql installation
sudo mysql_secure_installation
# login to mysql as root
sudo mysql -u root -p

While logged in as MySQL root user, create the user and database for Gitea.

CREATE USER 'gitea'@'localhost' IDENTIFIED BY 'somelongandcomplexpassword';
CREATE DATABASE gitea DEFAULT CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
GRANT ALL ON gitea.* TO 'gitea'@'localhost';
FLUSH PRIVILEGES;

We explicitly set UTF8 for the Gitea database above because the Gitea docs explicitly recommend using InnoDB and UTF8 with MySQL. Recent versions of MySQL should default to using InnoDB (instead of MyISAM), but we can check that ourselves by (still logged in as MySQL root):

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                        | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| InnoDB             | DEFAULT | Supports transactions, row-level locking, and foreign keys     | YES          | YES  | YES        |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| MyISAM             | YES     | MyISAM storage engine                                          | NO           | NO   | NO         |
| CSV                | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                         | NO           | NO   | NO         |
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                             | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
9 rows in set (0.00 sec)

Apache vhost with Let’s Encrypt certificate

This virtualhost configuration is for Apache v2.4.X. Created the file /etc/apache2/sites-available/git.solarchemist.se.conf, with the following contents:

<VirtualHost *:80>
  ServerAdmin email@solarchemist.se
  ServerName git.solarchemist.se
  ServerAlias gitea.solarchemist.se

  Redirect permanent / https://git.solarchemist.se/

  ErrorLog ${APACHE_LOG_DIR}/git.solarchemist.se_error.log
  CustomLog ${APACHE_LOG_DIR}/git.solarchemist.se_access.log combined
</VirtualHost>

<VirtualHost *:443>
  ServerAdmin email@solarchemist.se
  ServerName git.solarchemist.se
  ServerAlias gitea.solarchemist.se

  SSLEngine on
  SSLCertificateKeyFile   /etc/letsencrypt/live/git.solarchemist.se/privkey.pem
  SSLCertificateFile      /etc/letsencrypt/live/git.solarchemist.se/cert.pem
  SSLCertificateChainFile /etc/letsencrypt/live/git.solarchemist.se/chain.pem

  ProxyPreserveHost On
  ProxyRequests Off
  ProxyPass / http://192.168.1.444:3000/
  ProxyPassReverse / http://192.168.1.444:3000/

  ErrorLog ${APACHE_LOG_DIR}/git.solarchemist.se_error.log
  CustomLog ${APACHE_LOG_DIR}/git.solarchemist.se_access.log combined
</VirtualHost>

Before enabling this site, we need to have DNS records setup for the defined sub-domains. Once the DNS records have propagated, we can use certbot to fetch our Let’s Encrypt certificates.

# this one-liner stops the Apache server, fetches the certs, and restarts Apache
sudo certbot certonly --authenticator standalone --pre-hook "systemctl stop apache2.service" --post-hook "systemctl start apache2.service" --email email@solarchemist.se -d git.solarchemist.se

With that out of the way, it’s time to enable the Apache site configuration:

sudo a2ensite git.solarchemist.se.conf

When everything else is setup, you can let the Apache config take effect by issuing sudo systemctl reload apache2.service.

Running Gitea as a systemd job

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=mysqld.service

[Service]
RestartSec=2s
Type=simple
User=gitea
Group=gitea
WorkingDirectory=/var/lib/gitea/
ExecStart=/usr/local/bin/gitea web -c /etc/gitea/app.ini
Restart=always
Environment=USER=gitea HOME=/home/gitea GITEA_WORK_DIR=/var/lib/gitea

[Install]
WantedBy=multi-user.target

There is nothing special here, this systemd configuration is directly from the Gitea docs. I would just like to say a few words regarding the Gitea system user. As you can see, the systemd job specifies that Gitea is running as the user gitea. This should correspond to the user in RUN_USER in the /etc/gitea/app.ini file (unless you are doing something very special, I guess).

Here I would like to point something out that I learned the hard way. The Gitea process should run in a dedicated user account, as defined in systemd or similar and /etc/gitea/app.ini. Otherwise Git over SSH (i.e., pushing/pulling/cloning of your Gitea repos) will be a complete mess, or most likely, not work at all. This was perhaps not obvious from reading the docs, but is pointed out clearly enough in the Gitea forums. Gitea also expects to handle the /home/gitea/.ssh/authorized_keys file by itself, so don’t even bother trying to setup SSH keys for this user yourself. If you did, follow the instructions here to reset this file.

Gitea paths on NFS share: a bit of a challenge

This section is particular to my setup. I would like to point the Gitea repository root path to my RAID6 array on my NAS, mainly for reasons of redundancy (this way I don’t have to worry if the HDD suddenly dies, and also, I like centralising important files, makes backups a little easier to plan).

The RAID array has always been shared over the LAN as an NFS share. Mounting it on the Gitea server is thus very easy, just add the following line to your /etc/fstab file:

192.168.1.111:/media/share   /media/share   nfs   rw,hard,intr   0   0

To help keep things straight I will from now on refer to the Gitea-running machine as the NFS client and the NAS as the NFS server.

The challenging part concerns Linux user permissions. We have already established that Gitea needs to run as its own user. This presented an interesting problem: the Gitea user on the NFS client needs write permissions on the Gitea repository path on the NFS server. How do we achieve that?

As usual when it comes to all things Linux, we have a few alternatives. In my estimation, they are:

NFS idmapping,
using the nobody/nogroup on the NFS server,
or setting up the same username (with identical uid/gid) on both the NFS client and the NFS server.

In the end, I created the same user with identical uid/gid on both machines, and then simply set that user on the NFS server as the user and group owner of the Gitea repository root tree. It went like this:

# on the NFS server, created the user 'gitea'
$ sudo adduser \
     --system \
     --shell /bin/bash \
     --gecos 'Gitea user' \
     --group \
     --disabled-password \
     --no-create-home \
     gitea
# check the uid/gid
$ id gitea
uid=126(gitea) gid=122(gitea) groups=122(gitea)

Now all we have to do is create a user gitea with the same uid/gid on the NFS client (the NFS server has more existing user accounts, so creating the user there first minimised the risk of the uid/gid clashing with an existing one on the NFS client).

# on the NFS client, created the user 'gitea'
$ sudo adduser \
     --system \
     --shell /bin/bash \
     --gecos 'Gitea user' \
     --group \
     --disabled-password \
     --home /home/gitea \
     gitea
# check the uid/gid
$ id gitea
uid=113(gitea) gid=114(gitea) groups=114(gitea)
# change user id
$ sudo usermod -u 126 gitea
# change group id
sudo groupmod -g 122 gitea

And finally, we need to fix the uid/gid of all files owned by gitea. If you know where all such files are (and in this case we do, since they should only be inside /home/gitea at this point) this can be achieved by simply reassigning ownership (chown -R gitea:gitea /home/gitea). Otherwise, you could use something like find / -uid 113 ! -type l -exec chown gitea:gitea {} \;.

Issuing id gitea on the NFS client should now display the same uid/gid as on the NFS server.

Setting up the user on both machines could be considered a violation of DRY (Don’t Repeat Yourself, although I’m not sure if that applies to systems administration), but it’s also low-maintenance and (at least for me) required the least amount of time to accomplish. I did actually try the nobody/nogroup approach, but that did not give Gitea the required permissions on the NFS path, so I moved on. And this works.

Sources

There is actually an alternative to self-hosting: encrypt the contents of your git repo and keep using a hosted service as usual. Hard to do properly on your own, though. But keybase has a neat implementation of that.↩︎

Converting between common reference electrodes

Thu, 24 Nov 2016 16:00:00 +0000

A quickly put-together post on the relationships between common electrochemical reference electrode potentials and the vacuum potential, with a proposed algorithm for converting between them.

Reference electrode potentials

Figure 1: Equilibrium potentials at $\SI{25}{\degreeCelsius}$ for some commonly used reference electrodes. Reproduced from Roberge (2008).

Figure 2: Graphical comparison of some common reference electrode potential scales. Of course, the scales are going “the wrong way” (we like negative to be up). (Roberge 2008, 67)

Figure 3: Standard potentials of silver-silver chloride and calomel electrodes at various electrolyte concentrations and temperatures. Reproduced from (Sawyer, Sobkowiak, and Roberts 1995, 192).

Standard hydrogen electrode

The SHE contribution to the cell potential is by convention zero at all temperatures. (Inzelt, Lewenstam, and Scholz 2013, 77)

The expression “normal hydrogen electrode” (NHE) is not a synonym for SHE, in fact, the NHE is the historical standard (due to Nernst in 1889) defined as a metal in contact with a solution saturated with hydrogen gas at 1 atm partial pressure and containing $\SI{1}{\mol\per\cubic\dm}$ sulfuric acid. (Inzelt, Lewenstam, and Scholz 2013, 77) The NHE is thus experimentally realisable.

In contrast, the SHE is an ideal device and cannot be rigorously realised experimentally. For practical realisation the standard potential of the hydrogen electrode is not measured at standard conditions but recalculated to $\ce{H+}$ at unit activity and $\SI{10E5}{\pascal}$. These electrodes are often called reversible hydrogen electrodes (RHE).

— Inzelt, Lewenstam, and Scholz (2013), p. 78

Calomel electrode

Decimolar (decinormal) calomel electrode, $c_\ce{KCl}=\SI{0.1}{\mol\per\litre}$, $E = \SI{0.3337}{\volt}$.
Molar (normal) calomel electrode, $c_\ce{KCl}=\SI{1}{\mol\per\litre}$, $E = \SI{0.2801}{\volt}$.
Saturated calomel electrode (SCE), $c_\ce{KCl}\approx\SI{4.2}{\mol\per\litre}$, $E = \SI{0.2412}{\volt}$.

All potential values above at $\SI{25}{\degreeCelsius}$ and taken from Bagotsky (2006), p. 194. Because of solubility changes, SCE has a large temperature coefficient ($\SI{0.65}{\mV\per\kelvin}$). The potentials of calomel electrodes can be reproduced to $\SI{+-0.1}{\mV}$.

Calomel electrodes are unstable much above $\SI{50}{\degreeCelsius}$ owing to the disproportionation reaction $\ce{Hg2Cl2 -> Hg(liq) + HgCl2}$. (eDAQ wiki)

Marcus (1980) reports $E^\circ_{\ce{Hg/Hg2Cl2}} = \SI{0.26818+-0.00002}{\volt}$ at $\SI{298.15}{\kelvin}$ ($a_{\ce{Cl^-}} = 1$).

Silver-silver chloride electrode

$E^0 = \SI{0.2224}{\volt}$ at $\SI{25}{\degreeCelsius}$ ($a_{\ce{Cl-}}=1$). (Bagotsky 2006, 195; Inzelt, Lewenstam, and Scholz 2013, 87)

The standard potential of the Ag/AgCl electrode has been reported at temperatures exceeding $\SI{100}{\degreeCelsius}$.

The absolute potential scale

Figure 4: Conversion of relative reference electrode potentials into electronic energies for aqueous systems. Reproduced from Trasatti (1986).

Trasatti (1986) shows a calculation of the absolute potential of SHE in water and recommends the following conversion:

\[\begin{equation} E^\circ(\ce{H+}/\ce{H2})^{\ce{H2O}}\text{(abs)} = \SI{4.44+-0.02}{\volt}\quad\mathrm{at}~\SI{298.15}{\kelvin} \end{equation}\]

It is to be noted that the uncertainty in the knowledge of $E^\circ(\ce{H+}/\ce{H2})\text{(abs)}$ ($\SI{+-20}{\mV}$) by far exceeds the accuracy of most measurements of electrode potentials. Therefore, any convenience in the use of the absolute potential scale is confined to the specific cases when calculations of electronic energy levels are needed.

— Trasatti (1986)

On this topic there is also an interesting discussion in Morrison (1980), p. 31.

I did a quick scan of all papers that cite Trasatti (1986), and found no more recent recommendation on the relation between SHE and AVS. For now we will stick with Trasatti’s value.

The precise relation between the potential of a free electron in vacuum (absolute vacuum scale) and the conventional standard hydrogen electrode scale for aqueous solutions is still debated. (Inzelt, Lewenstam, and Scholz 2013)

Notes

[…] mercury-containing REs (calomel or mercury–mercuric oxide) are inappropriate for measurements in conjunction with platinum electrodes, since the mercury ions readily poison platinum surfaces.

— (Bagotsky 2006, 195)

If this is a rule, we break it quite often.

Convert to or from any* electrode scale

Note: this section was updated on Oct 18th, 2019 because I moved the reference electrode functions into their own R package refelectrodes.

Having the the conversion algorithm in its own package like this should make the codebase easier to maintain and improve going forward. If you have any suggestions or comments regarding the data or the code of this package please don’t hesitate to open an issue or pull request on Github. I would be very grateful to hear any feedback.

* Any here denotes any one of the reference electrode scales that the refelectrodes package currently supports:

AgCl/Ag in saturated NaCl(aq) at RT, in saturated KCl(aq) from $\SIrange{10}{40}{\degreeCelsius}$, and in $\SI{3.5}{\mol\per\litre}$ KCl(aq) from $\SIrange{10}{40}{\celsius}$,
Hg₂Cl₂/Hg in saturated KCl(aq) from $\SIrange{10}{40}{\celsius}$, in $\SI{3.5}{\mol\per\litre}$ KCl(aq) from $\SIrange{10}{40}{\celsius}$, in $\SI{1.0}{\mol\per\litre}$ KCl(aq) from $\SIrange{10}{40}{\celsius}$, and in $\SI{0.1}{\mol\per\litre}$ KCl(aq) from $\SIrange{10}{40}{\celsius}$,
SHE (it is zero at all temperatures and concentrations by definition),
Li at RT,
Na at RT,
Mg at RT, and
the absolute vacuum scale at RT.

This data is defined in the potentials.as.SHE() function which constitutes the core of the refelectrodes package.

As for the conversion functions, those are the aptly named as.SHE() which converts from any reference electrode scale to SHE, and from.SHE() which converts from SHE to any of the other scales.

The package also exports RefCanonicalName() which is a convenience function that allows the user to specify the name of each reference electrode scale using any of its common names. And that is effectively the extent of the package. ¹

Let’s take a quick walk through loading this package and testing its functions. You will want to start by installing and loading the package:

remotes::install_github("solarchemist/refelectrodes")
library(refelectrodes)
# I suggest you load these packages if you want to replicate the plot below
library(knitr)
library(ggplot2)
library(ggrepel)
library(dplyr)
library(magrittr)

The core function defines the dataset and when run will happily return this data as a tidy dataframe (this chunk shows just an excerpt of the tibble):

potentials.as.SHE() %>%
   group_by(electrode, electrolyte) %>%
   arrange(SHE) %>%
   # return first and last row (if any) within each group
   filter(row_number() %in% c(1, n())) %>%
   arrange(electrode, SHE)
## # A tibble: 11 × 9
## # Groups:   electrode, electrolyte [8]
##    electrode electrolyte conc.num conc.string   temp    SHE sid   reference     dEdT
##    <chr>     <chr>          <dbl> <chr>        <dbl>  <dbl> <chr> <chr>        <dbl>
##  1 AgCl/Ag   "KCl(aq)"        4.2 "saturated"    40   0.184 2     Sawyer19… -1.00e-3
##  2 AgCl/Ag   "KCl(aq)"        3.5 "3.5M at 25…   10   0.215 1     Sawyer19… -7.36e-4
##  3 AgCl/Ag   "NaCl(aq)"       5.9 "saturated"    25   0.263 9     CRC 97th… NA      
##  4 AVS       ""              NA   ""             25  -4.44  7     Trasatti… NA      
##  5 Hg2Cl2/Hg "KCl(aq)"        4.2 "saturated"    40   0.234 6     Sawyer19… -6.64e-4
##  6 Hg2Cl2/Hg "KCl(aq)"        0.1 "0.1M at 25…   25   0.336 3     Sawyer19… -7.86e-5
##  7 Li        ""               1   "1.0M at 25…   25  -3.04  10    CRC 97th… NA      
##  8 Mg        ""               1   "1.0M at 25…   25  -2.37  12    CRC 97th… NA      
##  9 Na        ""               1   "1.0M at 25…   25  -2.71  11    CRC 97th… NA      
## 10 SHE       ""              NA   ""           -273.  0     8     Inzelt20…  0      
## 11 SHE       ""              NA   ""            580   0     8     Inzelt20…  0

The dataframe is tidy, so more data can easily be added to this dataset by adding more rows.

Now, some of the the columns of this dataframe warrant some explanation. potentials$electrode is the name of the reference electrode, as defined by the function RefCanonicalName() (see below). potentials$conc.string was added so that the user does not need to remember the saturation concentration for various salts in solution. And for those unsaturated electrolytes it was a good place to put information on the temperature where the concentration was defined. Saturation does not depend on temperature, but the concentration of an unsaturated solution (expressed as amount/volume) does depend on temperature due to liquid expansion. Temperature and concentration values need no further explanation. potentials$setid is mainly for housekeeping - a simple way to keep track of the datasets in the dataframe (allows us to avoid having to resort to advanced matching against several parameters to later identify a particular set).

Note that no conversion happens in the potentials.as.SHE() function above. It just returns data.

Conversion is handled by the two functions as.SHE() and from.SHE(). The take the following arguments: the potential value (in volt), the scale (for the scale you want to convert from in the case of as.SHE(), and the one you want to convert to in the case of from.SHE()), and optional arguments for concentration and temperature (set to saturated and RT by default). The last argument expects the dataframe generated by potentials.as.SHE.

This way, all three of these functions are reasonably modular, and each one does a fairly well-defined job.

Some special considerations which are handled by these functions are perhaps worth mentioning here:

concentration may be a number or a character string, which means we need to compare it to the appropriate column in the dataset,
the temperature requested by the user may coincide with a datapoint in the dataset (in that case we just return that potential), otherwise we interpolate (by linear regression) based on the existing data,
and for the final calculation of potential in SHE scale, we need to handle electronic (vacuum) and electrochemical scales separately because they go in opposite directions (this is handled by the ifelse(df$vacuum[p], ...) call.

The current implementation requires concentration to match an existing value. A functionality that could (perhaps even should) be added is the ability to interpolate concentrations in the same way as temperature (unless the change in potential with changing concentration deviates strongly from linearity).

Plotting the dataset

ggplot(potentials.as.SHE() %>%
          filter(electrolyte == "KCl(aq)")) +
   facet_wrap(~ electrode, ncol = 2) +
   geom_line(aes(x = temp, y = SHE, colour = conc.string)) +
   geom_point(aes(x = temp, y = SHE, colour =  conc.string)) +
   geom_text_repel(data = subset(potentials.as.SHE(), electrode != "AVS" & temp == 30),
                   aes(y = SHE, x = temp, label = conc.string, colour = conc.string),
                   point.padding = unit(0.25, 'lines'),
                   # hide the line connecting datapoint and label
                   segment.color = "transparent") +
   scale_y_reverse() +
   labs(x = expression("T/"*degree*C), # "T/C",
        y = "E/V vs SHE") +
   theme(legend.position = "none")

Figure 5: Electrode potentials vs SHE for the two reference electrodes in the dataset using $\ce{KCl(aq)}$ electrolyte (each panel shows one reference electrode, each line shows a different electrolyte concentration). Just to demonstrate the extent of the dataset in parameter space.

A very simple interactive demo

Open the Shiny demo on its own page

Bibliography

SessionInfo

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] refelectrodes_0.0.0.9000 common_0.0.2             magrittr_2.0.2          
## [4] dplyr_1.0.8              ggrepel_0.9.1            ggplot2_3.3.5           
## [7] knitr_1.37              
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.8       bslib_0.3.1      compiler_4.0.5   pillar_1.7.0    
##  [5] jquerylib_0.1.4  tools_4.0.5      digest_0.6.29    jsonlite_1.7.3  
##  [9] evaluate_0.14    lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_1.0.1      DBI_1.1.2        cli_3.2.0       
## [17] yaml_2.2.2       blogdown_1.7     xfun_0.29        fastmap_1.1.0   
## [21] withr_2.4.3      stringr_1.4.0    generics_0.1.2   sass_0.4.0      
## [25] vctrs_0.3.8      tidyselect_1.1.2 grid_4.0.5       glue_1.6.1      
## [29] R6_2.5.1         fansi_1.0.2      rmarkdown_2.11   bookdown_0.24   
## [33] purrr_0.3.4      scales_1.1.1     htmltools_0.5.2  ellipsis_0.3.2  
## [37] assertthat_0.2.1 colorspace_2.0-2 utf8_1.2.2       stringi_1.7.6   
## [41] munsell_0.5.0    crayon_1.5.0

References

Bagotsky, V. S. 2006. Fundamentals of Electrochemistry. 2nd ed. Wiley-Interscience.

Inzelt, György, Andrzej Lewenstam, and Fritz Scholz, eds. 2013. Handbook of Reference Electrodes. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-36188-3.

Marcus, Y. 1980. “Compilation and Evaluation of Solubility Data in the Mercury (I) Chloride-Water System.” Journal of Physical and Chemical Reference Data 9 (4): 1307. https://doi.org/10.1063/1.555633.

Morrison, S. Roy. 1980. Electrochemistry at Semiconductor and Oxidized Metal Electrodes. 1st ed. Plenum press.

Roberge, Pierre R. 2008. Corrosion Engineering Principles and Practice. Corrosion Engineering. https://doi.org/10.1036/0071482431.

Sawyer, Donald T., Andrzej Sobkowiak, and Julian L. Roberts. 1995. Electrochemistry for Chemists. Second. John Wiley & Sons, Inc.

Trasatti, S. 1986. “The Absolute Electrode Potential: An Explanatory Note (Recommendations 1986).” Pure and Applied Chemistry 58 (7): 955–66. https://doi.org/10.1351/pac198658070955.

The package also contains a slew of deprecated functions that are clearly marked as such and may be removed in a future update.↩︎

Lamps and filters: spectral baselines and total power

Sat, 23 Apr 2016 20:00:00 +0000

Naming: Copacabana and Miramar

To quickly identify each setup, we have named them. From now on, the arc lamps shall be known as “Copacabana” and “Miramar”.

Figure 1: Where the Sun always shines…

Short note on history: Copacabana has a quite old bulb, and its intensity has clearly decreased over time. I re-adjusted the bulb and reflector of Copacabana today. Copacabana is equipped with a filter holder (attached to the lens tube) which holds a square AM1.5 filter.

Miramar has a new bulb, installed just a few days ago. Miramar does not come equipped with any attached filters.

Both lamps are set to operate at $\SI{200}{\watt}$ (Miramar was previously set to $\SI{250}{\watt}$ but was reset to the lower power limit after exhibiting a light fault error message and failing to ignite at the higher power).

Methods

Setup: total power using pyranometer

Figure 2: I measured the power output of Copacabana at varying distances.

Copacabana is equipped with a filter holder (on the lens tube) holding a square AM1.5 filter.

Figure 3: I measured the power output of Miramar at varying distances with an AM1.5 filter in the optical path.

I measured the power from the arc lamp at a range of distances using a Thorlabs PM160T power meter.

sensor.diameter <- 1.0 # cm
sensor.area <- pi * (0.5 * sensor.diameter)^2
area.corr.factor <- 1 / sensor.area

The sensor area is $\SI{0.7854}{\square\cm}$, and consequently we need to multiply the values we read out with a factor 1.273 to get power densities ($\si{\milli\watt\per\square\cm}$).

It is also pertinent to mention how distances between lamp and sensor were measured.

Figure 4: Sketch relating $d$ and $d_\text{corr}$ during measurement of total power

\[\begin{equation} d_\text{corr} = d + \frac{165}{2} + \frac{63}{2} = d + 114 \end{equation}\]

For practical reasons it was easier to measures the distances from one edge of each object to another, but for plotting and calculations I deemed it more correct to use centre-to-centre distance, which I termed $d_\text{corr}$ (for corrected).

Setup: spectral profile of AM-filters

Measured the transmittance and absorbance of air-mass filters using the OceanOptics fibre-optic spectrometer.

AM filters:

AM1.5 filter, square, usually mounted in arc lamp “Copacabana”
AM1.5 filter, circular, loan from Andreas Mattsson
AM0 filter, square (with small crack), loan from Andreas Mattsson

Note: according to Mattsson, his filters will produce AM1.5 when used in conjunction (both filters in the optical path).

Figure 5: Measuring the spectral response of the air-mass filters.

Setup: spectral profile of ND-filters

Measured the transmittance of neutral density filters using the OceanOptics fibre-optic spectrometer.

Photos of the ND-filter assembly. During each measurement the secondary ring was positioned so that an empty slot faced the optical beam. Combinations of filters were not measured (should be straight-forward to produce numerically if required).

Setup: spectral distribution of lamps

Characterised the spectra of both xenon arc lamps. The power of both lamps was high enough to saturate the spectrometer’s detector at all practical distances, so I did the spectral measurements with a 1% neutral density filter (that’s the wheel in the photos below) placed in front of the detector.

Figure 6: Measuring the spectral distribution of Miramar lamp.

The detector was at $d=\SI{70}{\cm}$ (ie, $d_\text{corr}=\SI{81.4}{\cm}$), although that has little bearing on the results as we only want to compare spectral distributions of the lamps (power is compared via the pyranometer readings).

Figure 7: Measuring the spectral distribution of Copacabana lamp.

The detector was at $d=\SI{60}{\cm}$ (ie, $d_\text{corr}=\SI{71.4}{\cm}$), although that has little bearing on the results as we only want to compare spectral distributions of the lamps (power is compared via the pyranometer readings).

Results

Total power of lamps

power <-
   cbind(lamp = "copacabana",
         read.csv(file = here::here("assets/copacabana-distance-vs-power.csv")))
power <-
   rbind(power,
         cbind(lamp = "miramar",
               read.csv(file = here::here("assets/miramar-distance-vs-power.csv"))))
power$corr.distance <- power$distance + 11.4 # cm
power$density <- round(power$power * area.corr.factor)

this.lamp <- "copacabana"
fit.copacabana <-
   lm(density ~ logb(corr.distance, base = exp(1)),
      data = subset(power, lamp == this.lamp))
x.copacabana <-
   c(seq(1, 41),
     subset(power, lamp = this.lamp)$corr.distance)
power.predicted.copacabana <-
   predict(fit.copacabana,
           newdata =
              data.frame(corr.distance = x.copacabana))
power.calc.copacabana <-
   data.frame(lamp = this.lamp,
              distance = x.copacabana,
              density = power.predicted.copacabana)

this.lamp <- "miramar"
fit.miramar <-
   lm(density ~ logb(corr.distance, base = exp(1)),
      data = subset(power, lamp == this.lamp))
x.miramar <-
   c(seq(1, 41),
     subset(power, lamp = this.lamp)$corr.distance)
power.predicted.miramar <-
   predict(fit.miramar,
           newdata =
              data.frame(corr.distance = x.miramar))
power.calc.miramar <-
   data.frame(lamp = this.lamp,
              distance = x.miramar,
              density = power.predicted.miramar)

# join power.calc dataframes
power.calc <-
   rbind(power.calc.copacabana, power.calc.miramar)

Figure 8: Caption.

What is the distance at which each lamp yields total power corresponding to 1 Sun, ie $\SI{100}{\milli\watt\per\square\cm}$? Calculated by linear interpolation of the two points straddling the 100-mark for each curve (we’re not using the fitted exponential functions because I’m having some trouble applying the parameters).

Miramar’s “1 Sun” distance is $\SI{57}{\cm}$ (centre-to-centre).

linfit.miramar <-
   approx(x = c(min(subset(subset(power, lamp == "miramar"), density <= 100)$corr.distance),
                max(subset(subset(power, lamp == "miramar"), density >= 100)$corr.distance)),
          y = c(min(subset(subset(power, lamp == "miramar"), density >= 100)$density),
                max(subset(subset(power, lamp == "miramar"), density <= 100)$density)),
          n = length(seq(max(subset(subset(power, lamp == "miramar"), density >= 100)$corr.distance),
                         min(subset(subset(power, lamp == "miramar"), density <= 100)$corr.distance),
                         by = 0.1)))
linfit.miramar$x[which(round(linfit.miramar$y) == 100)]
## [1] 55.5 55.6

And this is Copacabana’s “1 Sun” distance:

linfit.copacabana <-
   approx(x = c(min(subset(subset(power, lamp == "copacabana"), density <= 100)$corr.distance),
                max(subset(subset(power, lamp == "copacabana"), density >= 100)$corr.distance)),
          y = c(min(subset(subset(power, lamp == "copacabana"), density >= 100)$density),
                max(subset(subset(power, lamp == "copacabana"), density <= 100)$density)),
          n = length(seq(max(subset(subset(power, lamp == "copacabana"), density >= 100)$corr.distance),
                         min(subset(subset(power, lamp == "copacabana"), density <= 100)$corr.distance),
                         by = 0.1)))
linfit.copacabana$x[which(round(linfit.copacabana$y) == 100)]
## [1] 44.6

We’ll round that to $\SI{45}{\cm}$.

Spectral response of air-mass filters

abs.files <-
   list.files(path = "/media/bay/taha/chepec/laboratory/OOHR2000/160423",
              pattern = "filter-A-.*\\.txt$",
              full.names = TRUE)
abs.data <-
   OO2df(datafile = abs.files[1], version = "1")
for (i in 2:length(abs.files)) {
   abs.data <-
      rbind(abs.data,
            OO2df(datafile = abs.files[i], version = "1"))
}
abs.data$technique <- "abs"

tra.files <-
   list.files(path="/media/bay/taha/chepec/laboratory/OOHR2000/160423",
              pattern = "filter-T-.*\\.txt$",
              full.names = TRUE)
tra.data <-
   OO2df(datafile = tra.files[1], version = "1")
for (i in 2:length(tra.files)) {
   tra.data <-
      rbind(tra.data,
            OO2df(datafile = tra.files[i], version = "1"))
}
tra.data$technique <- "tra"

# some clean-up to simplify plotting
tra.data <- subset(tra.data, (intensity <= 100) & (intensity >= 0))
abs.data <- subset(abs.data, intensity > -0.01)
# join abs and tra dataframes
exp.data <- rbind(abs.data, tra.data)
# trim noisy ends of spectra
exp.data <- subset(exp.data, (wavelength <= 1050) & (wavelength >= 260))
# rename the sampleids
exp.data$sampleid[which(exp.data$sampleid
                        %in% c("AM0-AM15-filter-A-AMattson",
                               "AM0-AM15-filter-T-AMattson"))] <-
   "AM0+AM1.5"
exp.data$sampleid[which(exp.data$sampleid
                        %in% c("AM0-filter-A-AMattson", "AM0-filter-T-AMattson"))] <-
   "AM0"
exp.data$sampleid[which(exp.data$sampleid
                        %in% c("AM15-filter-A-AMattson", "AM15-filter-T-AMattson"))] <-
   "AM1.5-circ"
exp.data$sampleid[which(exp.data$sampleid
                        %in% c("AM15-filter-A-Copacabana", "AM15-filter-T-Copacabana"))] <-
   "AM1.5-sq"

Figure 9: Caption.

Looking only at the $\SIrange{300}{400}{\nm}$ range:

Figure 10: Absorbance and transmittance spectra of the AM filters ($\SIrange{300}{400}{\nm}$).

Using both the AM0 and AM1.5 (circular) filters in the optical path decrease transmittance markedly more than the AM1.5 (square) filter that we have used before.

Perhaps using only Mattson’s AM1.5 filter is the best compromise.

Spectral response of neutral density filters

nd.files <-
   list.files(path = "/media/bay/taha/chepec/laboratory/OOHR2000/171122-ND-filters",
              pattern = "\\.txt$",
              full.names = TRUE)
nd.data <- NULL
for (i in 1:length(nd.files)) {
   nd.data <-
      rbind(nd.data,
            OO2df(datafile = nd.files[i], version = "1"))
}
nd.data$technique <- "tra"

# some clean-up to simplify plotting
nd.data <-
   nd.data %>%
   filter(intensity <= 100 & intensity >= 0) %>%
   # trim noisy ends of spectra
   filter(wavelength >= 260 & wavelength <= 1050)
# a nicer label for the plot
nd.data$label <- paste0(sub("\\d{3}-ND-", "T = ", nd.data$sampleid), "%")

Figure 11: Spectral profile of each ND filter. Their absorbance is pretty constant within the range $\SIrange{400}{800}{\nm}$. But below $\SI{400}{\nm}$ their absorbance increases steeply.

Spectral distribution of lamps

# Copacabana with its "built-in" filter
lamp <-
   cbind(lamp = "Copacabana",
         filter = "AM1.5-sq",
         OO2df("/media/bay/taha/chepec/laboratory/OOHR2000/160423/Copacabana-lamp.txt",
               version = "1"))
# Miramar (which lacks built-in filter) while borrowing Copacabana's filter
lamp <-
   rbind(lamp,
         cbind(lamp = "Miramar",
               filter = "AM1.5-sq",
               OO2df("/media/bay/taha/chepec/laboratory/OOHR2000/160423/Miramar-lamp-AM15-Copacabana.txt",
                     version = "1")))
# Miramar with the circular AM1.5 filter
lamp <-
   rbind(lamp,
         cbind(lamp = "Miramar",
               filter = "AM1.5-circ",
               OO2df("/media/bay/taha/chepec/laboratory/OOHR2000/160423/Miramar-lamp-AM15-Mattson.txt",
                     version = "1")))
# Miramar with both of Mattson's filters (AM1.5 and AM0)
lamp <-
   rbind(lamp,
         cbind(lamp = "Miramar",
               filter = "AM0+AM1.5",
               OO2df("/media/bay/taha/chepec/laboratory/OOHR2000/160423/Miramar-lamp-AM15-AM0-Mattson.txt",
                     version = "1")))
# Finally, Miramar without any AM filters at all
lamp <-
   rbind(lamp,
         cbind(lamp = "Miramar",
               filter = "none",
               OO2df("/media/bay/taha/chepec/laboratory/OOHR2000/160423/Miramar-lamp.txt",
                     version = "1")))

Please keep in mind that the total power (area under the curve) in these spectra are not comparable between the two lamps since I did not bother placing the sensor at the same distance from the lamps. I was mainly interested in the spectral distribution.

Figure 12: Caption.

Figure 13: Spectral distribution of both lamps with the same AM1.5 filter in the optical path. Clearly, the spectral distribution is not affected by the age of the bulb (Copacabana ca 5 years old, Miramar brand new).

Actually, the main take-away from these measurements are that both lamps (new and old) produce the same spectral distribution, even though the stability and power of the new lamp is significantly better than the old.

Let’s have a closer look at short wavelengths:

Figure 14: Caption.

Keep in mind that the shown counts have not been zeroed, and that the zero-level is usually at approximately 650 counts.

Export experimental data to assets folder

# Note to self: use zip, not gzip, for compressing multiple files into one zip archive
system(paste("zip -j",
             here::here("assets/lnb-160423-filters.zip"),
             paste(list.files(pattern = "^.*-filter-.*\\.txt$",
                              full.names = TRUE,
                              "/media/bay/taha/chepec/laboratory/OOHR2000/160423"),
                   collapse = " ")))

Session info

## Commit:  0057708e5b134d9aff01a590348209be7d467dc9
## Author:  taha@luxor <taha@chepec.se>
## When:    2021-03-07 00:02:33 GMT
## 
##      Fixed the path to an asset
##      
##      (the mistake was introduced when assets were bundled...)
##      
## 1 file changed, 1 insertions, 1 deletions
## index.Rmd | -1 +1  in 1 hunk
## Untracked files:
##  Untracked:  index.Rmd.lock~
## 
## Unstaged changes:
##  Modified:   index.Rmd
## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
##  [1] here_1.0.1             oceanoptics_0.0.0.9004 common_0.0.2          
##  [4] git2r_0.29.0           directlabels_2021.1.13 ggrepel_0.9.1         
##  [7] ggplot2_3.3.5          dplyr_1.0.8            magrittr_2.0.2        
## [10] knitr_1.37            
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.8       pillar_1.7.0     bslib_0.3.1      compiler_4.0.5  
##  [5] jquerylib_0.1.4  tools_4.0.5      digest_0.6.29    jsonlite_1.7.3  
##  [9] evaluate_0.14    lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_1.0.1      cli_3.2.0        DBI_1.1.2       
## [17] yaml_2.2.2       blogdown_1.7     xfun_0.29        fastmap_1.1.0   
## [21] withr_2.4.3      stringr_1.4.0    generics_0.1.2   vctrs_0.3.8     
## [25] sass_0.4.0       rprojroot_2.0.2  tidyselect_1.1.2 glue_1.6.1      
## [29] R6_2.5.1         fansi_1.0.2      rmarkdown_2.11   bookdown_0.24   
## [33] purrr_0.3.4      scales_1.1.1     ellipsis_0.3.2   htmltools_0.5.2 
## [37] assertthat_0.2.1 colorspace_2.0-2 quadprog_1.5-8   utf8_1.2.2      
## [41] stringi_1.7.6    munsell_0.5.0    crayon_1.5.0
## Linux 4.15.0-173-generic #182-Ubuntu SMP Fri Mar 18 15:53:46 UTC 2022 x86_64

Extract SEM parameters from TIFF tags using R

Fri, 20 Mar 2015 16:00:00 +0100

Introduction

Scanning electron microscopy (SEM) is a powerful technique that is regularly used by chemists and other scientists to characterize an array of physical properties of a wide range of solid materials.

TIFF stands for Tagged Image File Format. Other than the image itself, the software can insert multiple tags with additional information, such as objective settings, acceleration voltage, stage position, etc. Timely recording of the experimental conditions is very important, so this feature of TIFF files is potentially very useful.

This post will demonstrate a method to extract TIFF tags from SEM micrographs using the R programming language. The method is particularly well-suited for SEM micrographs recorded using the SmartSEM® Operating Software¹.

Unfortunately, the Zeiss SmartSEM® portal offers no download link to the software manual nor any other documentation on the software’s capabilities. And after perusing the SmartSEM® V05.06 and V05.04 software manuals, I found no description of the tags that the SEM software demonstrably attaches to the TIFF images.

But the manual does note that “images in BMP and JPEG format cannot be reloaded to the SmartSEM® user interface. Besides, it is not possible to save additional information with the image”. And in table 2.2 (“Available licenses”) of the manual, we learn of a “SmartSEM report generator” that “enables an Office 2007 add in ribbon that imports CZTIFF images and can read the tags so users can create reports”. But I have not tested the functionality of this report generator.

The only Zeiss-issued documentation related to TIFF files that we could find was a “technical information” sheet titled “The Tag Imaged File Format in the LSM” by Bernhard Reissner (1994) hosted on the research blog of former KTH PhD student Erland Lewin. The information in the document is interesting and sheds some light on the origins of attaching additional information in TIFF tags, but is severely outdated.

Notwithstanding the poor documentation offered by the SmartSEM® publisher, other microscopists have created scripts, plugins and software that can handle TIFF images to various degrees.

Most ambitious is probably tiffile.py by Christoph Gohlke (2014), a very well-documented Python script for reading and writing data from and to TIFF files. The script is geared towards handling the TIFF format in general, and does not collect the specific tags generated by the SmartSEM® software, but it could probably easily be expanded upon by someone with Python familiarity.

There is a MATLAB® function (should also work in Octave) that handles TIFF files, although it is mostly geared towards biophysics and LSM applications. tiffread.m v3.31 (licensed as GNU GPL v3) imports TIFF files into MATLAB/Octave. By Francois Nedelec of the Cell biology and Biophysics unit at EMBL (2013). Older versions of the same script are also found online [1, 2].

Then there is also a few ImageJ plugins that can read or dump TIFF tags,

and a SetScaleFromTiffTag ImageJ macro that (in conjunction with the TIFF tags ImageJ plugin above) can set the scale for SEM images taken with the Zeiss SmartSEM® software.

The TIFF format supports a few different kind of tags, specifically,

baseline tags are those that are listed as part of the core of TIFF, the essentials that all mainstream TIFF developers should support in their products, extension tags are those listed as part of TIFF features that may not be supported by all TIFF readers, private tags are, at least originally, allocated by Adobe for organizations that wish to store information meaningful only to that organization in a TIFF file. […], private IFD tags if one needs more than 10 private tags or so, the TIFF specification suggests that, rather then using a large amount of private tags, one should instead allocate a single private tag, define it as datatype IFD, and use it to point to a so called “private IFD”. In that private IFD, one can next use whatever tags one wants. These private IFD tags do not need to be properly registered with Adobe, they live in a namespace of their own, private to the particular type of IFD. [boldface added]

The above section is quoted from Joris van Damme who maintains a list of known tags and their descriptions and definitions.

SmartSEM® TIFF images

Below is a SEM micrograph recorded using Zeiss SmartSEM® V05.04.05.00 on a LEO1550 microscope (at the Ångström Microstructure Laboratory, Uppsala university).

Short note: most TIFF editors honour the TIFF specification and discard tags/fields/IFD they do not understand. GIMP does too, and any edits made in GIMP on the original TIFF will cause all SmartSEM® tags to be discarded.

So if we want to read the tags that SmartSEM® so conveniently attaches to the TIFF images, we need to avoid editing the TIFF image or do our bidding before any edits.

To access the tags from the command line, we can use the TIFF tools collection, specifically, the tiffinfo utility. They are easily installed on GNU/Linux Debian-based systems:

sudo apt-get install libtiff-tools

The extracted TIFF tags are just text (with some non-UTF8 characters, unfortunately), and we can easily write them to a file using piping.

tiffinfo micrograph.tif > tags.txt

For both software versions, tiffinfo reports that the tags are in a field with tag 34118 (0x8546), which indicates that SmartSEM® uses a so called private IFD tag to attach its array of parameters. It is also encouraging to see that the newer SmartSEM® version significantly expands the number of recorded parameters. Please note that we have removed the following parameters from the TIFF tag files above for privacy: SV_FILE_NAME, SV_USER_NAME, SV_OPERATOR, SV_USER_TEXT, SV_SERIAL_NUMBER, SV_IMAGE_PATH, SV_SAMPLE_ID, AP_DATE, AP_TIME.

We encourage the reader to inspect the SEM parameters stored in the TIFF tags, to get a feel for the scope of data that is recorded by the microscope.

Advantages of utilising the SEM parameters stored in TIFF tags

Although this particular SEM operating software (and probably others) record all pertinent instrument parameters into each TIFF image, I suspect that most users are not aware of this fact and still rely on taking notes in a lab notebook or similar, in addition to the few parameters that are usually embedded visually into the image itself (scalebar, magnification, acceleration voltage, and such, see example micrograph above).

The extracted TIFF tags incorporated into a LaTeX-generated micrograph report.

We have used a system where each micrograph was added to a SEM experiment report generated by LaTeX, with the a selection of the most relevant parameters included on the same page (see example above). It has worked well, but the report can of course not be generated until the SEM session is finished and all micrographs have been collected. As such, it is not a substitute for the lab notebook. A nice improvement would be to have the report page generated on a portable computer or tablet immediately after the image is saved by the microscope, and even better would be if the tablet or similar would allow the user to scribble notes on that generated report page. But I guess that is what an electronic lab notebook system is for…

Saving all TIFF tags to an R dataframe

The following is an R function that extracts all private IFD tags attached to SmartSEM®-produced TIFF images and saves them in a R dataframe and also separates quantities and units into separate columns for easier data treatment.

tiffalltags2df.R

As written, tiffalltags2df() expects the text output of tiffinfo as input. This could easily be generated for an arbitrary number of existing TIFF files inside a directory tree using the following code snippet.

tagfiles <- list()
tif <- # all laboratory SEM TIFF files
   data.frame(files =
                 # the find command searches recursively into subdirectories
                 system("find /path/with/TIFF/files -type f -exec ls {} \\; | grep .tif",
                        intern = TRUE))
tif$tagfiles <- sub("tif$", "txt", tif$files)
for (i in 1:dim(tif)[1]) {
   message(paste("Saving tags for:", tif$files[i]))
   # generate TIFF tag text files for each TIFF image
   system(paste("tiffinfo", tif$files[i], ">", tif$tagfiles[i]),
          ignore.stderr = TRUE)
   tagfiles[[i]] <-
      # note that iconv() forcefully converts non-ASCII to ASCII
      # I have noted that degree symbols are lost in this conversion
      iconv(readLines(con = tif$tagfiles[i]), to = "ASCII", sub = "")
}

Now that we have a bunch of TIFF images of SEM micrographs, and we know about the amount of information stored inside each image’s TIFF tags, what can we do with them?

Exploring the SEM dataset

if (!file.exists(here::here("assets/tiffiles.rda"))) {
   tif <- # all laboratory SEM TIFF files
      data.frame(files =
                    system("find /media/bay/taha/chepec/laboratory/SEM -type f -exec ls {} \\; | grep .tif",
                           intern = TRUE))
   tif$files <- sub("tif$", "txt", tif$files)
   # drop any textfile that does not exist (check for the sake of completeness)
   tif$files <- tif$files[file.exists(tif$files)]
   save(tif, file = here::here("assets/tiffiles.rda"))
} else {
   load(here::here("assets/tiffiles.rda"))
}
if (!file.exists(here::here("assets/tags.rda"))) {
   tags <- tiffalltags2df(tif$files[1])
   for (i in 2:dim(tif)[1]) {
      # this loop can take a while
      message(paste("Reading file", i, "of", dim(tif)[1]))
      tags <- rbind(tags, tiffalltags2df(tif$files[i]))
      }
   save(tags, file = here::here("assets/tags.rda"))
} else {
   load(here::here("assets/tags.rda"))
}

Our dataset consists of 927 SEM micrographs, recorded from 2011 to 2014. 672 micrographs were recorded using the LEO1550 microscope, and 255 micrographs were recorded using the Merlin microscope.

We will start this exploration by looking at the brightness vs contrast setting for all SEM micrographs. Since the majority of all SEM experiments were performed on similar samples (solid metal oxides of similar composition and morphology), we can assume that there should be a “sweet spot” of B and C values that the operator keeps repeating for best results. Also, we know from looking at the Auto BC parameter that BC was set manually for 96.9% of all micrographs.

Brightness vs. contrast using two different microscopes. The same data plotted against date shows that there is also an apparent improvement in operator proficiency over time.

Other than clear evidence of the hypothesized “sweet spot”, there is also an observable difference in B and C value clustering between the different microscopes. Although that could also be attributed to improved operator proficiency.

Here is another plot that highlights operator behaviour when recording micrographs. Number of micrographs consistently drop off with increasing magnification, with a few notable peaks at 35k and 90k.

Magnification of all recorded SEM micrographs. Low magnifications quite reasonable dominate the total number of recorded micrographs (every sample is explored at low magnification before zooming in).

And finally, a plot that explores the instruments more than the operator.

Pixel size vs. magnification. The difference between the two microscopes is interesting. Is it real, or just the result of different calibration?

Conclusion

If your scanning electron microscope uses Zeiss SmartSEM® software, you can take advantage of the fact that each TIFF image contains all the instrument parameters. In this post we have shown how those TIFF tags can be retrieved and saved using the R programming. We then used the aggregated TIFF tags from more than 900 of our own SEM images to investigate the relationship between a few parameters.

With the inclusion of what appears to be all instrument parameters in the SEM TIFF file, Zeiss has managed to set a good and bad example simultaneously. Good because it uses an established and open format, bad because the documentation is sorely lacking. Still, let’s hope that more scientific instrument makers emulate Zeiss’ example and attach the full set of instrument parameters to each output file, or at least make it optional.

If you liked this post and would like to see more like it, comment or contact me.

sessionInfo()

sessionInfo()
## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] common_0.0.2   plotly_4.10.0  reshape2_1.4.4 ggplot2_3.3.5  knitr_1.37    
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_1.1.2  xfun_0.29         bslib_0.3.1       purrr_0.3.4      
##  [5] colorspace_2.0-2  vctrs_0.3.8       generics_0.1.2    htmltools_0.5.2  
##  [9] viridisLite_0.4.0 yaml_2.2.2        utf8_1.2.2        rlang_1.0.1      
## [13] jquerylib_0.1.4   pillar_1.7.0      glue_1.6.1        withr_2.4.3      
## [17] DBI_1.1.2         lifecycle_1.0.1   plyr_1.8.6        stringr_1.4.0    
## [21] munsell_0.5.0     blogdown_1.7      gtable_0.3.0      htmlwidgets_1.5.4
## [25] evaluate_0.14     fastmap_1.1.0     fansi_1.0.2       Rcpp_1.0.8       
## [29] scales_1.1.1      jsonlite_1.7.3    digest_0.6.29     stringi_1.7.6    
## [33] bookdown_0.24     dplyr_1.0.8       grid_4.0.5        cli_3.2.0        
## [37] tools_4.0.5       magrittr_2.0.2    sass_0.4.0        lazyeval_0.2.2   
## [41] tibble_3.1.6      crayon_1.5.0      tidyr_1.1.4       pkgconfig_2.0.3  
## [45] ellipsis_0.3.2    data.table_1.14.2 assertthat_0.2.1  rmarkdown_2.11   
## [49] httr_1.4.2        R6_2.5.1          compiler_4.0.5

SmartSEM® V05.06 Operating Software for Scanning Electron Microscopes by Carl Zeiss Microscopy GmbH. Software manual [pdf, 15 MB].↩︎

All available elemental properties plotted as periodic tables

Fri, 30 Jan 2015 20:00:00 +0100

Introduction

In the previous post we showed how ggplot2 can be used to effectively and quickly render a periodic table overlaid with near-arbitrary elemental data.

Let’s use this newfound ability to visualise all the datasets we have available to us.

library(ggplot2)
library(dplyr)
library(grid)
library(knitr)
library(common)

Available elemental properties

Some properties are not suited for this type of visualisation, and will be excluded (see donotplot vector below). Others are better visualised as discrete variables, although they are numeric in format. Such properties will be converted back to character (see numericasdiscrete below).

# drop some redundant columns
values <- values[, -grep(pattern = "^Absolute.*", names(values))]
units <- units[, -grep(pattern = "^Absolute.*", names(units))]
# some columns are not suitable for this format of visualisation
donotplot <-
   c("Name",
     "Symbol",
     "Group",
     "Graph.Group",
     "Period",
     "Graph.Period",
     "EU_Number",
     "RTECS_Number",
     "Alternate_Names",
     "Electron_Configuration",
     "CAS_Number",
     "CID_Number",
     "Gmelin_Number",
     "NSC_Number",
     "Quantum_Numbers",
     "Space_Group_Name",
     "Space_Group_Number")
# some numeric columns are better visualised as discrete variables
numericasdiscrete <-
   c("DOT_Hazard_Class",
     "NFPA_Fire_Rating",
     "NFPA_Health_Rating",
     "NFPA_Reactivity_Rating")
for (i in 1:length(numericasdiscrete)) {
   values[, which(names(values) == numericasdiscrete[i])] <-
      as.character(values[, which(names(values) == numericasdiscrete[i])])
}

Here are all the elemental properties that have been deemed suitable for visualisation.

str(values)
## 'data.frame':    118 obs. of  80 variables:
##  $ Name                          : chr  "Hydrogen" "Helium" "Lithium" "Beryllium" ...
##  $ Symbol                        : chr  "H" "He" "Li" "Be" ...
##  $ Atomic_Number                 : num  1 2 3 4 5 6 7 8 9 10 ...
##  $ Atomic_Weight                 : num  1.01 4 6.94 9.01 10.81 ...
##  $ Density                       : num  8.99e-02 1.78e-01 5.35e+02 1.85e+03 2.46e+03 ...
##  $ Melting_Point                 : num  14 NA 454 1560 2348 ...
##  $ Boiling_Point                 : num  20.28 4.22 1615.15 2743.15 4273.15 ...
##  $ Phase                         : chr  "Gas" "Gas" "Solid" "Solid" ...
##  $ Critical_Pressure             : num  1293000 NA NA NA 3390000 ...
##  $ Critical_Temperature          : num  32.97 5.19 3223 NA NA ...
##  $ Heat_of_Fusion                : num  0.558 0.02 3 7.95 50 105 0.36 0.222 0.26 0.34 ...
##  $ Heat_of_Vaporization          : num  0.452 0.083 147 297 507 715 2.79 3.41 3.27 1.75 ...
##  $ Heat_of_Combustion            : num  NA NA -298 NA NA -393 NA NA NA NA ...
##  $ Specific_Heat                 : num  14300 NA 1820 240 1040 165 1030 154 904 140 ...
##  $ Adiabatic_Index               : chr  "7/5" "5/3" NA NA ...
##  $ Neel_Point                    : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ Thermal_Conductivity          : num  0.18 0.151 85 190 27 ...
##  $ Thermal_Expansion             : num  NA NA 4.60e-05 1.13e-05 6.00e-06 7.10e-06 NA NA NA NA ...
##  $ Density_Liquid                : num  NA NA 512 1690 2080 NA NA NA NA NA ...
##  $ Molar_Volume                  : num  1.12e-02 2.24e-02 1.30e-05 4.88e-06 4.39e-06 ...
##  $ Brinell_Hardness              : num  NA NA NA 6e+08 NA NA NA NA NA NA ...
##  $ Mohs_Hardness                 : num  NA NA 600000 5500000 9300000 500000 NA NA NA NA ...
##  $ Vickers_Hardness              : num  NA NA NA 1.67e+09 4.90e+10 ...
##  $ Bulk_Modulus                  : num  NA NA 1.1e+10 1.3e+11 3.2e+11 ...
##  $ Shear_Modulus                 : num  NA NA 4.20e+09 1.32e+11 NA ...
##  $ Young_Modulus                 : num  NA NA 4.90e+09 2.87e+11 NA ...
##  $ Poisson_Ratio                 : num  NA NA NA 0.032 NA NA NA NA NA NA ...
##  $ Refractive_Index              : num  1 1 NA NA NA ...
##  $ Speed_of_Sound                : num  1270 970 6000 13000 16200 ...
##  $ Valence                       : num  1 0 1 2 3 4 3 2 1 0 ...
##  $ Electronegativity             : num  2.2 NA 0.98 1.57 2.04 2.55 3.04 3.44 3.98 NA ...
##  $ ElectronAffinity              : num  72.8 0 59.6 0 26.7 ...
##  $ Autoignition_Point            : num  809 NA 452 NA NA ...
##  $ Flashpoint                    : num  255 NA NA NA NA ...
##  $ DOT_Hazard_Class              : chr  "2.1" "2.2" "4.3" "6.1" ...
##  $ DOT_Numbers                   : num  1966 1963 1415 1567 NA ...
##  $ EU_Number                     : chr  "EU215-605-7" "EU231-168-5" "EU231-102-5" "EU231-150-7" ...
##  $ NFPA_Fire_Rating              : chr  "4" "0" "2" "1" ...
##  $ NFPA_Health_Rating            : chr  "3" "1" "3" "3" ...
##  $ NFPA_Reactivity_Rating        : chr  "0" "0" "2" "0" ...
##  $ RTECS_Number                  : chr  "RTECSMW8900000" NA "RTECSDS1750000" "RTECSLW3850000" ...
##  $ Alternate_Names               : chr  NA NA NA NA ...
##  $ Block                         : chr  "s" "p" "s" "s" ...
##  $ Group                         : chr  "1" "18" "1" "2" ...
##  $ Period                        : chr  "1" "1" "2" "2" ...
##  $ Electron_Configuration        : chr  "1s1" "[Xe]4f56s2" "[He]2s2" "[Xe]4f75d16s2" ...
##  $ Color                         : chr  "Colorless" "Colorless" "Silver" "SlateGray" ...
##  $ Gas_phase                     : chr  "Diatomic" "Monoatomic" NA NA ...
##  $ CAS_Number                    : chr  "CAS1333-74-0" "CAS7440-59-7" "CAS7439-93-2" "CAS7440-41-7" ...
##  $ CID_Number                    : chr  "CID783" "CID23987" "CID3028194" "CID5460467" ...
##  $ Gmelin_Number                 : chr  "Gmelin3" "Gmelin16294" "Gmelin30" "Gmelin16256" ...
##  $ NSC_Number                    : chr  NA NA NA NA ...
##  $ Electrical_Type               : chr  NA NA "Conductor" "Conductor" ...
##  $ Electrical_Conductivity       : num  NA NA 1.1e+07 2.5e+07 1.0e-04 1.0e+05 NA NA NA NA ...
##  $ Resistivity                   : num  NA NA 9.4e-08 4.0e-08 1.0e+04 1.0e-05 NA NA NA NA ...
##  $ Superconducting_Point         : num  NA NA NA 0.026 NA NA NA NA NA NA ...
##  $ Magnetic_Type                 : chr  "Diamagnetic" "Diamagnetic" "Paramagnetic" "Diamagnetic" ...
##  $ Curie_Point                   : num  NA NA NA NA NA NA NA NA NA NA ...
##  $ Mass_Magnetic_Susceptibility  : num  -2.48e-08 -5.90e-09 6.30e-09 -1.26e-08 -8.70e-09 ...
##  $ Molar_Magnetic_Susceptibility : num  -2.50e-11 -2.36e-11 4.37e-11 -1.14e-10 -9.41e-11 ...
##  $ Volume_Magnetic_Susceptibility: num  -2.23e-09 -1.05e-09 3.37e-06 -2.33e-05 -2.14e-05 ...
##  $ Percent_in_Universe           : num  7.5e+01 2.3e+01 6.0e-07 1.0e-07 1.0e-07 5.0e-01 1.0e-01 1.0 4.0e-05 1.3e-01 ...
##  $ Percent_in_Sun                : num  7.5e+01 2.3e+01 6.0e-09 1.0e-08 2.0e-07 3.0e-01 1.0e-01 9.0e-01 5.0e-05 1.0e-01 ...
##  $ Percent_in_Meteorites         : num  2.4 NA 1.7e-04 2.9e-06 1.6e-04 1.5 1.4e-01 4.0e+01 8.7e-03 NA ...
##  $ Percent_in_Earths_Crust       : num  1.5e-01 5.5e-07 1.7e-03 1.9e-04 8.6e-04 1.8e-01 2.0e-03 4.6e+01 5.4e-02 3.0e-07 ...
##  $ Percent_in_Oceans             : num  1.1e+01 7.2e-10 1.8e-05 6.0e-11 4.4e-04 ...
##  $ Percent_in_Humans             : num  1.0e+01 NA 3.0e-06 4.0e-08 7.0e-05 2.3e+01 2.6 6.1e+01 3.7e-03 NA ...
##  $ Atomic_Radius                 : num  5.30e-11 3.10e-11 1.67e-10 1.12e-10 8.70e-11 ...
##  $ Covalent_Radius               : num  3.70e-11 3.20e-11 1.34e-10 9.00e-11 8.20e-11 ...
##  $ Van_der_Waals_Radius          : num  1.20e-10 1.40e-10 1.82e-10 NA NA 1.70e-10 1.55e-10 1.52e-10 1.47e-10 1.54e-10 ...
##  $ Space_Group_Name              : chr  "P63/mmc" "Fm_\n3m" "Im_\n3m" "P63/mmc" ...
##  $ Space_Group_Number            : num  194 225 229 194 166 194 194 12 15 225 ...
##  $ HalfLife                      : num  Inf Inf Inf Inf Inf ...
##  $ Lifetime                      : num  Inf Inf Inf Inf Inf ...
##  $ Decay_Mode                    : chr  NA NA NA NA ...
##  $ Quantum_Numbers               : chr  "2S1/2" "1S0" "2S1/2" "1S0" ...
##  $ Neutron_Cross_Section         : num  3.32e-01 7.00e-03 4.50e-02 9.20e-03 7.55e+02 3.50e-03 1.91 2.80e-04 9.60e-03 4.00e-02 ...
##  $ Neutron_Mass_Absorption       : num  1.1e-02 1.0e-05 NA 3.0e-05 2.4 1.5e-05 4.8e-03 1.0e-06 2.0e-05 6.0e-04 ...
##  $ Graph.Period                  : num  1 1 2 2 2 2 2 2 2 2 ...
##  $ Graph.Group                   : num  1 18 1 2 13 14 15 16 17 18 ...

Properties, visualised

Note: the physical state at which the values are stated are not always determined in this dataset. This is a weakness of the web-scraping algorithm, which does not record the notes accompanying many datapoints. This means that the usability of these plots are limited in scope. If someone has an open, curated dataset of elemental properties available, let me know!

We can plot periodic tables of all available properties with just one chunk (using some looping, of course).

for (k in 1:dim(values)[2]) {
   # withhold some columns from plotting
   if (names(values)[k] %in% donotplot) {
      next
   }
   # numeric or character column?
   if (class(values[, k]) == "numeric") {
      p <-
         continuous_property_as_periodic_table(data = data.frame(
            Graph.Group = values$Graph.Group,
            Graph.Period = values$Graph.Period,
            Symbol = values$Symbol,
            Property = gsub("_", " ", names(values)[k]),
            Values = values[, k],
            Unit = units[, k]))
      print(p)
      cat(paste("Summary of", gsub("_", " ", names(values)[k])), "\n")
      print(summary(values[, k]))
   }
   if (class(values[, k]) == "character") {
      p <-
         discrete_property_as_periodic_table(data = data.frame(
            Graph.Group = values$Graph.Group,
            Graph.Period = values$Graph.Period,
            Symbol = values$Symbol,
            Values = values[, k]),
            scale.title = gsub("_", " ", names(values)[k]),
            scale.ncol = 3)
      print(p)
      cat(paste("Count of", gsub("_", " ", names(values)[k])), "\n")
      print(count(values, vars = values[, k]))
   }
}

## Summary of Atomic Number 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1.00   30.25   59.50   59.50   88.75  118.00

## Summary of Atomic Weight 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   1.008  65.409 140.908 144.877 226.000 294.000       1

## Summary of Density 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##     0.09  2545.00  7140.00  7645.57 10385.00 22650.00       23

## Summary of Melting Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   14.01  544.45 1204.15 1296.57 1811.15 3823.15      17

## Summary of Boiling Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    4.22 1193.90 2816.15 2558.87 3604.40 5869.15      24

## Count of Phase 
##     vars  n
## 1    Gas 11
## 2 Liquid  2
## 3  Solid 86
## 4   <NA> 19

## Summary of Critical Pressure 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
##    227000   5043000   7991000  20654048  16000000 172000000        97

## Summary of Critical Temperature 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    5.19  150.87  416.90  963.74 1766.00 3223.00      97

## Summary of Heat of Fusion 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.02    5.40   10.00   13.77   18.70  105.00      25

## Summary of Heat of Vaporization 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   0.083  73.225 272.500 265.906 380.000 800.000      24

## Summary of Heat of Combustion 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
## -9055.0  -829.0  -536.0 -1731.7  -345.5  -182.0     111

## Summary of Specific Heat 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    63.0   159.6   241.0   624.5   496.8 14300.0      30

## Count of Adiabatic Index 
##   vars   n
## 1  5/3   6
## 2  7/5   5
## 3 <NA> 107

## Summary of Neel Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    12.5    69.0   100.0   127.2   155.0   393.0     107

## Summary of Thermal Conductivity 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##   0.0036  10.0000  23.5000  58.0440  81.5000 430.0000       24

## Summary of Thermal Expansion 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
## 0.00000 0.00001 0.00001 0.00002 0.00002 0.00012      55

## Summary of Density Liquid 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##     512    4240    6980    7706    9330   20000      49

## Summary of Molar Volume 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
## 0.000004 0.000010 0.000017 0.002021 0.000023 0.022820       23

## Summary of Brinell Hardness 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
## 1.400e+05 1.995e+08 4.900e+08 7.087e+08 8.070e+08 3.920e+09        59

## Summary of Mohs Hardness 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##  200000 1875000 3000000 3740000 6000000 9300000      63

## Summary of Vickers Hardness 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
## 1.670e+08 4.060e+08 6.080e+08 2.108e+09 1.203e+09 4.900e+10        79

## Summary of Bulk Modulus 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
## 1.100e+09 2.800e+10 4.500e+10 9.079e+10 1.400e+11 3.800e+11        49

## Summary of Shear Modulus 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
## 1.300e+09 1.575e+10 2.650e+10 4.685e+10 4.725e+10 2.220e+11        58

## Summary of Young Modulus 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max.      NA's 
## 1.700e+09 3.700e+10 6.800e+10 1.105e+11 1.290e+11 5.280e+11        55

## Summary of Poisson Ratio 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##  0.0320  0.2525  0.2800  0.2924  0.3375  0.4500      64

## Summary of Refractive Index 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   1.000   1.000   1.001   1.075   1.001   2.417      99

## Summary of Speed of Sound 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##     206    1948    2750    3506    4626   18350      46

## Summary of Valence 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   0.000   3.000   4.000   3.776   5.000   7.000      11

## Summary of Electronegativity 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   0.700   1.285   1.625   1.737   2.175   3.980      24

## Summary of ElectronAffinity 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    0.00   21.07   50.00   76.16  105.38  349.00      32

## Summary of Autoignition Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   373.1   430.4   593.1   594.6   728.1   903.1     100

## Summary of Flashpoint 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   255.2   260.6   290.1   463.6   655.6   918.1     112

## Count of DOT Hazard Class 
##    vars  n
## 1   2.1  1
## 2   2.2  7
## 3   2.3  1
## 4   4.1 26
## 5   4.2  8
## 6   4.3  8
## 7   6.1  6
## 8     7  3
## 9     8  4
## 10 <NA> 54

## Summary of DOT Numbers 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    1073    1748    2730    2516    3089    9192      52

## Count of NFPA Fire Rating 
##   vars  n
## 1    0 17
## 2    1 13
## 3    2  6
## 4    3  7
## 5    4  4
## 6 <NA> 71

## Count of NFPA Health Rating 
##   vars  n
## 1    0  3
## 2    1 15
## 3    2 13
## 4    3 12
## 5    4  2
## 6 <NA> 73

## Count of NFPA Reactivity Rating 
##   vars  n
## 1    0 26
## 2    1  7
## 3    2  9
## 4    3  1
## 5    4  1
## 6 <NA> 74

## Count of Block 
##   vars  n
## 1    d 40
## 2    f 28
## 3    p 37
## 4    s 13

## Count of Color 
##         vars  n
## 1      Black  2
## 2  Colorless 11
## 3     Copper  1
## 4       Gold  1
## 5       Gray 14
## 6        Red  1
## 7     Silver 59
## 8  SlateGray  5
## 9     Yellow  2
## 10      <NA> 22

## Count of Gas phase 
##         vars   n
## 1   Diatomic   5
## 2 Monoatomic   6
## 3       <NA> 107

## Count of Electrical Type 
##            vars  n
## 1     Conductor 72
## 2     Insulator  5
## 3 Semiconductor  3
## 4          <NA> 38

## Summary of Electrical Conductivity 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##        0  1100000  4200000  8775038 11250000 62000000       38

## Summary of Resistivity 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
## 0.00e+00 0.00e+00 0.00e+00 1.25e+13 0.00e+00 1.00e+15       38

## Summary of Superconducting Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   0.015   0.490   1.083   2.059   3.410   9.250      85

## Count of Magnetic Type 
##                vars  n
## 1 Antiferromagnetic  1
## 2       Diamagnetic 31
## 3     Ferromagnetic  4
## 4      Paramagnetic 29
## 5              <NA> 53

## Summary of Curie Point 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##    20.0    32.0   222.0   416.2   631.0  1394.0     109

## Summary of Mass Magnetic Susceptibility 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##   0e+00   0e+00   0e+00   0e+00   0e+00   1e-05      36

## Summary of Molar Magnetic Susceptibility 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##       0       0       0       0       0       0      36

## Summary of Volume Magnetic Susceptibility 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
## -0.00017 -0.00001  0.00001  0.00340  0.00026  0.11178       37

## Summary of Percent in Universe 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##  0.0000  0.0000  0.0000  1.2056  0.0002 75.0000      35

## Summary of Percent in Sun 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##  0.00000  0.00000  0.00000  0.95902  0.00003 75.00000       14

## Summary of Percent in Meteorites 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##  0.00000  0.00000  0.00002  0.94255  0.00071 40.00000       11

## Summary of Percent in Earths Crust 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##  0.00000  0.00000  0.00011  0.88241  0.00298 46.00000        4

## Summary of Percent in Oceans 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##  0.0000  0.0000  0.0000  0.9125  0.0000 86.0000       8

## Summary of Percent in Humans 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##  0.00000  0.00000  0.00000  1.33104  0.00014 61.00000       43

## Summary of Atomic Radius 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##       0       0       0       0       0       0      34

## Summary of Covalent Radius 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##       0       0       0       0       0       0      47

## Summary of Van der Waals Radius 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##       0       0       0       0       0       0      80

## Summary of HalfLife 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## 0.000e+00 4.333e+11       Inf       Inf       Inf       Inf

## Summary of Lifetime 
##      Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
## 0.000e+00 6.247e+11       Inf       Inf       Inf       Inf

## Count of Decay Mode 
##              vars  n
## 1   AlphaEmission 32
## 2       BetaDecay  3
## 3   BetaPlusDecay  1
## 4 ElectronCapture  1
## 5            <NA> 81

## Summary of Neutron Cross Section 
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##     0.00     1.16     8.28   871.63    81.50 49000.00       20

## Summary of Neutron Mass Absorption 
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
##  0.0000  0.0006  0.0023  0.2093  0.0165  7.3000      35

And that’s all for today.

sessionInfo()

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] common_0.0.2  knitr_1.37    dplyr_1.0.8   ggplot2_3.3.5
## 
## loaded via a namespace (and not attached):
##  [1] bslib_0.3.1      compiler_4.0.5   pillar_1.7.0     jquerylib_0.1.4 
##  [5] tools_4.0.5      digest_0.6.29    jsonlite_1.7.3   evaluate_0.14   
##  [9] lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0     pkgconfig_2.0.3 
## [13] rlang_1.0.1      DBI_1.1.2        cli_3.2.0        yaml_2.2.2      
## [17] blogdown_1.7     xfun_0.29        fastmap_1.1.0    withr_2.4.3     
## [21] stringr_1.4.0    generics_0.1.2   sass_0.4.0       vctrs_0.3.8     
## [25] tidyselect_1.1.2 glue_1.6.1       R6_2.5.1         fansi_1.0.2     
## [29] rmarkdown_2.11   bookdown_0.24    purrr_0.3.4      magrittr_2.0.2  
## [33] scales_1.1.1     htmltools_0.5.2  ellipsis_0.3.2   assertthat_0.2.1
## [37] colorspace_2.0-2 utf8_1.2.2       stringi_1.7.6    munsell_0.5.0   
## [41] crayon_1.5.0

Properties of the elements: data collection and ggplot2 tables

Sun, 16 Nov 2014 08:00:00 +0100

Introduction

We will use rvest and magrittr to scrape elemental properties from periodictable.com, and then plot the data in the form of the periodic table using ggplot2.

It is very common to visualise periodic trends in the elemental properties by using the well-established IUPAC periodic table as a canvas. In this post we will demonstrate how common R tools can make the job quickly and near effortless, once the data is at hand.

library(ggplot2)
library(dplyr)
library(grid)
library(XML)
library(httr)
library(rvest)
library(magrittr)
library(stringr)
library(knitr)
library(common)

Collecting elemental data

The website periodictable.com lists a large number of properties for each element, and the data is displayed as not overly complicated HTML tables. The website states that it was created with Mathematica (by Wolfram Research), but even so, the quality of the data on the website is not too good. It appears to have suffered from whatever conversion was applied from the original Mathematica format.

In any case, in this post the quality of the data is secondary to the goal of achieving a working proof of concept. Plus, our own scraping will inevatibly degrade data quality even more.

element_data <- read_html("http://periodictable.com/Elements/001/data.html")

Starting from this URL, we crawl the page and collect the URLs to all property pages, along with the name of each property. We put them together in a dataframe, so it is clear which URL belongs to which elemental property.

# extract raw links and place it in a dataframe together with the link text
elemental_properties <-
   data.frame(property = element_data %>%
                 html_nodes(xpath = "//a") %>%
                 html_text())
elemental_properties$links_raw <-
   element_data %>%
   html_nodes(xpath = "//a") %>%
   html_attr("href")
# drop any rows that do not begin "../../"
elemental_properties <-
   elemental_properties[
      grep(pattern = "^\\.\\./\\.\\./",
           x = elemental_properties$links_raw), ]
# remove the leading ../.. (relative link signifier)
elemental_properties$links_trail <-
   gsub(pattern = "^\\.\\./\\.\\./",
        replacement = "",
        x = elemental_properties$links_raw)
# keep only the rows that begin with "Properties"
elemental_properties <-
   elemental_properties[
      grep(pattern = "^Properties",
           x = elemental_properties$links_trail), ]
# make full urls
elemental_properties$url <-
   paste0("http://periodictable.com/",
          elemental_properties$links_trail)
# modify the URL so it sorts the table by atomic number
# (this is a feature of periodictable.com, they offer lists sortered in different ways
# for each property)
elemental_properties$url <-
   sub(pattern = "\\.html$",
       replacement = ".an.html",
       x = elemental_properties$url)
# we are done, go ahead and drop the no longer required columns
# links_raw, links_trail
elemental_properties <-
   elemental_properties[, -c(2,3)]
# oh, and add a sanitised version of the property names
elemental_properties$sanitized <-
   # replace spaces with underscores
   gsub("\\s+", "_",
        # replace "%" with "percent"
        gsub("%", "Percent",
             # remove parentheses and dashes
             gsub("[()-]", "",
                  # remove apostrophes
                  gsub("'", "", elemental_properties$property))))
# NOTE: Melting point and Boiling point are duplicated because they are displayed twice
#       on the data page for Hydrogen, both under "Overview" and "Thermal properties".
#       So we should deduplicate the dataframe.
elemental_properties <- unique(elemental_properties)
# reset the row numbering
row.names(elemental_properties) <- seq(1, dim(elemental_properties)[1])

All elemental properties

The list of elemental properties available from periodictable.com. As collected (may contain some duplicates, these will be removed later).

print(elemental_properties$property)
##  [1] "Name"                           "Symbol"                        
##  [3] "Atomic Number"                  "Atomic Weight"                 
##  [5] "Density"                        "Melting Point"                 
##  [7] "Boiling Point"                  "Phase"                         
##  [9] "Absolute Melting Point"         "Absolute Boiling Point"        
## [11] "Critical Pressure"              "Critical Temperature"          
## [13] "Heat of Fusion"                 "Heat of Vaporization"          
## [15] "Specific Heat"                  "Adiabatic Index"               
## [17] "Neel Point"                     "Thermal Conductivity"          
## [19] "Thermal Expansion"              "Density (Liquid)"              
## [21] "Molar Volume"                   "Brinell Hardness"              
## [23] "Mohs Hardness"                  "Vickers Hardness"              
## [25] "Bulk Modulus"                   "Shear Modulus"                 
## [27] "Young Modulus"                  "Poisson Ratio"                 
## [29] "Refractive Index"               "Speed of Sound"                
## [31] "Valence"                        "Electronegativity"             
## [33] "ElectronAffinity"               "Ionization Energies"           
## [35] "DOT Hazard Class"               "DOT Numbers"                   
## [37] "RTECS Number"                   "NFPA Label"                    
## [39] "Alternate Names"                "Names of Allotropes"           
## [41] "Block"                          "Group"                         
## [43] "Period"                         "Series"                        
## [45] "Electron Configuration"         "Color"                         
## [47] "Discovery"                      "Gas phase"                     
## [49] "CAS Number"                     "CID Number"                    
## [51] "Electrical Type"                "Electrical Conductivity"       
## [53] "Resistivity"                    "Superconducting Point"         
## [55] "Magnetic Type"                  "Curie Point"                   
## [57] "Mass Magnetic Susceptibility"   "Molar Magnetic Susceptibility" 
## [59] "Volume Magnetic Susceptibility" "% in Universe"                 
## [61] "% in Sun"                       "% in Meteorites"               
## [63] "% in Earth's Crust"             "% in Oceans"                   
## [65] "% in Humans"                    "Atomic Radius"                 
## [67] "Covalent Radius"                "Van der Waals Radius"          
## [69] "Crystal Structure"              "Lattice Angles"                
## [71] "Lattice Constants"              "Space Group Name"              
## [73] "Space Group Number"             "Half-Life"                     
## [75] "Lifetime"                       "Decay Mode"                    
## [77] "Quantum Numbers"                "Neutron Cross Section"         
## [79] "Neutron Mass Absorption"        "Known Isotopes"                
## [81] "Stable Isotopes"                "Isotopic Abundances"

Collect the values for each property

Each property, e.g., density, will have a value for each element of the periodic table. This value is just a string, and depending on the type of the property, it may be just a number, or a quantity with a unit, or some text with various attributes.

At this stage, we don’t mind the internal structure of the value, we just want to collect it. We will collect all property values into one dataframe.

Looking slightly ahead, you will realise that the only way to allow different types (character, numeric, etc.) in different columns is if each property is mapped onto one column. Thus, we will build the dataframe with properties as columns, and elements as rows.

# get the name of all the elements (two-step process)
element_names <-
   read_html(elemental_properties$url[1]) %>%
   html_nodes("table") %>%
   extract2(1) %>%
   html_nodes("table") %>%
   extract2(8) %>%
   html_nodes("td") %>%
   html_text()
element_names <-
   element_names[
      read_html(elemental_properties$url[1]) %>%
         html_nodes("table") %>%
         extract2(1) %>%
         html_nodes("table") %>%
         extract2(8) %>%
         html_nodes("td") %>%
         html_attr("align") == "left"
      ]
# create a properly oriented elements dataframe
elements_raw <-
   data.frame(matrix(data = "",
                     ncol = dim(elemental_properties)[1],
                     nrow = length(element_names),
                     byrow = TRUE))
names(elements_raw) <- elemental_properties$sanitized
elements_raw$Name <- element_names

Some property pages (listed below) are difficult to parse (due to the way the data is presented on the webpage). For now, we just skip those pages (no big loss).

skip_properties <-
   c("Ionization_Energies",
     "NFPA_Hazards",
     "NFPA_Label",
     "Names_of_Allotropes",
     "Discovery",
     "Crystal_Structure",
     "Lattice_Angles",
     "Lattice_Constants",
     "Known_Isotopes",
     "Stable_Isotopes",
     "Isotopic_Abundances")
# drop those rows from elemental_properties
elemental_properties <-
   elemental_properties[-which(elemental_properties$sanitized %in% skip_properties), ]
# drop those columns from elements_raw
elements_raw <-
   elements_raw[, -which(names(elements_raw) %in% skip_properties)]

Ok, now we have the skeleton for a dataframe. Let’s populate it.

# re-read only if rda file does not exist (saves time when compiling)
if (!file.exists(here::here("assets/element-data-raw.rda"))) {
   for (k in 2:length(elemental_properties$property)) {
      message(paste0("Reading property page (", k-1, " of ", length(elemental_properties$property)-1, "): ", elemental_properties$property[k]))
      property <-
         read_html(elemental_properties$url[k]) %>%
         html_nodes("table") %>%
         extract2(1) %>%
         html_nodes("table") %>%
         extract2(8) %>%
         html_nodes("td") %>%
         html_text()
      property <-
         property[
            read_html(elemental_properties$url[k]) %>%
               html_nodes("table") %>%
               extract2(1) %>%
               html_nodes("table") %>%
               extract2(8) %>%
               html_nodes("td") %>%
               html_attr("align") == "left"
            ]
      # for debugging purposes
      # print(paste(elemental_properties$property[k], ":", length(property)))
      # assign to elements_raw
      elements_raw[, k] <- property
   }
   # save this dataframe to file
   # (it is not large, but re-scraping the contents takes time)
   save(elements_raw, file = here::here("assets/element-data-raw.rda"))
} else {
   load(file = here::here("assets/element-data-raw.rda"))
}

Elemental property data collected! Next, we need to sanitize the data. As we mentioned earlier, the values may be of several different types. The biggest job is to deal with quantities and units.

There might well be an easier way to do this, but to make the coding easy, we will create two empty dataframes (based on the existing elements_raw, with identical dimensions and column names), and place quantities (and unit-less values) into one, and units into the other.

# creating new dataframes, templated on elements_raw
# notes is not used
values <- units <- notes <-
   data.frame(matrix(nrow = dim(elements_raw)[1],
                     ncol = dim(elements_raw)[2],
                     dimnames = list(seq(1, dim(elements_raw)[1]),
                                     names(elements_raw)),
                     byrow = TRUE))
# create a work-copy of elements_raw
elements_tmp <- elements_raw

The tricky part will be to correctly determine what part of the values string is a quantity and what part is a unit. We will have to resort to regular expressions. Because of that, the following code is probably the most likely to break if anything should change at the data source.

Clean up all the value strings

We will use regular expressions to identify the different parts of each value string. See comments in the code below. Unfortunately, we found no way to clean up the value strings of each type of column (non-numeric, numbers-only, and quantity+unit) without using explicit, static assignments, as seen in the following three chunks. It is ugly, but it works, as long as no properties (i.e., columns) are added, removed, or renamed at the source.

# non-numeric columns to copy requiring only minor cleanup
# col numbers: 1  2  8 17 39 43 44 45 48 49 50 51 52 53 54 55 59 73 77 78
# 73 (Space_Group_Name, may require some fixing of notation)
cols <-
   c("Name",
     "Symbol",
     "Phase",
     "Adiabatic_Index",
     "EU_Number",
     "RTECS_Number",
     "Alternate_Names",
     "Block",
     "Group",
     "Period",
     "Electron_Configuration",
     "Color",
     "Gas_phase",
     "CAS_Number",
     "CID_Number",
     "Gmelin_Number",
     "NSC_Number",
     "Electrical_Type",
     "Magnetic_Type",
     "Space_Group_Name",
     "Decay_Mode",
     "Quantum_Numbers")
for (k in 1:length(cols)) {
   values[, which(names(values) == cols[k])] <-
      gsub("^None$", "",
           gsub("^N/A$", "",
                gsub("\\[note\\]",
                     "",
                     elements_tmp[, which(names(elements_tmp) == cols[k])])
                )
           )
   # set empty strings to NA (proper NA, not the string "NA")
   values[which(values[, which(names(values) == cols[k])] == ""),
          which(names(values) == cols[k])] <- NA
}

# numbers only cols
cols <-
   c("Atomic_Number",
     "Atomic_Weight",
     "Molar_Volume",
     "Poisson_Ratio",
     "Refractive_Index",
     "Valence",
     "Electronegativity",
     "DOT_Hazard_Class",
     "DOT_Numbers",
     "NFPA_Fire_Rating",
     "NFPA_Health_Rating",
     "NFPA_Reactivity_Rating",
     "Superconducting_Point",
     "Mass_Magnetic_Susceptibility",
     "Molar_Magnetic_Susceptibility",
     "Volume_Magnetic_Susceptibility",
     "Space_Group_Number",
     "Neutron_Cross_Section",
     "Neutron_Mass_Absorption")
for (k in 1:length(cols)) {
   values[, which(names(values) == cols[k])] <-
      as.numeric(gsub("×10", "E",
                      gsub("\\[note\\]",
                           "",
                           elements_tmp[, which(names(elements_tmp) == cols[k])])
                      )
                 )
}

# numbers and units cols
cols <-
   c("Density",
     "Melting_Point",
     "Boiling_Point",
     "Absolute_Melting_Point",
     "Absolute_Boiling_Point",
     "Critical_Pressure", # contains converted values in parentheses
     "Critical_Temperature",
     "Heat_of_Fusion",
     "Heat_of_Vaporization",
     "Heat_of_Combustion",
     "Specific_Heat", # complex unit J/(Kg K)
     "Neel_Point",
     "Thermal_Conductivity", # complex unit W/(m K)
     "Thermal_Expansion",
     "Density_Liquid",
     "Brinell_Hardness",
     "Mohs_Hardness",
     "Vickers_Hardness",
     "Bulk_Modulus",
     "Shear_Modulus",
     "Young_Modulus",
     "Speed_of_Sound",
     "ElectronAffinity",
     "Autoignition_Point",
     "Flashpoint",
     "Electrical_Conductivity",
     "Resistivity",
     "Curie_Point",
     "Percent_in_Universe",
     "Percent_in_Sun", # makes use of "None" in lieu of zero
     "Percent_in_Meteorites", # makes use of "None" in lieu of zero
     "Percent_in_Earths_Crust", # makes use of "None" in lieu of zero
     "Percent_in_Oceans", # makes use of "None" in lieu of zero
     "Percent_in_Humans", # makes use of "None" in lieu of zero
     "Atomic_Radius",
     "Covalent_Radius",
     "Van_der_Waals_Radius",
     "HalfLife", # mix between "Stable" and num + unit
     "Lifetime") # mix between "Stable" and num + unit
for (k in 1:length(cols)) {
   for (i in 1:length(elements_tmp[, which(names(elements_tmp) %in% cols[k])])) {
      # replace the string "None" with actual zero
      elements_tmp[, which(names(elements_tmp) %in% cols)][i, k] <-
         ifelse(elements_tmp[, which(names(elements_tmp) %in% cols)][i, k] == "None",
                "0",
                elements_tmp[, which(names(elements_tmp) %in% cols)][i, k])
      # replace the string "Stable" with "Inf"
      elements_tmp[, which(names(elements_tmp) %in% cols)][i, k] <-
         ifelse(elements_tmp[, which(names(elements_tmp) %in% cols)][i, k] == "Stable",
                "Inf",
                elements_tmp[, which(names(elements_tmp) %in% cols)][i, k])
      # look for numbers (also for "Inf")
      mtch <-
         regexpr(pattern = "Inf|[-×\\.0-9]+",
                 text = elements_tmp[, which(names(elements_tmp) %in% cols)][i, k])
      # assign numeric value to "values"
      values[i, which(names(values) == cols[k])] <-
         as.numeric(
            sub(
               pattern = "×10",
               replacement = "E",
               x = substr(x = elements_tmp[, which(names(elements_tmp) %in% cols)][i, k],
                          start = 1,
                          stop = attr(mtch, "match.length"))))
      # assign unit part to "units" by eliminating the numeric part
      units[i, which(names(values) == cols[k])] <-
         # remove leading or trailing spaces
         sub("^\\s+", "", sub("\\s+$", "",
                 # remove the numeric part
                 sub("Inf|[-×\\.0-9]+", "",
                     # remove any numbers+units within parentheses
                     sub("\\([.0-9]+\\s[A-Za-z]+\\)", "",
                         # remove notes
                         sub("\\[note\\]", "",
                             # remove "N/A" strings
                             sub("N/A",
                                 "",
                                 elements_tmp[, which(
                                    names(elements_tmp) %in% cols)][i, k]
                                 )
                             )
                         )
                     )
                 )
             )
   }
   # set empty strings to NA (proper NA, not the string "NA")
   values[which(values[, which(names(values) == cols[k])] == ""),
          which(names(values) == cols[k])] <- NA
}

Cleaning up the units

Now that we have separated the units and the quantities into separate strings (dataframes, actually), let’s have a look at the units, to see if there’s any fixing needed.

# Units by property (looking at the first 10)
tail(sapply(units, unique), 10)
## $Covalent_Radius
## [1] "pm" ""  
## 
## $Van_der_Waals_Radius
## [1] "pm" ""  
## 
## $Space_Group_Name
## [1] NA
## 
## $Space_Group_Number
## [1] NA
## 
## $HalfLife
## [1] ""   "y"  "h"  "d"  "m"  "ms"
## 
## $Lifetime
## [1] ""   "y"  "h"  "d"  "m"  "ms"
## 
## $Decay_Mode
## [1] NA
## 
## $Quantum_Numbers
## [1] NA
## 
## $Neutron_Cross_Section
## [1] NA
## 
## $Neutron_Mass_Absorption
## [1] NA

As you can see, some properties (for example HalfLife) use more than one unit. This is problematic, since we will only be plotting against one y-axis, not several. So we will have to convert all such occurrences to their standard units, which means we have to take the numerical conversion of the quantity into consideration as well. Let’s do it.

First, let’s see all units in the set.

# All units in the dataset
cat(paste(sort(unique(unlist(sapply(units, unique)))), collapse = "\n"))
## 
## %
## °C
## d
## g/cc
## g/cm3
## g/l
## GPa
## h
## J/(Kg K)
## K
## K-1
## kJ/mol
## KJ/mol
## m
## m Ω
## m/s
## MPa
## ms
## pm
## S/m
## W/(m K)
## y

Some are equivalent to each other, and others can be reduced to base SI units, according to the following list, which we put together manually after inspecting the output above.

## %
## GPa         => Pa
## J/(Kg K)
## K
## K-1
## KJ/mol      => kJ/mol (wrong case)
## MPa         => Pa
## S/m
## W/(m K)
## d           => s
## g/cc        => kg/m^3
## g/cm3       => kg/m^3
## g/l         => kg/m^3
## h           => s
## kJ/mol
## m
## m Ω
## m/s
## ms          => s
## pm          => m
## y           => s
## °C          => K

Someone should probably tell periodictable.com that they have a typo in some of their values with units of KJ/mol instead of kJ/mol.

Building up a match-and-replace dataframe to convert non-standard or simplifiable units to standard units. This will help simplify any visualisations as well as possible comparisons between properties.

# Build up a match-and-convert dataframe based on the list above
pcf <-
   # find any values with these units...
   data.frame(pattern = c("GPa",
                          "KJ/mol",
                          "MPa",
                          "d",
                          "g/cc",
                          "g/cm3",
                          "g/l",
                          "h",
                          "ms",
                          "pm",
                          "y",
                          "°C"),
              # ... and convert them into these units
              convert = c("Pa",
                          "kJ/mol",
                          "Pa",
                          "s",
                          "kg/m^3",
                          "kg/m^3",
                          "kg/m^3",
                          "s",
                          "s",
                          "m",
                          "s",
                          "K"),
              # using these conversion factors
              factor = c("1E9",
                         "1",
                         "1E6",
                         "86400",
                         "1E3",
                         "1E3",
                         "1",
                         "3600",
                         "1E-3",
                         "1E-12",
                         "3.154E7",
                         "+273.15"))
# Here we replace the values and the units according to pcf
for (k in 1:dim(units)[2]) {
   # if the entire column is NA, move to the next one
   if (all(is.na(units[, k]))) {next}
   for (i in 1:dim(units)[1]) {
      # jump to the next cell immediately if unit is empty
      if (units[i, k] == "") {next}
      # for each cell, compare the unit to pcf$pattern,
      # and if they match, replace it with pcf$convert
      # and apply the conversion factor on the value
      # match() returns the position of the match
      # in pcf$pattern, or NA if no match was found
      hit <- match(units[i, k], pcf$pattern)
      if (!is.na(hit)) {
         # go ahead and replace unit and convert value
         units[i, k] <- pcf$convert[hit]
         # had to find a way to handle the odd addition operation,
         # opted to do it with a string operation and this if-else
         if (substr(pcf$factor[hit], 1, 1) %in% c("+", "-")) {
            if (substr(pcf$factor[hit], 1, 1) == "-") {
               values[i, k] <-
                  values[i, k] + as.numeric(pcf$factor[hit])
            } else {
               values[i, k] <-
                  values[i, k] + as.numeric(sub("^\\+", "", pcf$factor[hit]))
            }
         } else {
            # not addition/subtraction operation
            values[i, k] <-
               values[i, k] * as.numeric(pcf$factor[hit])
         }
      }
   }
}

Visualising elemental data

First, an abridged look at the data itself.

# show the first 10 elemental property quantities/values
str(values, list.len = 10)
## 'data.frame':    118 obs. of  80 variables:
##  $ Name                          : chr  "Hydrogen" "Niobium" "Thallium" "Helium" ...
##  $ Symbol                        : chr  "H" "Nb" "Tl" "He" ...
##  $ Atomic_Number                 : num  1 41 81 2 42 82 3 43 83 4 ...
##  $ Atomic_Weight                 : num  1.01 92.91 204.38 4 95.94 ...
##  $ Density                       : num  8.99e-02 8.57e+03 1.18e+04 1.78e-01 1.03e+04 ...
##  $ Melting_Point                 : num  14 2750 577 NA 2896 ...
##  $ Boiling_Point                 : num  20.28 5017.15 1746.15 4.22 4912.15 ...
##  $ Phase                         : chr  "Gas" "Solid" "Solid" "Gas" ...
##  $ Absolute_Melting_Point        : num  14 2750 577 NA 2896 ...
##  $ Absolute_Boiling_Point        : num  20.28 5017 1746 4.22 4912 ...
##   [list output truncated]
# show the first 10 elemental units
str(units, list.len = 10)
## 'data.frame':    118 obs. of  80 variables:
##  $ Name                          : logi  NA NA NA NA NA NA ...
##  $ Symbol                        : logi  NA NA NA NA NA NA ...
##  $ Atomic_Number                 : logi  NA NA NA NA NA NA ...
##  $ Atomic_Weight                 : logi  NA NA NA NA NA NA ...
##  $ Density                       : chr  "kg/m^3" "kg/m^3" "kg/m^3" "kg/m^3" ...
##  $ Melting_Point                 : chr  "K" "K" "K" "" ...
##  $ Boiling_Point                 : chr  "K" "K" "K" "K" ...
##  $ Phase                         : logi  NA NA NA NA NA NA ...
##  $ Absolute_Melting_Point        : chr  "K" "K" "K" "" ...
##  $ Absolute_Boiling_Point        : chr  "K" "K" "K" "K" ...
##   [list output truncated]

Next, let’s try a typical plot of a property against atomic number.

Figure 1: Atomic weight vs Z.

Another plot of a property (density, this time) against atomic number.

Figure 2: Density vs Z.

Ok, it looks as one would expect. Not very exciting, though.

We are now ready to attempt to overlay elemental properties on top of the periodic table. I think this is a worthwhile enterprise, because I have so far yet to see a way to programmatically create a periodic table and overlay data visually on it.

As a long-time LaTeX user myself, the drawing of the periodic table using TikZ by Ivan Griffin was at the same time brilliant and frustrating.

Brilliant, because for the first time it gave writers (well, at least writers familiar with LaTeX and TikZ) the ability to easily create our own periodic tables, as well as to customise them. Frustrating, because there was no easy way to tie the generation of the TikZ-based periodic table into available periodic trend data. Of course, this was an inherent shortcoming of TikZ/LaTeX more than anything else.

In this work, I have used R with the ggplot2 plotting system to achieve this the visualisation of arbitrary elemental data over the canvas of the periodic table.

Final preparations before plotting

The periodic table places the elements next to each other, organised in rows (periods) and columns (groups) on a two-dimensional plot. Obviously, each element’s position on this plot is completely specified by its group and period.

We will therefore introduce new group and new period variables, specifically for the purpose of plotting. We will not use the original group and period data for two reasons: some elements were not assigned a group and period in the original dataset, and we want some flexibility to adjust the coordinates to control the aesthetics of the plot.

# rearrange values dataframe by atomic number
# simply to make assignments based on atomic number (below) possible
values <- arrange(values, Atomic_Number)
values$Graph.Period <- values$Period
units$Graph.Period <- "" # to maintain same dims as values
values$Graph.Group <- values$Group
units$Graph.Group <- "" # to maintain same dims as values
# lanthanoids 57-71: Period = 8, Group = seq(3, 17)
values$Graph.Period[seq(57,71)] <- 8.5
values$Graph.Group[seq(57,71)] <- seq(3, 17)
# actinoids 89-103: Period = 9, Group = seq(3, 17)
# increase period to increase the gap up to the transition block
values$Graph.Period[seq(89,103)] <- 9.5
values$Graph.Group[seq(89,103)] <- seq(3, 17)
# make graphical Group and Period numeric
values$Graph.Period <- as.numeric(values$Graph.Period)
values$Graph.Group <- as.numeric(values$Graph.Group)

With just a few lines, we can make a very quick-and-dirty periodic table (sans data) using ggplot2:

ggplot() +
   scale_y_reverse() +
   geom_point(data = values,
              size = 14,
              shape = 0,
              aes(y = Graph.Period,
                  x = Graph.Group)) +
   geom_text(data = values,
             aes(label = Symbol,
                 y = Graph.Period,
                 x = Graph.Group))

Plotting continuous elemental properties

Let’s use the periodic table we created with ggplot2 to plot some of the continuous variables.

ggplot() +
   geom_point(data = values,
              # size 14 fits well with fig.width = 9, fig.height = 5.25
              size = 14,
              # shape #22 allows both fill and colour to be
              # shape #15 only registers colour (is always filled)
              # shape #0 only registers colour (is always empty)
              shape = 15,
              aes(y = Graph.Period, x = Graph.Group, colour = Density)) +
   geom_text(data = values,
             colour = "white",
             size = 3,
             fontface = "bold",
             aes(label = Symbol, y = Graph.Period, x = Graph.Group)) +
   scale_x_continuous(breaks = seq(min(values$Graph.Group),
                                   max(values$Graph.Group)),
                      limits = c(min(values$Graph.Group) - 1,
                                 max(values$Graph.Group) + 1),
                      expand = c(0,0)) +
   scale_y_continuous(trans = "reverse",
                      breaks = seq(min(values$Graph.Period),
                                   max(values$Graph.Period)),
                      limits = c(max(values$Graph.Period) + 1,
                                 min(values$Graph.Period) - 1.5),
                      expand = c(0,0)) +
   # set breaks and labels in the colourbar legend
   scale_colour_continuous(breaks = c(5E3, 10E3, 15E3, 20E3),
                           labels = c("5", "10", "15", "20"),
                           # range of colour
                           low = "#56B1F7", high = "#132B43",
                           # colour if value is NA
                           na.value = "grey70") +
   # plot title (usually property and unit)
   annotate("text", x = 8, y = 0.6,
            vjust = 0,
            label = paste("Density/", "10^3*~kg~m", "^-3", sep=""),
            # parse required if using superscripts or subscripts
            parse = TRUE) +
   theme(panel.grid.major = element_blank(),
         panel.grid.minor = element_blank(),
         plot.margin = unit(c(0, 0, -0.85, -0.85), "line"),
         axis.title = element_blank(),
         axis.text = element_blank(),
         axis.ticks = element_blank(),
         # center (roughly) over transition metal block
         legend.position = c(0.42, 0.91),
         legend.justification = c(0.5, 1),
         legend.direction = "horizontal",
         # make the legend colourbar a little longer
         legend.key.width = unit(2.5, "line"),
         legend.title = element_blank(),
         legend.background = element_rect(fill = "transparent"))

A short explanation is in order. The boxes are inherently square, but we adjusted the plot dimensions (fig.width and fig.height in knitr parlance) to make sure the final appearance of the boxes is indeed square. Colour is mapped to the numeric values of the plotted property, using the built-in ggplot2::scale_colour_continuous() function. And we positioned the required legend over the transition metal block, to conform with most other periodic tables out there. Grayed-out elements lack data for the shown property.

Here are some more periodic tables overlaid with other continuous elemental data.

We could easily change the print size of these plots, as well as export them to most common image formats. We could also easily switch from the current knitr and markdown document system to knitr and LaTeX to take advantage of the excellent math and symbol support of LaTeX.

Plotting discrete elemental properties

ggplot() +
   geom_point(data = values,
              # size 14 fits well with fig.width = 9, fig.height = 5.25
              size = 14,
              # shape #22 allows both fill and colour to be
              # shape #15 only registers colour (is always filled)
              # shape #0 only registers colour (is always empty)
              shape = 15,
              aes(y = Graph.Period, x = Graph.Group, colour = Magnetic_Type)) +
   geom_text(data = values,
             colour = "white",
             size = 3,
             fontface = "bold",
             aes(label = Symbol, y = Graph.Period, x = Graph.Group)) +
   scale_x_continuous(breaks = seq(min(values$Graph.Group),
                                   max(values$Graph.Group)),
                      limits = c(min(values$Graph.Group) - 1,
                                 max(values$Graph.Group) + 1),
                      expand = c(0,0)) +
   scale_y_continuous(trans = "reverse",
                      breaks = seq(min(values$Graph.Period),
                                   max(values$Graph.Period)),
                      limits = c(max(values$Graph.Period) + 1,
                                 min(values$Graph.Period) - 1.5),
                      expand = c(0,0)) +
   scale_colour_discrete(na.value = "grey70") +
   # set the size of the legend boxes independent of geom_point's aes
   guides(colour = guide_legend(override.aes = list(size = 5),
                                title = "Magnetic type",
                                nrow = 2,
                                title.hjust = 0.5)) +
   theme(panel.grid.major = element_blank(),
         panel.grid.minor = element_blank(),
         plot.margin = unit(c(0, 0, -0.85, -0.85), "line"),
         axis.ticks = element_blank(),
         axis.title = element_blank(),
         axis.text = element_blank(),
         # center (roughly) over transition metal block
         legend.position = c(0.42, 0.93),
         legend.justification = c(0.5, 1),
         legend.direction = "vertical",
         legend.key.size = unit(0.8, "line"),
         legend.title = element_text(size = 15,
                                     face = "plain"),
         legend.key = element_rect(fill = "transparent",
                                   colour = "transparent"),
         legend.background = element_rect(fill = "transparent"))

Some more with discrete variables:

Conclusion

We have successfully demonstrated how ggplot2 may be used to programmatically generate plots of elemental properties in the form of the typical periodic table.

To do this, we also had to scrape and collect a database of elemental property data from public webpages.

Our hope is that this will make it easier for chemists and others to generate periodic tables of whatever trend they wish to visualise.

To make it easier to repeat this code, we have included an MWE below.

Minimal working example

library(ggplot2)
library(dplyr)
library(grid)
library(XML)
library(httr)
library(rvest)
library(magrittr)
library(stringr)
library(knitr)
library(common)

Simply load this dataset: (Ahmed 2014).

load(url("http://files.figshare.com/1873544/element_data.rda"))

Use this ggplot2 code to generate a periodic table visualising a continuous variable.

ggplot() +
   geom_point(data = values,
              # size 14 fits well with fig.width = 9, fig.height = 5.25
              size = 14,
              # shape #22 allows both fill and colour to be
              # shape #15 only registers colour (is always filled)
              # shape #0 only registers colour (is always empty)
              shape = 15,
              aes(y = Graph.Period, x = Graph.Group, colour = Density)) +
   geom_text(data = values,
             colour = "white",
             size = 3,
             fontface = "bold",
             aes(label = Symbol, y = Graph.Period, x = Graph.Group)) +
   scale_x_continuous(breaks = seq(min(values$Graph.Group),
                                   max(values$Graph.Group)),
                      limits = c(min(values$Graph.Group) - 1,
                                 max(values$Graph.Group) + 1),
                      expand = c(0,0)) +
   scale_y_continuous(trans = "reverse",
                      breaks = seq(min(values$Graph.Period),
                                   max(values$Graph.Period)),
                      limits = c(max(values$Graph.Period) + 1,
                                 min(values$Graph.Period) - 1.5),
                      expand = c(0,0)) +
   # set breaks and labels in the colourbar legend
   scale_colour_continuous(breaks = c(5E3, 10E3, 15E3, 20E3),
                           labels = c("5", "10", "15", "20"),
                           # range of colour
                           low = "#56B1F7", high = "#132B43",
                           # colour if value is NA
                           na.value = "grey70") +
   # plot title (usually property and unit)
   annotate("text", x = 8, y = 0.6,
            vjust = 0,
            label = paste("Density/", "10^3*~kg~m", "^-3", sep=""),
            # parse required if using superscripts or subscripts
            parse = TRUE) +
   theme(panel.grid.major = element_blank(),
         panel.grid.minor = element_blank(),
         plot.margin = unit(c(0, 0, -0.85, -0.85), "line"),
         axis.title = element_blank(),
         axis.text = element_blank(),
         axis.ticks = element_blank(),
         # center (roughly) over transition metal block
         legend.position = c(0.42, 0.91),
         legend.justification = c(0.5, 1),
         legend.direction = "horizontal",
         # make the legend colourbar a little longer
         legend.key.width = unit(2.5, "line"),
         legend.title = element_blank(),
         legend.background = element_rect(fill = "transparent"))

Or this code to visualise a discrete variable.

ggplot() +
   geom_point(data = values,
              # size 14 fits well with fig.width = 9, fig.height = 5.25
              size = 14,
              # shape #22 allows both fill and colour to be
              # shape #15 only registers colour (is always filled)
              # shape #0 only registers colour (is always empty)
              shape = 15,
              aes(y = Graph.Period, x = Graph.Group, colour = Magnetic_Type)) +
   geom_text(data = values,
             colour = "white",
             size = 3,
             fontface = "bold",
             aes(label = Symbol, y = Graph.Period, x = Graph.Group)) +
   scale_x_continuous(breaks = seq(min(values$Graph.Group),
                                   max(values$Graph.Group)),
                      limits = c(min(values$Graph.Group) - 1,
                                 max(values$Graph.Group) + 1),
                      expand = c(0,0)) +
   scale_y_continuous(trans = "reverse",
                      breaks = seq(min(values$Graph.Period),
                                   max(values$Graph.Period)),
                      limits = c(max(values$Graph.Period) + 1,
                                 min(values$Graph.Period) - 1.5),
                      expand = c(0,0)) +
   scale_colour_discrete(na.value = "grey70") +
   # set the size of the legend boxes independent of geom_point's aes
   guides(colour = guide_legend(override.aes = list(size = 5),
                                title = "Magnetic type",
                                nrow = 2,
                                title.hjust = 0.5)) +
   theme(panel.grid.major = element_blank(),
         panel.grid.minor = element_blank(),
         plot.margin = unit(c(0, 0, -0.85, -0.85), "line"),
         axis.ticks = element_blank(),
         axis.title = element_blank(),
         axis.text = element_blank(),
         # center (roughly) over transition metal block
         legend.position = c(0.42, 0.93),
         legend.justification = c(0.5, 1),
         legend.direction = "vertical",
         legend.key.size = unit(0.8, "line"),
         legend.title = element_text(size = 15,
                                     face = "plain"),
         legend.key = element_rect(fill = "transparent",
                                   colour = "transparent"),
         legend.background = element_rect(fill = "transparent"))

And there you are.

Appendix

sessionInfo()

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] grid      stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] common_0.0.2   knitr_1.37     stringr_1.4.0  magrittr_2.0.2 rvest_1.0.2   
## [6] httr_1.4.2     XML_3.99-0.8   dplyr_1.0.8    ggplot2_3.3.5 
## 
## loaded via a namespace (and not attached):
##  [1] bslib_0.3.1      compiler_4.0.5   pillar_1.7.0     jquerylib_0.1.4 
##  [5] tools_4.0.5      digest_0.6.29    jsonlite_1.7.3   evaluate_0.14   
##  [9] lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0     pkgconfig_2.0.3 
## [13] rlang_1.0.1      DBI_1.1.2        cli_3.2.0        yaml_2.2.2      
## [17] blogdown_1.7     xfun_0.29        fastmap_1.1.0    xml2_1.3.3      
## [21] withr_2.4.3      generics_0.1.2   sass_0.4.0       vctrs_0.3.8     
## [25] tidyselect_1.1.2 glue_1.6.1       R6_2.5.1         fansi_1.0.2     
## [29] rmarkdown_2.11   bookdown_0.24    purrr_0.3.4      scales_1.1.1    
## [33] htmltools_0.5.2  ellipsis_0.3.2   assertthat_0.2.1 colorspace_2.0-2
## [37] utf8_1.2.2       stringi_1.7.6    munsell_0.5.0    crayon_1.5.0

Todo

Record notes and save them along with values and units.

References

Ahmed, Taha. 2014. “Properties of the elements.” https://doi.org/10.6084/m9.figshare.1295585.

Characteristic x-ray line energies of the elements

Sat, 13 Sep 2014 08:00:00 +0100

Introduction

Having a list of all x-ray transitions of the elements is quite useful, in particular when analysing x-ray fluorescence spectra.

This work was based on data from the PANalytical Epsilon 5 software, version 2.0J/ICSW 2.8 of Dec 7, 2010 with kernel PW5050. In this software, data on every transition’s energy and relative intensity was stored in a user-accessible format, as well as the edge energies for each element.

The Epsilon 5 data was used to amend my original dataset, which I got from Kaye & Laby, Tables of Physical & Chemical Constants, National Physical Laboratory. Where the two datasets diverged (usually at the third decimal) I gave priority to the Epsilon 5 data.

I have improved on this dataset, and included natural line width and fluorescence yield data from the works of Krause and Oliver (1979) and Krause (1979).

I collated this data into two datasets (Ahmed 2014b, 2014a) which we will explore below.

Notation: IUPAC more stringent than Siegbahn

At this stage in the work, it gradually became untenable to keep using the Siegbahn notation for x-ray transitions, since it is both incomplete (not all transitions have labels) and slightly arbitrary (different authors use the same label for different transitions). So I decided to switch to the IUPAC notation, which offers a complete, coherent system. This required me to translate all the existing Siegbahn labels in my dataset into IUPAC ones. To do that, I needed a “translator”. Luckily I found a good start in table 10.11 of Lengyel, Ure, and Inczédy (1998), and from there it was not too hard to work out the corresponding IUPAC names of the remaining transitions in my dataset.

Table 1: IUPAC notation and corresponding Siegbahn notation
IUPAC	Siegbahn	IUPAC	Siegbahn	IUPAC	Siegbahn
$\mathrm{K}\mathrm{L}_{2,3}$	$\mathrm{K}\alpha_{1,2}$	$\mathrm{L}_1\mathrm{M}_{2,3}$	$\mathrm{K}\beta_{3,4}$	$\mathrm{M}_{4,5}\mathrm{N}_{2,3}$	$\mathrm{M}\zeta_{1,2}$
$\mathrm{K}\mathrm{L}_1$	$\mathrm{K}\alpha_3$	$\mathrm{L}_1\mathrm{M}_2$	$\mathrm{L}\beta_4$	$\mathrm{M}_1\mathrm{N}_3$	NA
$\mathrm{K}\mathrm{L}_2$	$\mathrm{K}\alpha_2$	$\mathrm{L}_1\mathrm{M}_3$	$\mathrm{L}\beta_3$	$\mathrm{M}_2\mathrm{N}_4$	NA
$\mathrm{K}\mathrm{L}_3$	$\mathrm{K}\alpha_1$	$\mathrm{L}_1\mathrm{M}_4$	$\mathrm{L}\beta_{10}$	$\mathrm{M}_3\mathrm{N}_1$	NA
$\mathrm{K}\mathrm{M}_{2,3}$	$\mathrm{K}\beta_{1,3}$	$\mathrm{L}_1\mathrm{M}_5$	$\mathrm{L}\beta_9$	$\mathrm{M}_3\mathrm{N}_4$	$\mathrm{M}\gamma_2$
$\mathrm{K}\mathrm{M}_{4,5}$	$\mathrm{K}\beta_5$	$\mathrm{L}_1\mathrm{N}_2$	$\mathrm{L}\gamma_2$	$\mathrm{M}_3\mathrm{N}_5$	$\mathrm{M}\gamma_1$
$\mathrm{K}\mathrm{M}_2$	$\mathrm{K}\beta_3$	$\mathrm{L}_1\mathrm{N}_3$	$\mathrm{L}\gamma_3$	$\mathrm{M}_3\mathrm{O}_1$	NA
$\mathrm{K}\mathrm{M}_3$	$\mathrm{K}\beta_1$	$\mathrm{L}_1\mathrm{N}_4$	NA	$\mathrm{M}_3\mathrm{O}_4$	NA
$\mathrm{K}\mathrm{N}_{2,3}$	$\mathrm{K}\beta_2$	$\mathrm{L}_1\mathrm{N}_5$	$\mathrm{L}\gamma_{11}$	$\mathrm{M}_3\mathrm{O}_5$	NA
$\mathrm{K}\mathrm{N}_{4,5}$	$\mathrm{K}\beta_4$	$\mathrm{L}_1\mathrm{O}_2$	$\mathrm{L}\gamma^\prime_4$	$\mathrm{M}_4\mathrm{N}_{2,3}$	$\mathrm{M}\zeta_1$
$\mathrm{K}\mathrm{O}_{2}$	$\mathrm{K}\delta_2$	$\mathrm{L}_1\mathrm{O}_3$	$\mathrm{L}\gamma_4$	$\mathrm{M}_4\mathrm{N}_3$	NA
$\mathrm{K}\mathrm{O}_{3}$	$\mathrm{K}\delta_1$	$\mathrm{L}_2\mathrm{M}_1$	$\mathrm{L}\eta$	$\mathrm{M}_4\mathrm{N}_6$	$\mathrm{M}\beta_1$
		$\mathrm{L}_2\mathrm{M}_3$	$\mathrm{L}\beta_{17}$	$\mathrm{M}_5\mathrm{N}_{6,7}$	$\mathrm{M}\alpha_{1,2}$
		$\mathrm{L}_2\mathrm{M}_4$	$\mathrm{L}\beta_1$	$\mathrm{M}_5\mathrm{N}_3$	$\mathrm{M}\zeta_2$
		$\mathrm{L}_2\mathrm{N}_{6,7}$	$\mathrm{L}\nu$	$\mathrm{M}_5\mathrm{N}_6$	$\mathrm{M}\alpha_2$
		$\mathrm{L}_2\mathrm{N}_1$	$\mathrm{L}\gamma_5$	$\mathrm{M}_5\mathrm{N}_7$	$\mathrm{M}\alpha_1$
		$\mathrm{L}_2\mathrm{N}_4$	$\mathrm{L}\gamma_1$
		$\mathrm{L}_2\mathrm{O}_1$	$\mathrm{L}\gamma_8$
		$\mathrm{L}_2\mathrm{O}_4$	$\mathrm{L}\gamma_6$
		$\mathrm{L}_3\mathrm{M}_{4,5}$	$\mathrm{L}\alpha_{1,2}$
		$\mathrm{L}_3\mathrm{M}_1$	$\mathrm{L}\ell$
		$\mathrm{L}_3\mathrm{M}_2$	$\mathrm{L}t$
		$\mathrm{L}_3\mathrm{M}_3$	$\mathrm{L}s$
		$\mathrm{L}_3\mathrm{M}_4$	$\mathrm{L}\alpha_2$
		$\mathrm{L}_3\mathrm{M}_5$	$\mathrm{L}\alpha_1$
		$\mathrm{L}_3\mathrm{N}_{6,7}$	$\mathrm{L}\beta^\prime_{6,7}$
		$\mathrm{L}_3\mathrm{N}_{6,7}$	$\mathrm{L}u$
		$\mathrm{L}_3\mathrm{N}_1$	$\mathrm{L}\beta_6$
		$\mathrm{L}_3\mathrm{N}_4$	$\mathrm{L}\beta_{15}$
		$\mathrm{L}_3\mathrm{N}_5$	$\mathrm{L}\beta_2$
		$\mathrm{L}_3\mathrm{O}_{4,5}$	$\mathrm{L}\beta_5$
		$\mathrm{L}_3\mathrm{O}_1$	$\mathrm{L}\beta_7$

Line energies

Figure 1: Transition energy vs Z.

Figure 2: Distribution of the transition series.

Figure 3: Natural width of the $\mathrm{K}\alpha_1$ and $\mathrm{K}\alpha_2$ lines.

Figure 4: Fluorescence yields of the $\mathrm{K}$, $\mathrm{L}_1$, $\mathrm{L}_2$, and $\mathrm{L}_3$ lines.

Figure 5: $\mathrm{K}\beta$ and $\mathrm{K}\alpha$ relative intensities (in percent). This looks extra messy because most elements have more than one $\mathrm{K}\beta$ or (in particular) more than one $\mathrm{K}\alpha$ transition.

Figure 6: Variations of $\mathrm{K}\beta/\mathrm{K}\alpha$ with atomic number.

The energy of the strongest $\mathrm{K}$, $\mathrm{L}$, and $\mathrm{M}$ lines

Table 2: The energy of the strongest $\mathrm{K}$, $\mathrm{L}$, and $\mathrm{M}$ lines of the elements.
Element	Z	K/keV	L/keV	M/keV
Be	4	0.109
B	5	0.183
C	6	0.277
N	7	0.393
O	8	0.525
F	9	0.677
Ne	10	0.848
Na	11	1.040
Mg	12	1.254
Al	13	1.487
Si	14	1.740
P	15	2.011
S	16	2.310
Cl	17	2.622
Ar	18	2.958
K	19	3.314
Ca	20	3.692
Sc	21	4.093	0.395
Ti	22	4.512	0.452
V	23	4.953	0.51
Cr	24	5.415	0.572
Mn	25	5.900	0.637
Fe	26	6.405	0.705
Co	27	6.931	0.775
Ni	28	7.480	0.849
Cu	29	8.046	0.928
Zn	30	8.637	1.012
Ga	31	9.251	1.098
Ge	32	9.886	1.188
As	33	10.543	1.282
Se	34	11.224	1.379
Br	35	11.924	1.481
Kr	36	12.648	1.585
Rb	37	13.396	1.692
Sr	38	14.165	1.806
Y	39	14.958	1.924
Zr	40	15.775	2.044
Nb	41	16.615	2.169
Mo	42	17.480	2.292
Tc	43	18.367	2.423
Ru	44	19.279	2.558
Rh	45	20.216	2.697
Pd	46	21.177	2.838
Ag	47	22.163	2.983
Cd	48	23.173	3.133
In	49	24.210	3.286
Sn	50	25.271	3.444
Sb	51	26.359	3.604
Te	52	27.473	3.678
I	53	28.612	3.938
Xe	54	29.775	4.11
Cs	55	30.973	4.285
Ba	56	32.194	4.467
La	57	33.442	4.647	0.833
Ce	58	34.720	4.839	0.882
Pr	59	36.027	5.035	0.929
Nd	60	37.361	5.228	0.978
Pm	61	38.725	4.532	1.023
Sm	62	40.118	5.633	1.085
Eu	63	41.542	5.85	1.132
Gd	64	42.996	6.053	1.19
Tb	65	44.482	6.273	1.248
Dy	66	45.999	6.498	1.299
Ho	67	47.547	6.72	1.353
Er	68	49.128	6.949	1.409
Tm	69	50.742	7.18	1.466
Yb	70	52.388	7.416	1.523
Lu	71	54.070	7.655	1.581
Hf	72	55.790	7.899	1.645
Ta	73	57.535	8.146	1.71
W	74	59.318	8.398	1.776
Re	75	61.141	8.652	1.843
Os	76	63.000	8.911	1.91
Ir	77	64.896	9.175	1.98
Pt	78	66.831	9.442	2.073
Au	79	68.806	9.713	2.123
Hg	80	70.818	9.989	2.195
Tl	81	72.872	10.269	2.271
Pb	82	74.970	10.551	2.346
Bi	83	77.107	10.839	2.423
Po	84	79.291	11.131	2.502
At	85	81.516	11.427	2.582
Rn	86	83.785	11.727	2.654
Fr	87	86.106	12.031	2.732
Ra	88	88.478	12.339	2.806
Ac	89	90.884	12.652	2.9
Th	90	93.351	12.968	2.996
Pa	91	95.868	13.291	3.083
U	92	98.440	13.614	3.171
Np	93	101.059	13.946	3.25
Pu	94	103.734	14.282	3.339
Am	95	105.970	14.616	3.429
Cm	96	108.759	14.954	3.525
Bk	97	111.698	15.307	3.616
Cf	98	114.801	15.655	3.709
Es	99	119.080	16.036
Fm	100	122.190	16.402
Md	101	125.390	16.768
No	102	128.660	17.139
Lr	103	132.020	17.5

Table 3: The strongest $\mathrm{K}$, $\mathrm{L}$, and $\mathrm{M}$ lines of the elements sorted by increasing transition energy.
Energy/keV	Elemental transition
0.109	4 Be $\mathrm{K}\mathrm{L}_{2,3}$
0.183	5 B $\mathrm{K}\mathrm{L}_{2,3}$
0.277	6 C $\mathrm{K}\mathrm{L}_{2,3}$
0.393	7 N $\mathrm{K}\mathrm{L}_{2,3}$
0.395	21 Sc $\mathrm{L}_3\mathrm{M}_{4,5}$
0.452	22 Ti $\mathrm{L}_3\mathrm{M}_{4,5}$
0.510	23 V $\mathrm{L}_3\mathrm{M}_5$
0.525	8 O $\mathrm{K}\mathrm{L}_{2,3}$
0.572	24 Cr $\mathrm{L}_3\mathrm{M}_5$
0.637	25 Mn $\mathrm{L}_3\mathrm{M}_5$
0.677	9 F $\mathrm{K}\mathrm{L}_{2,3}$
0.705	26 Fe $\mathrm{L}_3\mathrm{M}_5$
0.775	27 Co $\mathrm{L}_3\mathrm{M}_5$
0.833	57 La $\mathrm{M}_5\mathrm{N}_{6,7}$
0.848	10 Ne $\mathrm{K}\mathrm{L}_{2,3}$
0.849	28 Ni $\mathrm{L}_3\mathrm{M}_5$
0.882	58 Ce $\mathrm{M}_5\mathrm{N}_{6,7}$
0.928	29 Cu $\mathrm{L}_3\mathrm{M}_5$
0.929	59 Pr $\mathrm{M}_5\mathrm{N}_{6,7}$
0.978	60 Nd $\mathrm{M}_5\mathrm{N}_{6,7}$
1.012	30 Zn $\mathrm{L}_3\mathrm{M}_5$
1.023	61 Pm $\mathrm{M}_5\mathrm{N}_{6,7}$
1.040	11 Na $\mathrm{K}\mathrm{L}_3$
1.085	62 Sm $\mathrm{M}_5\mathrm{N}_{6,7}$
1.098	31 Ga $\mathrm{L}_3\mathrm{M}_5$
1.132	63 Eu $\mathrm{M}_5\mathrm{N}_{6,7}$
1.188	32 Ge $\mathrm{L}_3\mathrm{M}_5$
1.190	64 Gd $\mathrm{M}_5\mathrm{N}_{6,7}$
1.248	65 Tb $\mathrm{M}_5\mathrm{N}_{6,7}$
1.254	12 Mg $\mathrm{K}\mathrm{L}_3$
1.282	33 As $\mathrm{L}_3\mathrm{M}_5$
1.299	66 Dy $\mathrm{M}_5\mathrm{N}_{6,7}$
1.353	67 Ho $\mathrm{M}_5\mathrm{N}_{6,7}$
1.379	34 Se $\mathrm{L}_3\mathrm{M}_5$
1.409	68 Er $\mathrm{M}_5\mathrm{N}_{6,7}$
1.466	69 Tm $\mathrm{M}_5\mathrm{N}_{6,7}$
1.481	35 Br $\mathrm{L}_3\mathrm{M}_5$
1.487	13 Al $\mathrm{K}\mathrm{L}_3$
1.523	70 Yb $\mathrm{M}_5\mathrm{N}_7$
1.581	71 Lu $\mathrm{M}_5\mathrm{N}_{6,7}$
1.585	36 Kr $\mathrm{L}_3\mathrm{M}_5$
1.645	72 Hf $\mathrm{M}_5\mathrm{N}_7$
1.692	37 Rb $\mathrm{L}_3\mathrm{M}_5$
1.710	73 Ta $\mathrm{M}_5\mathrm{N}_7$
1.740	14 Si $\mathrm{K}\mathrm{L}_3$
1.776	74 W $\mathrm{M}_5\mathrm{N}_7$
1.806	38 Sr $\mathrm{L}_3\mathrm{M}_5$
1.843	75 Re $\mathrm{M}_5\mathrm{N}_{6,7}$
1.910	76 Os $\mathrm{M}_5\mathrm{N}_{6,7}$
1.924	39 Y $\mathrm{L}_3\mathrm{M}_5$
1.980	77 Ir $\mathrm{M}_5\mathrm{N}_7$
2.011	15 P $\mathrm{K}\mathrm{L}_3$
2.044	40 Zr $\mathrm{L}_3\mathrm{M}_5$
2.073	78 Pt $\mathrm{M}_5\mathrm{N}_7$
2.123	79 Au $\mathrm{M}_5\mathrm{N}_7$
2.169	41 Nb $\mathrm{L}_3\mathrm{M}_5$
2.195	80 Hg $\mathrm{M}_5\mathrm{N}_7$
2.271	81 Tl $\mathrm{M}_5\mathrm{N}_7$
2.292	42 Mo $\mathrm{L}_3\mathrm{M}_5$
2.310	16 S $\mathrm{K}\mathrm{L}_3$
2.346	82 Pb $\mathrm{M}_5\mathrm{N}_7$
2.423	43 Tc $\mathrm{L}_3\mathrm{M}_5$
2.423	83 Bi $\mathrm{M}_5\mathrm{N}_7$
2.502	84 Po $\mathrm{M}_5\mathrm{N}_7$
2.558	44 Ru $\mathrm{L}_3\mathrm{M}_5$
2.582	85 At $\mathrm{M}_5\mathrm{N}_7$
2.622	17 Cl $\mathrm{K}\mathrm{L}_3$
2.654	86 Rn $\mathrm{M}_5\mathrm{N}_7$
2.697	45 Rh $\mathrm{L}_3\mathrm{M}_5$
2.732	87 Fr $\mathrm{M}_5\mathrm{N}_7$
2.806	88 Ra $\mathrm{M}_5\mathrm{N}_7$
2.838	46 Pd $\mathrm{L}_3\mathrm{M}_5$
2.900	89 Ac $\mathrm{M}_5\mathrm{N}_7$
2.958	18 Ar $\mathrm{K}\mathrm{L}_3$
2.983	47 Ag $\mathrm{L}_3\mathrm{M}_5$
2.996	90 Th $\mathrm{M}_5\mathrm{N}_7$
3.083	91 Pa $\mathrm{M}_5\mathrm{N}_7$
3.133	48 Cd $\mathrm{L}_3\mathrm{M}_5$
3.171	92 U $\mathrm{M}_5\mathrm{N}_7$
3.250	93 Np $\mathrm{M}_5\mathrm{N}_7$
3.286	49 In $\mathrm{L}_3\mathrm{M}_5$
3.314	19 K $\mathrm{K}\mathrm{L}_3$
3.339	94 Pu $\mathrm{M}_5\mathrm{N}_7$
3.429	95 Am $\mathrm{M}_5\mathrm{N}_7$
3.444	50 Sn $\mathrm{L}_3\mathrm{M}_5$
3.525	96 Cm $\mathrm{M}_5\mathrm{N}_7$
3.604	51 Sb $\mathrm{L}_3\mathrm{M}_5$
3.616	97 Bk $\mathrm{M}_5\mathrm{N}_7$
3.678	52 Te $\mathrm{L}_3\mathrm{M}_5$
3.692	20 Ca $\mathrm{K}\mathrm{L}_3$
3.709	98 Cf $\mathrm{M}_5\mathrm{N}_7$
3.938	53 I $\mathrm{L}_3\mathrm{M}_5$
4.093	21 Sc $\mathrm{K}\mathrm{L}_3$
4.110	54 Xe $\mathrm{L}_3\mathrm{M}_5$
4.285	55 Cs $\mathrm{L}_3\mathrm{M}_5$
4.467	56 Ba $\mathrm{L}_3\mathrm{M}_5$
4.512	22 Ti $\mathrm{K}\mathrm{L}_3$
4.532	61 Pm $\mathrm{L}_3\mathrm{M}_5$
4.647	57 La $\mathrm{L}_3\mathrm{M}_5$
4.839	58 Ce $\mathrm{L}_3\mathrm{M}_5$
4.953	23 V $\mathrm{K}\mathrm{L}_3$
5.035	59 Pr $\mathrm{L}_3\mathrm{M}_5$
5.228	60 Nd $\mathrm{L}_3\mathrm{M}_5$
5.415	24 Cr $\mathrm{K}\mathrm{L}_3$
5.633	62 Sm $\mathrm{L}_3\mathrm{M}_5$
5.850	63 Eu $\mathrm{L}_3\mathrm{M}_5$
5.900	25 Mn $\mathrm{K}\mathrm{L}_3$
6.053	64 Gd $\mathrm{L}_3\mathrm{M}_5$
6.273	65 Tb $\mathrm{L}_3\mathrm{M}_5$
6.405	26 Fe $\mathrm{K}\mathrm{L}_3$
6.498	66 Dy $\mathrm{L}_3\mathrm{M}_5$
6.720	67 Ho $\mathrm{L}_3\mathrm{M}_5$
6.931	27 Co $\mathrm{K}\mathrm{L}_3$
6.949	68 Er $\mathrm{L}_3\mathrm{M}_5$
7.180	69 Tm $\mathrm{L}_3\mathrm{M}_5$
7.416	70 Yb $\mathrm{L}_3\mathrm{M}_5$
7.480	28 Ni $\mathrm{K}\mathrm{L}_3$
7.655	71 Lu $\mathrm{L}_3\mathrm{M}_5$
7.899	72 Hf $\mathrm{L}_3\mathrm{M}_5$
8.046	29 Cu $\mathrm{K}\mathrm{L}_3$
8.146	73 Ta $\mathrm{L}_3\mathrm{M}_5$
8.398	74 W $\mathrm{L}_3\mathrm{M}_5$
8.637	30 Zn $\mathrm{K}\mathrm{L}_3$
8.652	75 Re $\mathrm{L}_3\mathrm{M}_5$
8.911	76 Os $\mathrm{L}_3\mathrm{M}_5$
9.175	77 Ir $\mathrm{L}_3\mathrm{M}_5$
9.251	31 Ga $\mathrm{K}\mathrm{L}_3$
9.442	78 Pt $\mathrm{L}_3\mathrm{M}_5$
9.713	79 Au $\mathrm{L}_3\mathrm{M}_5$
9.886	32 Ge $\mathrm{K}\mathrm{L}_3$
9.989	80 Hg $\mathrm{L}_3\mathrm{M}_5$
10.269	81 Tl $\mathrm{L}_3\mathrm{M}_5$
10.543	33 As $\mathrm{K}\mathrm{L}_3$
10.551	82 Pb $\mathrm{L}_3\mathrm{M}_5$
10.839	83 Bi $\mathrm{L}_3\mathrm{M}_5$
11.131	84 Po $\mathrm{L}_3\mathrm{M}_5$
11.224	34 Se $\mathrm{K}\mathrm{L}_3$
11.427	85 At $\mathrm{L}_3\mathrm{M}_5$
11.727	86 Rn $\mathrm{L}_3\mathrm{M}_5$
11.924	35 Br $\mathrm{K}\mathrm{L}_3$
12.031	87 Fr $\mathrm{L}_3\mathrm{M}_5$
12.339	88 Ra $\mathrm{L}_3\mathrm{M}_5$
12.648	36 Kr $\mathrm{K}\mathrm{L}_3$
12.652	89 Ac $\mathrm{L}_3\mathrm{M}_5$
12.968	90 Th $\mathrm{L}_3\mathrm{M}_5$
13.291	91 Pa $\mathrm{L}_3\mathrm{M}_5$
13.396	37 Rb $\mathrm{K}\mathrm{L}_3$
13.614	92 U $\mathrm{L}_3\mathrm{M}_5$
13.946	93 Np $\mathrm{L}_3\mathrm{M}_5$
14.165	38 Sr $\mathrm{K}\mathrm{L}_3$
14.282	94 Pu $\mathrm{L}_3\mathrm{M}_5$
14.616	95 Am $\mathrm{L}_3\mathrm{M}_5$
14.954	96 Cm $\mathrm{L}_3\mathrm{M}_5$
14.958	39 Y $\mathrm{K}\mathrm{L}_3$
15.307	97 Bk $\mathrm{L}_3\mathrm{M}_5$
15.655	98 Cf $\mathrm{L}_3\mathrm{M}_5$
15.775	40 Zr $\mathrm{K}\mathrm{L}_3$
16.036	99 Es $\mathrm{L}_3\mathrm{M}_5$
16.402	100 Fm $\mathrm{L}_3\mathrm{M}_5$
16.615	41 Nb $\mathrm{K}\mathrm{L}_3$
16.768	101 Md $\mathrm{L}_3\mathrm{M}_5$
17.139	102 No $\mathrm{L}_3\mathrm{M}_5$
17.480	42 Mo $\mathrm{K}\mathrm{L}_3$
17.500	103 Lr $\mathrm{L}_3\mathrm{M}_5$
18.367	43 Tc $\mathrm{K}\mathrm{L}_3$
19.279	44 Ru $\mathrm{K}\mathrm{L}_3$
20.216	45 Rh $\mathrm{K}\mathrm{L}_3$
21.177	46 Pd $\mathrm{K}\mathrm{L}_3$
22.163	47 Ag $\mathrm{K}\mathrm{L}_3$
23.173	48 Cd $\mathrm{K}\mathrm{L}_3$
24.210	49 In $\mathrm{K}\mathrm{L}_3$
25.271	50 Sn $\mathrm{K}\mathrm{L}_3$
26.359	51 Sb $\mathrm{K}\mathrm{L}_3$
27.473	52 Te $\mathrm{K}\mathrm{L}_3$
28.612	53 I $\mathrm{K}\mathrm{L}_3$
29.775	54 Xe $\mathrm{K}\mathrm{L}_3$
30.973	55 Cs $\mathrm{K}\mathrm{L}_3$
32.194	56 Ba $\mathrm{K}\mathrm{L}_3$
33.442	57 La $\mathrm{K}\mathrm{L}_3$
34.720	58 Ce $\mathrm{K}\mathrm{L}_3$
36.027	59 Pr $\mathrm{K}\mathrm{L}_3$
37.361	60 Nd $\mathrm{K}\mathrm{L}_3$
38.725	61 Pm $\mathrm{K}\mathrm{L}_3$
40.118	62 Sm $\mathrm{K}\mathrm{L}_3$
41.542	63 Eu $\mathrm{K}\mathrm{L}_3$
42.996	64 Gd $\mathrm{K}\mathrm{L}_3$
44.482	65 Tb $\mathrm{K}\mathrm{L}_3$
45.999	66 Dy $\mathrm{K}\mathrm{L}_3$
47.547	67 Ho $\mathrm{K}\mathrm{L}_3$
49.128	68 Er $\mathrm{K}\mathrm{L}_3$
50.742	69 Tm $\mathrm{K}\mathrm{L}_3$
52.388	70 Yb $\mathrm{K}\mathrm{L}_3$
54.070	71 Lu $\mathrm{K}\mathrm{L}_3$
55.790	72 Hf $\mathrm{K}\mathrm{L}_3$
57.535	73 Ta $\mathrm{K}\mathrm{L}_3$
59.318	74 W $\mathrm{K}\mathrm{L}_3$
61.141	75 Re $\mathrm{K}\mathrm{L}_3$
63.000	76 Os $\mathrm{K}\mathrm{L}_3$
64.896	77 Ir $\mathrm{K}\mathrm{L}_3$
66.831	78 Pt $\mathrm{K}\mathrm{L}_3$
68.806	79 Au $\mathrm{K}\mathrm{L}_3$
70.818	80 Hg $\mathrm{K}\mathrm{L}_3$
72.872	81 Tl $\mathrm{K}\mathrm{L}_3$
74.970	82 Pb $\mathrm{K}\mathrm{L}_3$
77.107	83 Bi $\mathrm{K}\mathrm{L}_3$
79.291	84 Po $\mathrm{K}\mathrm{L}_3$
81.516	85 At $\mathrm{K}\mathrm{L}_3$
83.785	86 Rn $\mathrm{K}\mathrm{L}_3$
86.106	87 Fr $\mathrm{K}\mathrm{L}_3$
88.478	88 Ra $\mathrm{K}\mathrm{L}_3$
90.884	89 Ac $\mathrm{K}\mathrm{L}_3$
93.351	90 Th $\mathrm{K}\mathrm{L}_3$
95.868	91 Pa $\mathrm{K}\mathrm{L}_3$
98.440	92 U $\mathrm{K}\mathrm{L}_3$
101.059	93 Np $\mathrm{K}\mathrm{L}_3$
103.734	94 Pu $\mathrm{K}\mathrm{L}_3$
105.970	95 Am $\mathrm{K}\mathrm{L}_3$
108.759	96 Cm $\mathrm{K}\mathrm{L}_3$
111.698	97 Bk $\mathrm{K}\mathrm{L}_3$
114.801	98 Cf $\mathrm{K}\mathrm{L}_3$
119.080	99 Es $\mathrm{K}\mathrm{L}_3$
122.190	100 Fm $\mathrm{K}\mathrm{L}_3$
125.390	101 Md $\mathrm{K}\mathrm{L}_3$
128.660	102 No $\mathrm{K}\mathrm{L}_3$
132.020	103 Lr $\mathrm{K}\mathrm{L}_3$

Bibliography

Offered in no particular order, and in no particular style.

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] common_0.0.2  xtable_1.8-4  readODS_1.7.0 reshape_0.8.8 plyr_1.8.6   
## [6] ggplot2_3.3.5 knitr_1.37   
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.8       bslib_0.3.1      compiler_4.0.5   pillar_1.7.0    
##  [5] jquerylib_0.1.4  tools_4.0.5      digest_0.6.29    jsonlite_1.7.3  
##  [9] evaluate_0.14    lifecycle_1.0.1  tibble_3.1.6     gtable_0.3.0    
## [13] pkgconfig_2.0.3  rlang_1.0.1      DBI_1.1.2        cli_3.2.0       
## [17] yaml_2.2.2       blogdown_1.7     xfun_0.29        fastmap_1.1.0   
## [21] withr_2.4.3      dplyr_1.0.8      stringr_1.4.0    generics_0.1.2  
## [25] sass_0.4.0       vctrs_0.3.8      tidyselect_1.1.2 grid_4.0.5      
## [29] glue_1.6.1       R6_2.5.1         fansi_1.0.2      rmarkdown_2.11  
## [33] bookdown_0.24    purrr_0.3.4      magrittr_2.0.2   scales_1.1.1    
## [37] htmltools_0.5.2  ellipsis_0.3.2   assertthat_0.2.1 colorspace_2.0-2
## [41] utf8_1.2.2       stringi_1.7.6    munsell_0.5.0    crayon_1.5.0

References

Ahmed, Taha. 2014a. “Characteristic X-ray lines of the elements.” https://doi.org/10.6084/m9.figshare.1168939.

———. 2014b. “IUPAC vs Siegbahn notation of x-ray transitions.” https://doi.org/10.6084/m9.figshare.1168938.

Krause, M. O. 1979. “Atomic Radiative and Radiationless Yields for K and L Shells.” Journal of Physical and Chemical Reference Data 8 (2): 307–27. https://doi.org/10.1063/1.555594.

Krause, M. O., and J. H. Oliver. 1979. “Natural Widths of Atomic K and L Levels, K$\alpha$ X-ray Lines and Several KLL Auger Lines.” Journal of Physical and Chemical Reference Data 8 (2): 329. https://doi.org/10.1063/1.555595.

Lengyel, Tamás, Allan M. Ure, and János Inczédy. 1998. Compendium of Analytical Nomenclature: Definitive Rules 1997. Edited by Tamás Lengyel, Allan M. Ure, and János Inczédy. 3rd ed. Blackwell Science. http://old.iupac.org/publications/analytical_compendium/.

How to install Rstudio's Shiny on Ubuntu 14.04

Thu, 17 Jul 2014 20:00:00 +0100

Installing shiny and shiny-server (on localhost)

Assuming you already have a working R installation on a Debian-like system.

First step, install the shiny R package.

sudo su - -c "R -e \"install.packages('shiny', repos='http://cran.rstudio.com/')\""

Download and install the shiny server software from RStudio (the shiny package is installed using gdebi, so you need to install that too if you don’t already have it).

sudo apt-get install gdebi-core
wget http://download3.rstudio.org/ubuntu-12.04/x86_64/shiny-server-1.3.0.403-amd64.deb
sudo gdebi shiny-server-1.3.0.403-amd64.deb

RStudio offers shiny-server for free with the AGPL v3 license but with the limit of a single R process per shiny application, and without support for authentication or SSL. The former limit is not an issue for lightweight shiny apps, but the latter can be problematic if you intend to host public-facing shiny apps on your own server. One possible way of getting around this limit could be to setup a reverse proxy using an SSH tunnel to access the shiny app, but I have not been able to make such a setup work myself.

Running the shiny app

I use a dashboard to keep track of my project’s sample matrix (as we chemists like to call it). It’s run using a short bash script with essentially the following contents:

#!/bin/bash
PORT=$(shuf -i 2000-10000 -n 1)
# write the port number to file
echo $PORT > /path/to/shinyapp/directory/shiny.port
nohup R -e "shiny::runApp('/path/to/shinyapp/directory', port = $PORT)"

The bash script is called from an Ubuntu autostart script so that the dashboard starts on system login. Accessing it is then as simple as pointing your browser to localhost:<port-number>. Since we don’t want to check the port number manually, something along these lines is used to start the browser:

#!/bin/bash
PORT=$(</path/to/shinyapp/directory/shiny.port)
s="http://localhost:$PORT"
google-chrome $s

Of course, randomly assigning the shiny app port number is not necessary. It is simply a remnant from the development phase of this app - changing the port number on each launch of the app made debugging easier (one less variable to worry about…).

The finished dashboard app contains a lot of links to local pdf files, so depending on browser you may have to disable the restriction to open file:///-type hyperlinks.

Sources

Publishing R Markdown using Jekyll

Wed, 16 Jul 2014 16:00:00 +0100

After getting a barebones Jekyll install up and running, my next step was to work out how to post R Markdown files to it. There is nothing wrong with regular markdown, but after using R for several years and LaTeX for many more years, I preferred the larger set of possibilities that come with R Markdown.

There are two major routes to work with R Markdown source files with Jekyll, one involves using the Jekyll plugins system, and the other uses R (in effect the knitr::render_jekyll() function) to pre-process all R Markdown files to Jekyll-compatible markdown files.

The plugins route appeared quite convoluted, and being unfamiliar with Ruby, gems, etc. did not make my situation any better. I found a few pages describing the process, but the temptation of using more familiar tools was too great.

R (knitr) pre-process route

I used all of the listed pages for inspiration, but I based my solution heavily on dfeng’s make.R script.

In my Jekyll site source directory, I created a _knitr directory, which will hold R Markdown files. In the same directory, I placed my Rmd -> md conversion script (called render_post.R).

#!/usr/bin/env Rscript
options(stringsAsFactors = FALSE)

# inspiration sources:
# http://www.jonzelner.net/jekyll/knitr/r/2014/07/02/autogen-knitr/
# http://gtog.github.io/workflow/2013/06/12/rmarkdown-to-rbloggers/

KnitPost <- function(bashwd = "", convert_file = "", overwrite = FALSE) {
   # CONVERT ALL RMD FILES TO MARKDOWN?
   #    REQUIRED: overwrite
   # CONVERT A SPECIFIC RMD FILE TO MARKDOWN?
   #    REQUIRED: bashwd, convert_file

   # bashwd: working directory (passed on from bash, only used below
   #         if a specific Rmd file is to be converted)
   # convert_file: name/path to specific Rmd file to convert
   # overwrite: flag that tells us whether to overwrite md files
   #            when converting all Rmd files

   # directory of jekyll blog (including trailing slash)
   site.path <- "/home/taha/jekyll/chepec/"
   # directory where your Rmd-files reside (relative to base)
   rmd.path <- paste0(site.path, "_knitr")
   # directory to save figures
   fig.dir <- "figures/"
   # directory for converted markdown files
   posts.path <- paste0(site.path, "_posts")
   # cache
   cache.path <- paste0(site.path, "_cache")

   require(knitr)
   render_jekyll(highlight = "pygments")
   # "base.dir is never used when composing the URL of the figures; it is
   # only used to save the figures to a different directory, which can
   # be useful when you do not want the figures to be saved under the
   # current working directory.
   # The URL of an image is always base.url + fig.path"
   # https://groups.google.com/forum/#!topic/knitr/18aXpOmsumQ
   opts_knit$set(
      base.url = "/",
      base.dir = site.path)
   opts_chunk$set(
      fig.path   = fig.dir,
      fig.width  = 8.5,
      fig.height = 5.25,
      dev        = 'svg',
      cache      = FALSE,
      warning    = FALSE,
      message    = FALSE,
      cache.path = cache.path,
      tidy       = FALSE)


   if (convert_file == "") {
      # convert all Rmd files in {site-url}/_knitr/ to markdown files
      # contingent on whether overwrite requested and if md exists

      # setwd to Rmd folder
      setwd(rmd.path)

      files.rmd <-
         data.frame(rmd = list.files(
            path = rmd.path,
            full.names = TRUE,
            pattern = "\\.Rmd$",
            ignore.case = TRUE,
            recursive = FALSE))

      files.rmd$corresponding.md.file <-
         paste0(posts.path, "/",
            basename(gsub(pattern = "\\.Rmd$",
               replacement = ".md",
               x = files.rmd$rmd)))

      files.rmd$corresponding.md.exists <-
         file.exists(files.rmd$corresponding.md.file)
      files.rmd$md.overwrite <- overwrite
      files.rmd$md.render <- FALSE
      # check if corresponding md file exists for each Rmd file,
      # if not, set flag to convert to markdown
      # (also consider the overwrite flag set by user)
      for (i in 1:dim(files.rmd)[1]) {
         if (files.rmd$corresponding.md.exists[i] == FALSE) {
            files.rmd$md.render[i] <- TRUE
         }
         if ((files.rmd$corresponding.md.exists[i] == TRUE) &&
             (files.rmd$md.overwrite[i] == TRUE)) {
            files.rmd$md.render[i] <- TRUE
         }
      }

      # For each Rmd file, render markdown (contingent on the flags set above)
      for (i in 1:dim(files.rmd)[1]) {
         # only re-knit if overwrite==TRUE or .md not already existing
         if (files.rmd$md.render[i] == TRUE) {
            # KNITTING ====
            message(paste0("=== KnitPost(overwrite=", overwrite, "): ",
                           basename(files.rmd$rmd[i])))
            out.file <-
               knit(files.rmd$rmd[i],
                  output = files.rmd$corresponding.md.file[i],
                  envir = parent.frame(),
                  quiet = TRUE)
         }
      }
   } else {
      # convert a single Rmd file to markdown
      # setwd to bash pwd
      setwd(bashwd)
      convert.path <- paste0(bashwd, "/", convert_file)
      md.path <-
         paste0(posts.path, "/",
         basename(gsub(pattern = "\\.Rmd$",
            replacement = ".md",
            x = convert_file)))
      # KNITTING ====
      message(paste0("=== KnitPost(", convert.path, ")"))
      out.file <-
         knit(convert.path,
            output = md.path,
            envir = parent.frame(),
            quiet = TRUE)
   }
}

Now we just need to run this script whenever we have new or updated Rmd-files in the _knitr directory. Since I already created a bash script that handles the jekyll build command, I decided to handle the R script in the same place.

#!/bin/bash

function show_help {
   echo "Usage: buildsite.sh [OPTION]..."
    echo "Knit posts, rebuild jekyll blog <chepec>"
    echo ""
    echo "-c   convert all _knitr/*.Rmd files to _posts/*.md (does not overwrite existing md)"
    echo "-o   overwrite existing *.md files"
   echo "-f   convert '~/jekyll/chepec/_knitr/<filename>.Rmd' to md in _posts"
    echo "-b   build the jekyll site"
    echo "-s   send changes to live dir"
    echo "-h   show this help"
}

rmdfile=""
bashpwd=$(pwd)

# initialise flags here
# convert new *.Rmd => *.md?
convert=false
# convert even if *.md exists?
overwrite_md=false
# convert a specific Rmd file?
convert_file=false
# build jekyll site?
buildsite=false
# make changes live?
send_live=false

if [ $# -eq 0 ] ; then
   # no args at all? show usage
   show_help
   exit 0
fi

# reset OPTINDEX in case getopts has been used previously in the shell
OPTIND=1
while getopts "h?cf:obs" opt; do
    case $opt in
        c)
            # just a flag, no options
            convert=true
            ;;
      f)
         # has a mandatory option, filename
         convert_file=true
         rmdfile=$OPTARG
         ;;
        o)
            # just a flag, no options
            overwrite_md=true
            ;;
        b)
            # just a flag, no options
            buildsite=true
            ;;
      s)
            # just a flag, no options
            send_live=true
            ;;
        h|\?)
            show_help
            exit 0
    esac
done
shift $((OPTIND-1))

#### 1. convert/overwrite *.Rmd to *.md
# uses the function KnitPost() from the R-script render_post.R
if [ "$convert" = true ] ; then
    echo ">>>> jc: Converting RMarkdown to Markdown (overwrite = $overwrite_md)"
    if [ "$overwrite_md" = true ] ; then
        Rscript -e "source('~/jekyll/chepec/_knitr/render_post.R'); KnitPost(overwrite=TRUE)"
    else
        Rscript -e "source('~/jekyll/chepec/_knitr/render_post.R'); KnitPost()"
    fi
fi

#### 2. convert a specific Rmd-file
# uses the function KnitPost() from the R-script render_post.R
if [ "$convert_file" = true ] ; then
   # check that $rmdfile exists
   if [ -f $rmdfile ]; then
      Rscript -e "source('~/jekyll/chepec/_knitr/render_post.R'); KnitPost(bashwd='$bashpwd',convert_file='$rmdfile')"
   else
      show_help
      exit 0
   fi
fi

#### 3. build the site
if [ "$buildsite" = true ] ; then
    echo ">>>> jc: Building site to /var/www/blog"
    jekyll build -s ~/jekyll/chepec -d /var/www/blog
fi

#### 4. make changes live (send to public server)
if [ "$send_live" = true ] ; then
    echo ">>>> jc: Sending changes to live server"
    unison jekyll-damietta -auto
fi

exit 0

This way, we keep the Jekyll source directory exclusively on the local machine, and only sync the site directory with the remote machine (the public-facing server). ~~We still need to maintain a Jekyll install on both the local and remote machine, though – haven’t found a way around that.~~ Correction: we only require a Jekyll stack on the machine where we build the site. Once Jekyll has built the site, the result is just a bunch of html, css and js that can be served to almost any browser.

Installing Jekyll

Fri, 11 Jul 2014 16:00:00 +0100

I got Jekyll up and running on my server, and most of the time it was simply a matter of following the instructions. But some steps were not so obvious, so for posterity, here it is.

Installing Jekyll via gems and Ruby

gem is a Ruby command, so we will need to install Ruby first and then Rubygems.

Ruby

There are many ways to install Ruby, but since the Debian/Ubuntu repos tend to be slightly outdated

Downloaded and ran the RVM installation script

\curl -sSL https://get.rvm.io | bash -s stable

Sourced it:

source ~/.rvm/scripts/rvm

To test that RVM has been properly installed, we can issue

type rvm | head -n 1

and should get rvm is a function back. We have thus successfully installed RVM. But we have not installed Ruby yet.

RVM gives us plenty of choice when it comes to the flavour of Ruby we want, and it took some googling to figure out what distinguishes them. It turns out, not so much, and since we are noobs, we will go with MRI Ruby which is the original implementation, named after Ruby inventor and author Yukihiro Matsumoto.

rvm install 2.1.1

We installed MRI Ruby 2.1.1. Now we need to tell RVM to use this as default for new shells.

rvm use 2.1.1 --default

That’s it for Ruby.

RubyGems

This is fairly straight-forward. Download, unpack and install the latest rubygems.

wget http://production.cf.rubygems.org/rubygems/rubygems-2.3.0.zip
unzip rubygems-2.3.0.zip
cd rubygems-2.3.0
ruby setup.rb

That’s it for RubyGems.

NodeJS

This is another requirement of Jekyll. On a recent Ubuntu install, it would be as easy as

apt-get install nodejs

but on Debian, it is available through Wheezy backports or you’ll have to build from source. The former is easier so we’ll add the backport repo and install as usual.

echo "deb http://ftp.us.debian.org/debian wheezy-backports main" >> /etc/apt/sources.list
apt-get update
apt-get install nodejs

Jekyll

Now we can install Jekyll by running:

gem install jekyll

(this will install a bunch of other gems as well).

At this point, running jekyll without any arguments should present you with

jekyll 2.1.1 -- Jekyll is a blog-aware, static site generator in Ruby

followed by jekyll’s help message.

Creating and building the site

I put the source directory in my home directory tree, and the site folder in the /var/www tree.

The initial source skeleton was created using jekyll new, and after making some changes to _config.yml I built the site using jekyll build --source $SOURCE --destination $DESTINATION where $SOURCE and $DESTINATION were the path of the source directory (in the home tree) and the path of the destination directory (in the webserver tree), respectively.

Note that it was very important to set the baseurl: property in _config.yml so that it corresponds to where the destination directory resides. For example, if the site is placed in /var/www/blog, we should set baseurl: /blog. At first, I did not realise this, and the site rendered very poorly due to all relative links being broken, including those that pointed to the css-files.

Actually, I ended up removing the option baseurl altogether from _config.yml. Turns out baseurl can be quirky.

Working the site (basics)

I dabbled a little with git, trying to setup a post-receive hook, but lost patience and created a bash script that handles it instead.

That part is described in my next post.

Conference talk: ECS 2013

Tue, 29 Oct 2013 18:00:00 +0100

I attended the 224th Electrochemical Society meeting in San Francisco, California.

I contributed an oral presentation titled “Electrodeposited zinc oxide nanorods ALD-coated with iron oxide: photocatalytic and photoelectrochemical properties”.

The slides and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.3569688.v1

The slides were prepared using beamer and knitr. Several plots were dynamically generated on the slide using knitr and ggplot2, as can be seen in the source.

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

Conference poster: MRS spring 2013

Fri, 05 Apr 2013 18:00:00 +0100

I attended the 2013 Materials Research Society (MRS) Spring Meeting at Moscone Center, San Francisco, California.

We presented the poster “A Facile Approach to ZnO/CdS Nanoarrays and Their Photocatalytic and Photoelectrochemical Properties”, which was based on our recent publication. (Li et al. 2013)

The poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.850371.v2

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

References

Li, Chunmei, Taha Ahmed, Mingguo Ma, Tomas Edvinsson, and Jiefang Zhu. 2013. “A Facile Approach to ZnO/CdS Nanoarrays and Their Photocatalytic and Photoelectrochemical Properties.” Applied Catalysis B: Environmental 138–139 (July): 175–83. https://doi.org/10.1016/j.apcatb.2013.02.042.

Conference poster: IPS-19 2012

Fri, 03 Aug 2012 18:00:00 +0100

I attended the 19th International Conference on Photochemical Conversion and Storage of Solar Energy (IPS-19) at Caltech, Pasadena, California.

We presented two posters, the first on “Preparation, structure and properties of zinc oxide/cadmium sulfide photoanodes”, and the second on “Electrodeposited zinc oxide nanorods coated with atomic-layer-deposited hematite film”.

Most plots were made using ggplot2, and physical peak fitting was done using the R package diffractometry by Davies et al.

This work was later published as Li et al. (2013).

The above poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.850370.v2

The above poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.3569661.v1

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

References

Conference poster: GradSAM 2012

Wed, 08 Feb 2012 18:00:00 +0100

I attended the GradSAM 2012 conference, which was organised by the Graduate School in Advanced Materials for the 21st century at the faculty of Science and technology, Uppsala university (a local conference primarily for the doctoral students at the faculty).

We presented the poster “Electrodeposited zinc oxide nanorods coated with atomic-layer deposited iron oxide”, which included some X-ray flyuorescence, SEM, and optical UV/Vis characterisation data of $\ce{ZnO}/\ce{Fe2O3}$ nanorod arrays.

I made the poster using LaTeX (with the beamerposter package), Sweave and R.

The poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.3569655.v1

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

Conference poster: GradSAM 2011

Wed, 09 Feb 2011 18:00:00 +0100

I attended the GradSAM 2011 conference, which was organised by the Graduate School in Advanced Materials for the 21st century at the faculty of Science and technology, Uppsala university (a local conference primarily for the doctoral students at the faculty).

We presented the poster “Template-assisted electrodeposition of $\ce{TiO2}$ on transparent conductive oxide substrates”, which dealt with some aspects of the synthesis of nanoparticulate titania.

I made the poster using LaTeX (with the beamerposter package), Sweave and R, with all of the illustrations done using PGF/TikZ. The periodic table was written in TikZ and adapted from Ivan Griffin’s Periodic Table of Chemical Elements.

The poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.3569628.v1

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

Conference poster: Oorgandagarna 2010

Sat, 19 Jun 2010 18:00:00 +0100

I attended the conference Oorgandagarna 2010 (“Inorganic Days”) which was organised by the Swedish Chemical Society at SLU, Uppsala, June 16–19 2010.

We presented the poster “Semiconducting oxides for photoelectrochemical water splitting”. I made the poster using LaTeX and TikZ with the beamerposter package.

The poster and its source files are posted on Figshare.
https://dx.doi.org/10.6084/m9.figshare.3569154.v1

Tools

Session info

## R version 4.0.5 (2021-03-31)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 18.04.6 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] knitr_1.37
## 
## loaded via a namespace (and not attached):
##  [1] bookdown_0.24   digest_0.6.29   R6_2.5.1        jsonlite_1.7.3  magrittr_2.0.2 
##  [6] evaluate_0.14   blogdown_1.7    stringi_1.7.6   rlang_1.0.1     cli_3.2.0      
## [11] jquerylib_0.1.4 bslib_0.3.1     rmarkdown_2.11  tools_4.0.5     stringr_1.4.0  
## [16] xfun_0.29       yaml_2.2.2      fastmap_1.1.0   compiler_4.0.5  htmltools_0.5.2
## [21] sass_0.4.0

Mon, 01 Jan 0001 00:00:00 +0000

template-blog-post

Template git repo for blog post, used by create-blog-post.sh script

IUPAC	Siegbahn	IUPAC	Siegbahn	IUPAC	Siegbahn
\(\mathrm{K}\mathrm{L}_{2,3}\)	\(\mathrm{K}\alpha_{1,2}\)	\(\mathrm{L}_1\mathrm{M}_{2,3}\)	\(\mathrm{K}\beta_{3,4}\)	\(\mathrm{M}_{4,5}\mathrm{N}_{2,3}\)	\(\mathrm{M}\zeta_{1,2}\)
\(\mathrm{K}\mathrm{L}_1\)	\(\mathrm{K}\alpha_3\)	\(\mathrm{L}_1\mathrm{M}_2\)	\(\mathrm{L}\beta_4\)	\(\mathrm{M}_1\mathrm{N}_3\)	NA
\(\mathrm{K}\mathrm{L}_2\)	\(\mathrm{K}\alpha_2\)	\(\mathrm{L}_1\mathrm{M}_3\)	\(\mathrm{L}\beta_3\)	\(\mathrm{M}_2\mathrm{N}_4\)	NA
\(\mathrm{K}\mathrm{L}_3\)	\(\mathrm{K}\alpha_1\)	\(\mathrm{L}_1\mathrm{M}_4\)	\(\mathrm{L}\beta_{10}\)	\(\mathrm{M}_3\mathrm{N}_1\)	NA
\(\mathrm{K}\mathrm{M}_{2,3}\)	\(\mathrm{K}\beta_{1,3}\)	\(\mathrm{L}_1\mathrm{M}_5\)	\(\mathrm{L}\beta_9\)	\(\mathrm{M}_3\mathrm{N}_4\)	\(\mathrm{M}\gamma_2\)
\(\mathrm{K}\mathrm{M}_{4,5}\)	\(\mathrm{K}\beta_5\)	\(\mathrm{L}_1\mathrm{N}_2\)	\(\mathrm{L}\gamma_2\)	\(\mathrm{M}_3\mathrm{N}_5\)	\(\mathrm{M}\gamma_1\)
\(\mathrm{K}\mathrm{M}_2\)	\(\mathrm{K}\beta_3\)	\(\mathrm{L}_1\mathrm{N}_3\)	\(\mathrm{L}\gamma_3\)	\(\mathrm{M}_3\mathrm{O}_1\)	NA
\(\mathrm{K}\mathrm{M}_3\)	\(\mathrm{K}\beta_1\)	\(\mathrm{L}_1\mathrm{N}_4\)	NA	\(\mathrm{M}_3\mathrm{O}_4\)	NA
\(\mathrm{K}\mathrm{N}_{2,3}\)	\(\mathrm{K}\beta_2\)	\(\mathrm{L}_1\mathrm{N}_5\)	\(\mathrm{L}\gamma_{11}\)	\(\mathrm{M}_3\mathrm{O}_5\)	NA
\(\mathrm{K}\mathrm{N}_{4,5}\)	\(\mathrm{K}\beta_4\)	\(\mathrm{L}_1\mathrm{O}_2\)	\(\mathrm{L}\gamma^\prime_4\)	\(\mathrm{M}_4\mathrm{N}_{2,3}\)	\(\mathrm{M}\zeta_1\)
\(\mathrm{K}\mathrm{O}_{2}\)	\(\mathrm{K}\delta_2\)	\(\mathrm{L}_1\mathrm{O}_3\)	\(\mathrm{L}\gamma_4\)	\(\mathrm{M}_4\mathrm{N}_3\)	NA
\(\mathrm{K}\mathrm{O}_{3}\)	\(\mathrm{K}\delta_1\)	\(\mathrm{L}_2\mathrm{M}_1\)	\(\mathrm{L}\eta\)	\(\mathrm{M}_4\mathrm{N}_6\)	\(\mathrm{M}\beta_1\)
		\(\mathrm{L}_2\mathrm{M}_3\)	\(\mathrm{L}\beta_{17}\)	\(\mathrm{M}_5\mathrm{N}_{6,7}\)	\(\mathrm{M}\alpha_{1,2}\)
		\(\mathrm{L}_2\mathrm{M}_4\)	\(\mathrm{L}\beta_1\)	\(\mathrm{M}_5\mathrm{N}_3\)	\(\mathrm{M}\zeta_2\)
		\(\mathrm{L}_2\mathrm{N}_{6,7}\)	\(\mathrm{L}\nu\)	\(\mathrm{M}_5\mathrm{N}_6\)	\(\mathrm{M}\alpha_2\)
		\(\mathrm{L}_2\mathrm{N}_1\)	\(\mathrm{L}\gamma_5\)	\(\mathrm{M}_5\mathrm{N}_7\)	\(\mathrm{M}\alpha_1\)
		\(\mathrm{L}_2\mathrm{N}_4\)	\(\mathrm{L}\gamma_1\)
		\(\mathrm{L}_2\mathrm{O}_1\)	\(\mathrm{L}\gamma_8\)
		\(\mathrm{L}_2\mathrm{O}_4\)	\(\mathrm{L}\gamma_6\)
		\(\mathrm{L}_3\mathrm{M}_{4,5}\)	\(\mathrm{L}\alpha_{1,2}\)
		\(\mathrm{L}_3\mathrm{M}_1\)	\(\mathrm{L}\ell\)
		\(\mathrm{L}_3\mathrm{M}_2\)	\(\mathrm{L}t\)
		\(\mathrm{L}_3\mathrm{M}_3\)	\(\mathrm{L}s\)
		\(\mathrm{L}_3\mathrm{M}_4\)	\(\mathrm{L}\alpha_2\)
		\(\mathrm{L}_3\mathrm{M}_5\)	\(\mathrm{L}\alpha_1\)
		\(\mathrm{L}_3\mathrm{N}_{6,7}\)	\(\mathrm{L}\beta^\prime_{6,7}\)
		\(\mathrm{L}_3\mathrm{N}_{6,7}\)	\(\mathrm{L}u\)
		\(\mathrm{L}_3\mathrm{N}_1\)	\(\mathrm{L}\beta_6\)
		\(\mathrm{L}_3\mathrm{N}_4\)	\(\mathrm{L}\beta_{15}\)
		\(\mathrm{L}_3\mathrm{N}_5\)	\(\mathrm{L}\beta_2\)
		\(\mathrm{L}_3\mathrm{O}_{4,5}\)	\(\mathrm{L}\beta_5\)
		\(\mathrm{L}_3\mathrm{O}_1\)	\(\mathrm{L}\beta_7\)

Energy/keV	Elemental transition
0.109	4 Be \(\mathrm{K}\mathrm{L}_{2,3}\)
0.183	5 B \(\mathrm{K}\mathrm{L}_{2,3}\)
0.277	6 C \(\mathrm{K}\mathrm{L}_{2,3}\)
0.393	7 N \(\mathrm{K}\mathrm{L}_{2,3}\)
0.395	21 Sc \(\mathrm{L}_3\mathrm{M}_{4,5}\)
0.452	22 Ti \(\mathrm{L}_3\mathrm{M}_{4,5}\)
0.510	23 V \(\mathrm{L}_3\mathrm{M}_5\)
0.525	8 O \(\mathrm{K}\mathrm{L}_{2,3}\)
0.572	24 Cr \(\mathrm{L}_3\mathrm{M}_5\)
0.637	25 Mn \(\mathrm{L}_3\mathrm{M}_5\)
0.677	9 F \(\mathrm{K}\mathrm{L}_{2,3}\)
0.705	26 Fe \(\mathrm{L}_3\mathrm{M}_5\)
0.775	27 Co \(\mathrm{L}_3\mathrm{M}_5\)
0.833	57 La \(\mathrm{M}_5\mathrm{N}_{6,7}\)
0.848	10 Ne \(\mathrm{K}\mathrm{L}_{2,3}\)
0.849	28 Ni \(\mathrm{L}_3\mathrm{M}_5\)
0.882	58 Ce \(\mathrm{M}_5\mathrm{N}_{6,7}\)
0.928	29 Cu \(\mathrm{L}_3\mathrm{M}_5\)
0.929	59 Pr \(\mathrm{M}_5\mathrm{N}_{6,7}\)
0.978	60 Nd \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.012	30 Zn \(\mathrm{L}_3\mathrm{M}_5\)
1.023	61 Pm \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.040	11 Na \(\mathrm{K}\mathrm{L}_3\)
1.085	62 Sm \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.098	31 Ga \(\mathrm{L}_3\mathrm{M}_5\)
1.132	63 Eu \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.188	32 Ge \(\mathrm{L}_3\mathrm{M}_5\)
1.190	64 Gd \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.248	65 Tb \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.254	12 Mg \(\mathrm{K}\mathrm{L}_3\)
1.282	33 As \(\mathrm{L}_3\mathrm{M}_5\)
1.299	66 Dy \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.353	67 Ho \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.379	34 Se \(\mathrm{L}_3\mathrm{M}_5\)
1.409	68 Er \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.466	69 Tm \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.481	35 Br \(\mathrm{L}_3\mathrm{M}_5\)
1.487	13 Al \(\mathrm{K}\mathrm{L}_3\)
1.523	70 Yb \(\mathrm{M}_5\mathrm{N}_7\)
1.581	71 Lu \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.585	36 Kr \(\mathrm{L}_3\mathrm{M}_5\)
1.645	72 Hf \(\mathrm{M}_5\mathrm{N}_7\)
1.692	37 Rb \(\mathrm{L}_3\mathrm{M}_5\)
1.710	73 Ta \(\mathrm{M}_5\mathrm{N}_7\)
1.740	14 Si \(\mathrm{K}\mathrm{L}_3\)
1.776	74 W \(\mathrm{M}_5\mathrm{N}_7\)
1.806	38 Sr \(\mathrm{L}_3\mathrm{M}_5\)
1.843	75 Re \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.910	76 Os \(\mathrm{M}_5\mathrm{N}_{6,7}\)
1.924	39 Y \(\mathrm{L}_3\mathrm{M}_5\)
1.980	77 Ir \(\mathrm{M}_5\mathrm{N}_7\)
2.011	15 P \(\mathrm{K}\mathrm{L}_3\)
2.044	40 Zr \(\mathrm{L}_3\mathrm{M}_5\)
2.073	78 Pt \(\mathrm{M}_5\mathrm{N}_7\)
2.123	79 Au \(\mathrm{M}_5\mathrm{N}_7\)
2.169	41 Nb \(\mathrm{L}_3\mathrm{M}_5\)
2.195	80 Hg \(\mathrm{M}_5\mathrm{N}_7\)
2.271	81 Tl \(\mathrm{M}_5\mathrm{N}_7\)
2.292	42 Mo \(\mathrm{L}_3\mathrm{M}_5\)
2.310	16 S \(\mathrm{K}\mathrm{L}_3\)
2.346	82 Pb \(\mathrm{M}_5\mathrm{N}_7\)
2.423	43 Tc \(\mathrm{L}_3\mathrm{M}_5\)
2.423	83 Bi \(\mathrm{M}_5\mathrm{N}_7\)
2.502	84 Po \(\mathrm{M}_5\mathrm{N}_7\)
2.558	44 Ru \(\mathrm{L}_3\mathrm{M}_5\)
2.582	85 At \(\mathrm{M}_5\mathrm{N}_7\)
2.622	17 Cl \(\mathrm{K}\mathrm{L}_3\)
2.654	86 Rn \(\mathrm{M}_5\mathrm{N}_7\)
2.697	45 Rh \(\mathrm{L}_3\mathrm{M}_5\)
2.732	87 Fr \(\mathrm{M}_5\mathrm{N}_7\)
2.806	88 Ra \(\mathrm{M}_5\mathrm{N}_7\)
2.838	46 Pd \(\mathrm{L}_3\mathrm{M}_5\)
2.900	89 Ac \(\mathrm{M}_5\mathrm{N}_7\)
2.958	18 Ar \(\mathrm{K}\mathrm{L}_3\)
2.983	47 Ag \(\mathrm{L}_3\mathrm{M}_5\)
2.996	90 Th \(\mathrm{M}_5\mathrm{N}_7\)
3.083	91 Pa \(\mathrm{M}_5\mathrm{N}_7\)
3.133	48 Cd \(\mathrm{L}_3\mathrm{M}_5\)
3.171	92 U \(\mathrm{M}_5\mathrm{N}_7\)
3.250	93 Np \(\mathrm{M}_5\mathrm{N}_7\)
3.286	49 In \(\mathrm{L}_3\mathrm{M}_5\)
3.314	19 K \(\mathrm{K}\mathrm{L}_3\)
3.339	94 Pu \(\mathrm{M}_5\mathrm{N}_7\)
3.429	95 Am \(\mathrm{M}_5\mathrm{N}_7\)
3.444	50 Sn \(\mathrm{L}_3\mathrm{M}_5\)
3.525	96 Cm \(\mathrm{M}_5\mathrm{N}_7\)
3.604	51 Sb \(\mathrm{L}_3\mathrm{M}_5\)
3.616	97 Bk \(\mathrm{M}_5\mathrm{N}_7\)
3.678	52 Te \(\mathrm{L}_3\mathrm{M}_5\)
3.692	20 Ca \(\mathrm{K}\mathrm{L}_3\)
3.709	98 Cf \(\mathrm{M}_5\mathrm{N}_7\)
3.938	53 I \(\mathrm{L}_3\mathrm{M}_5\)
4.093	21 Sc \(\mathrm{K}\mathrm{L}_3\)
4.110	54 Xe \(\mathrm{L}_3\mathrm{M}_5\)
4.285	55 Cs \(\mathrm{L}_3\mathrm{M}_5\)
4.467	56 Ba \(\mathrm{L}_3\mathrm{M}_5\)
4.512	22 Ti \(\mathrm{K}\mathrm{L}_3\)
4.532	61 Pm \(\mathrm{L}_3\mathrm{M}_5\)
4.647	57 La \(\mathrm{L}_3\mathrm{M}_5\)
4.839	58 Ce \(\mathrm{L}_3\mathrm{M}_5\)
4.953	23 V \(\mathrm{K}\mathrm{L}_3\)
5.035	59 Pr \(\mathrm{L}_3\mathrm{M}_5\)
5.228	60 Nd \(\mathrm{L}_3\mathrm{M}_5\)
5.415	24 Cr \(\mathrm{K}\mathrm{L}_3\)
5.633	62 Sm \(\mathrm{L}_3\mathrm{M}_5\)
5.850	63 Eu \(\mathrm{L}_3\mathrm{M}_5\)
5.900	25 Mn \(\mathrm{K}\mathrm{L}_3\)
6.053	64 Gd \(\mathrm{L}_3\mathrm{M}_5\)
6.273	65 Tb \(\mathrm{L}_3\mathrm{M}_5\)
6.405	26 Fe \(\mathrm{K}\mathrm{L}_3\)
6.498	66 Dy \(\mathrm{L}_3\mathrm{M}_5\)
6.720	67 Ho \(\mathrm{L}_3\mathrm{M}_5\)
6.931	27 Co \(\mathrm{K}\mathrm{L}_3\)
6.949	68 Er \(\mathrm{L}_3\mathrm{M}_5\)
7.180	69 Tm \(\mathrm{L}_3\mathrm{M}_5\)
7.416	70 Yb \(\mathrm{L}_3\mathrm{M}_5\)
7.480	28 Ni \(\mathrm{K}\mathrm{L}_3\)
7.655	71 Lu \(\mathrm{L}_3\mathrm{M}_5\)
7.899	72 Hf \(\mathrm{L}_3\mathrm{M}_5\)
8.046	29 Cu \(\mathrm{K}\mathrm{L}_3\)
8.146	73 Ta \(\mathrm{L}_3\mathrm{M}_5\)
8.398	74 W \(\mathrm{L}_3\mathrm{M}_5\)
8.637	30 Zn \(\mathrm{K}\mathrm{L}_3\)
8.652	75 Re \(\mathrm{L}_3\mathrm{M}_5\)
8.911	76 Os \(\mathrm{L}_3\mathrm{M}_5\)
9.175	77 Ir \(\mathrm{L}_3\mathrm{M}_5\)
9.251	31 Ga \(\mathrm{K}\mathrm{L}_3\)
9.442	78 Pt \(\mathrm{L}_3\mathrm{M}_5\)
9.713	79 Au \(\mathrm{L}_3\mathrm{M}_5\)
9.886	32 Ge \(\mathrm{K}\mathrm{L}_3\)
9.989	80 Hg \(\mathrm{L}_3\mathrm{M}_5\)
10.269	81 Tl \(\mathrm{L}_3\mathrm{M}_5\)
10.543	33 As \(\mathrm{K}\mathrm{L}_3\)
10.551	82 Pb \(\mathrm{L}_3\mathrm{M}_5\)
10.839	83 Bi \(\mathrm{L}_3\mathrm{M}_5\)
11.131	84 Po \(\mathrm{L}_3\mathrm{M}_5\)
11.224	34 Se \(\mathrm{K}\mathrm{L}_3\)
11.427	85 At \(\mathrm{L}_3\mathrm{M}_5\)
11.727	86 Rn \(\mathrm{L}_3\mathrm{M}_5\)
11.924	35 Br \(\mathrm{K}\mathrm{L}_3\)
12.031	87 Fr \(\mathrm{L}_3\mathrm{M}_5\)
12.339	88 Ra \(\mathrm{L}_3\mathrm{M}_5\)
12.648	36 Kr \(\mathrm{K}\mathrm{L}_3\)
12.652	89 Ac \(\mathrm{L}_3\mathrm{M}_5\)
12.968	90 Th \(\mathrm{L}_3\mathrm{M}_5\)
13.291	91 Pa \(\mathrm{L}_3\mathrm{M}_5\)
13.396	37 Rb \(\mathrm{K}\mathrm{L}_3\)
13.614	92 U \(\mathrm{L}_3\mathrm{M}_5\)
13.946	93 Np \(\mathrm{L}_3\mathrm{M}_5\)
14.165	38 Sr \(\mathrm{K}\mathrm{L}_3\)
14.282	94 Pu \(\mathrm{L}_3\mathrm{M}_5\)
14.616	95 Am \(\mathrm{L}_3\mathrm{M}_5\)
14.954	96 Cm \(\mathrm{L}_3\mathrm{M}_5\)
14.958	39 Y \(\mathrm{K}\mathrm{L}_3\)
15.307	97 Bk \(\mathrm{L}_3\mathrm{M}_5\)
15.655	98 Cf \(\mathrm{L}_3\mathrm{M}_5\)
15.775	40 Zr \(\mathrm{K}\mathrm{L}_3\)
16.036	99 Es \(\mathrm{L}_3\mathrm{M}_5\)
16.402	100 Fm \(\mathrm{L}_3\mathrm{M}_5\)
16.615	41 Nb \(\mathrm{K}\mathrm{L}_3\)
16.768	101 Md \(\mathrm{L}_3\mathrm{M}_5\)
17.139	102 No \(\mathrm{L}_3\mathrm{M}_5\)
17.480	42 Mo \(\mathrm{K}\mathrm{L}_3\)
17.500	103 Lr \(\mathrm{L}_3\mathrm{M}_5\)
18.367	43 Tc \(\mathrm{K}\mathrm{L}_3\)
19.279	44 Ru \(\mathrm{K}\mathrm{L}_3\)
20.216	45 Rh \(\mathrm{K}\mathrm{L}_3\)
21.177	46 Pd \(\mathrm{K}\mathrm{L}_3\)
22.163	47 Ag \(\mathrm{K}\mathrm{L}_3\)
23.173	48 Cd \(\mathrm{K}\mathrm{L}_3\)
24.210	49 In \(\mathrm{K}\mathrm{L}_3\)
25.271	50 Sn \(\mathrm{K}\mathrm{L}_3\)
26.359	51 Sb \(\mathrm{K}\mathrm{L}_3\)
27.473	52 Te \(\mathrm{K}\mathrm{L}_3\)
28.612	53 I \(\mathrm{K}\mathrm{L}_3\)
29.775	54 Xe \(\mathrm{K}\mathrm{L}_3\)
30.973	55 Cs \(\mathrm{K}\mathrm{L}_3\)
32.194	56 Ba \(\mathrm{K}\mathrm{L}_3\)
33.442	57 La \(\mathrm{K}\mathrm{L}_3\)
34.720	58 Ce \(\mathrm{K}\mathrm{L}_3\)
36.027	59 Pr \(\mathrm{K}\mathrm{L}_3\)
37.361	60 Nd \(\mathrm{K}\mathrm{L}_3\)
38.725	61 Pm \(\mathrm{K}\mathrm{L}_3\)
40.118	62 Sm \(\mathrm{K}\mathrm{L}_3\)
41.542	63 Eu \(\mathrm{K}\mathrm{L}_3\)
42.996	64 Gd \(\mathrm{K}\mathrm{L}_3\)
44.482	65 Tb \(\mathrm{K}\mathrm{L}_3\)
45.999	66 Dy \(\mathrm{K}\mathrm{L}_3\)
47.547	67 Ho \(\mathrm{K}\mathrm{L}_3\)
49.128	68 Er \(\mathrm{K}\mathrm{L}_3\)
50.742	69 Tm \(\mathrm{K}\mathrm{L}_3\)
52.388	70 Yb \(\mathrm{K}\mathrm{L}_3\)
54.070	71 Lu \(\mathrm{K}\mathrm{L}_3\)
55.790	72 Hf \(\mathrm{K}\mathrm{L}_3\)
57.535	73 Ta \(\mathrm{K}\mathrm{L}_3\)
59.318	74 W \(\mathrm{K}\mathrm{L}_3\)
61.141	75 Re \(\mathrm{K}\mathrm{L}_3\)
63.000	76 Os \(\mathrm{K}\mathrm{L}_3\)
64.896	77 Ir \(\mathrm{K}\mathrm{L}_3\)
66.831	78 Pt \(\mathrm{K}\mathrm{L}_3\)
68.806	79 Au \(\mathrm{K}\mathrm{L}_3\)
70.818	80 Hg \(\mathrm{K}\mathrm{L}_3\)
72.872	81 Tl \(\mathrm{K}\mathrm{L}_3\)
74.970	82 Pb \(\mathrm{K}\mathrm{L}_3\)
77.107	83 Bi \(\mathrm{K}\mathrm{L}_3\)
79.291	84 Po \(\mathrm{K}\mathrm{L}_3\)
81.516	85 At \(\mathrm{K}\mathrm{L}_3\)
83.785	86 Rn \(\mathrm{K}\mathrm{L}_3\)
86.106	87 Fr \(\mathrm{K}\mathrm{L}_3\)
88.478	88 Ra \(\mathrm{K}\mathrm{L}_3\)
90.884	89 Ac \(\mathrm{K}\mathrm{L}_3\)
93.351	90 Th \(\mathrm{K}\mathrm{L}_3\)
95.868	91 Pa \(\mathrm{K}\mathrm{L}_3\)
98.440	92 U \(\mathrm{K}\mathrm{L}_3\)
101.059	93 Np \(\mathrm{K}\mathrm{L}_3\)
103.734	94 Pu \(\mathrm{K}\mathrm{L}_3\)
105.970	95 Am \(\mathrm{K}\mathrm{L}_3\)
108.759	96 Cm \(\mathrm{K}\mathrm{L}_3\)
111.698	97 Bk \(\mathrm{K}\mathrm{L}_3\)
114.801	98 Cf \(\mathrm{K}\mathrm{L}_3\)
119.080	99 Es \(\mathrm{K}\mathrm{L}_3\)
122.190	100 Fm \(\mathrm{K}\mathrm{L}_3\)
125.390	101 Md \(\mathrm{K}\mathrm{L}_3\)
128.660	102 No \(\mathrm{K}\mathrm{L}_3\)
132.020	103 Lr \(\mathrm{K}\mathrm{L}_3\)

solarchemist.se

Contact

E-mail ✉️

On the Fediverse

Signal chat

Matrix chat

Other services

Taha’s PhD thesis

Nanostructured ZnO and metal chalcogenide filmsfor solar photocatalysis

Source code and reproducibility

Links and notes

Other PhD theses with source code

Other thesis templates

The solar origins of spectroscopy

The beginnings of the science of spectroscopy

Sunlight above and below the atmosphere

Notes and links

Session info

References

About

Notes on Kelvin probe microscopy

History of the Kelvin probe technique

The contact potential difference (CPD)

The work function

Principle of operation of a Kelvin probe

Kelvin probe force microscopy (KPFM)

Notes and links

References

Observing the global energy transition (from over here)

Solar power plants

Electrolysers

Fuel cell vehicles

Photoelectrochemical plants

Other renewables

Manufacturing of solar PV, electrolysers, wind turbines, batteries, and heat pumps

And perhaps not so good news

Background

A short primer on the greenhouse effect and climate change

Solar energy collection and conversion

Atmospheric carbon capture: inefficient, but may be unavoidable

Notes and assorted links

References

Spectrochemical data of the dyes EBT, MB, MO and RB5

Spectral absorption coefficients

Experimental notes

Preparation of dye stock solutions

Mixing alcohol into the aqueous dye solutions

Configuration of the light-source and spectrometer

Small study into the pH and conductivity of aqueous dye solutions

Links and notes

Session info

References

Sources and license

Using Python and RSS to play Youtube channels on the desktop

RSS is awesome!

ttrss-video-mpv

Ideas for future development?

Notes and links

Use RSS to follow new content

Self-host your RSS reader

Watch youtube using non-google clients (e.g., RSS readers)

Watch youtube videos via RSS in mpv or youtube-dl

What is RSS

Credits

Creating a composite gif with multiple gganimate panels

The original, static, multi-panel plot

Animated plot with multiple overlaid panels

Encountered issues and how to handle them

Further reading

gganimate

Creating animations with gganimate

Composite animations with gganimate

Composite animations in R without using gganimate

sessionInfo()

Measurement errors and dimensional analysis in R

Install WSL2 on Windows 10

Parenthesis: if your Windows 10 is itself a virtual machine

Parenthesis: managing and rebooting your WSL2 virtual machines

Install R and RStudio server on WSL2 Ubuntu

Working with quantities and measurement errors

Nanostructured ZnO and metal chalcogenide films
for solar photocatalysis

Wrapper function redefining `lm()` to support units/errors