Thursday, September 4, 2025

Tools of Excellence: Cb2bib, Yazi, Sioyek, Entangle, XnViewMP, Emacs,...: WOW!

In the summer of either 1982 or 1983, I enrolled in a summer school class at UCSB offered by the College of Engineering on Computer Architecture.  At about this time two of my regular classes required the use of Wordstar, on a mainframe terminal.  I had been required to write a program in Fortran for a math course; now, in this Computer Architecture class, I wrote some minimal code in PDP-1170 Assembly Language.   Intel's early microprocessors---the engines that  would spark a revolution in Personal Computing---had come into being.   I had enrolled on a Pass/Fail basis, fearing that it would be a technical challenge.  I had enrolled later in an amazing  Invertebrate  Paleontology course, also Pass/Fail.  Both times, I would have earned an honest "A."         Yet another Lesson learned.  

 The theory behind digital computers, as it turns out, is straightforward and fascinating, logical and understandable.  I later came to realize that these systems, invented  by humans, reflected human logic; they are non-complex, a fact I understood more fully after being introduced to Molecular Biology and Genetics, and the extreme complexity of the Central Dogma  By the end of the short summer session, I had a tentative grasp of the nature of digital computers.  The graduate student instructor took me aside and advised that the knowledge I had gained in this brief introduction would be valuable in the future, and I would be able to advise others about what computers can do.  

 This advice could not have been more applicable when I moved to Chuuk Lagoon, and started a project collecting animal names in local dialects of the islands.  At the University of Guam Marine Lab, I was exposed to early Personal Computers, the IBM XT and IBM AT.  My basic knowledge of the workings of computing systems provided useful perspective for solving problems.  I eschewed the commercial/proprietary solutions, the high priced computer programs that would make the entrepreneurs wealthy.  

 I learned what I could and scrounged software of many kinds.  Except the big brands.  There are always other solutions that work better; those behemoths aren't worth the cost to society.  I believed this then, and I believe it even more today.  

 My Toolbox

Previously, I have written a little about Yazi and Sioyek.   They, like Cb2Bib and Entangle, are functionally remarkable tools.   Among Free Software users, they are not the most popular.  I also rely upon the highly functional, low overhead foundation, the i3 tiling window manager.  Emacs is a mainstay; like these other tools, the simplicity and functional design of Emacs makes it run circles around most proprietary software.  Windows is dumbed down; the stress of using such a dysfunctional tool might be likened to trying to build a cabinet using a jackhammer; a poorly made jackhammer at that.   I use LibreOffice from time to time, and some other more mainstream tools.  They are fine; but the workhorses in my toolbox are more special. The packages I am writing about today are truly special, and enabling. 
 
I as initially attracted to Free Software for financial reasons.  The story is too long to recount here.  But as I have come to understand the GNU Project, and the grass-roots nature of the development of these tools, it became clear to me that the Free in Free Software actually refers to freedom, not free beer.   
 

Sioyek

The PDF reader Sioyek caught my attention relatively recently, with its plain exterior, focus on research, and its  hidden gems.  For a couple of days, I returned to sioyek, after setting it aside a while; a closer reading of the documentation has given me a new appreciation of it's wonders.  And today, after some digging, I was able to configure yazi to open files in Sioyek, or Evince, at will.  Gradually, the levers and buttons of sioyek are revealed.
 
The documentation is relatively straightforward.   But I struggled with configuration.  Later for that.  PrintHere are some of the more interesting features:
  • A keystroke sends the text at the cursor to Google Scholar (GS), or another tool I am unfamilar with.  A browser opens, Google Chrome.  Keystrokes are available, either standard or by a plugin, to download the pdf from GS.  
  • Sioyek remembers bookmarks of two or three kinds.  A keystroke causes display of a list of bookmarks in either the current file, or all pdfs sioyek knows about.  
  • One of these "bookmark" functionalites is called "highlighting."  Selected text may be highlighted.  The keystroke for highlighting must be followed by a single key, which represents the type of material.  So someone like me, whose mind travels in so many directions, can keep track of various threads of interest.  If several files are highlighted in this way, retrieval of highlights can optionally display all highlights of a given type in all files (globally).  
  • The Global retrieval feature holds for bookmarks as well.  So if I am reading four PDFs that relate to each other, I can bounce around.  And selections can be copied to the system clipboard.  You know where this is going.
  • Portals are something I don't yet understand.  I position may be set as a target for a portal, so as I'm reading I can refer back to a certain specific piece of text, diagram, table, whatever.  Backspace brings me home.
  •  I was able to print.
  • Sioyek is mainly keyboard driven; but scrolling with a mouse is possible.
  • A "visual mark" (or whatever it's called) underlines current text if that is wanted; and this mark can be manipulated in various ways.  More to learn hear.
  • Zooming is great.
  • In PDFs text can be selected and various functions are possible.  A click on a figure number will take me directly to that figure.  This built-in feature seems almost intelligent, and it reminds me of snarfing references in Cb2Bib, a remarkable feature.  
  • Synctex mode allows the latex source to be opened and followed when reading a pdf.  Wow.

Some thought went into this tool.  

 

Yazi 

Yazi is a console based file manager with superpowers.  I have gotten into the configuration details far enough to make it possible to choose which pdf viewer is desired.  Sometimes, I think sioyek may be too far out, for some things; so I set up to be able to open in either  evince or sioyek.  Shift-O opens a minimal menu; included here is a button to open it in a folder.  I think some more tweaking is needed to identify low-overhead tools like qutebrowser for Google Scholar searches, and a minimal Graphical File Manager with lower overhead.  My laptop lags seriously, but it's a very old laptop.
 
Yazi works swell with kitty.  Previews of jpegs and pngs, and possibly tiffs (with some tweaking) is extremely fast, and a plugin "zoom" feature enables the image to zoom to the full height of the window.  Image and PDF previews are useful.  
 
Bunny is an amazing bookmarking tool.  
 
A tweak in .bashrc (or .zshrc) makes it possible to open yazi by typing 'y'.  As a bonus feature, when one changes directory, then types "q" to quit, yazi leaves the console in the last folder visited.  
 
This tool is undergoing rapid development, and plugins have been known to fail after updating.  
 
I use this for editing filenames, with tags.  A filter function ("f") then makes possible isolating files with a specific tag, marking them all, creating a directory, and dumping all these into a directory.   
 
Cb2Bib
 
This is an extremely useful tool, to me.  I think I'm the only one who uses it in the way I do, to write annotations to records in my bibtex database, and print out annotated bibliographies on demand.  I write, for example, a call number of a book, and notes on pages, compile it into LaTeX, and print.  Very convenient.  The developer was kind enough to explain how to modify the code to incorporate an "annotate" field in the bibtex entry, and also modify the tools that generate a pdf, so an annotation is formated together with the citation.  
 
In this space, I hope to present instructions for the benefit of others.  

Cb2Bib also automatically picks up citations that are highlighted in other programs.  Also, one can copy a bibtex citation from Google Scholar, that is instantly picked up and incorporated into the database associated with the program.  

This program does much more than I am able to understand, but what I can understand, and do---with the help of Pere Constans---is amazing.  
 
This is another reason I don't stick to the commercial / proprietary software packages.
 
Entangle
 
More to follow. 
 
 


Saturday, July 12, 2025

sioyek and yazi

I have set Sioyek as the default pdf viewer for Yazi; actually, probably for the entire system.  

 It was *relatively* simple to do this, after installing xdg-utils-mimeo.  I will not go back through that here, but it's not hard.  

 Sioyek has no native capacity to print.  This is a potential show stopper.  I have attempted to set this up from instructions from some random internet post.  The location of the user config files is in doubt, however.  By "reports," config files should be found in ~/.config/sioyek; however, other reports indicate .local/share/sioyek.  I did install a rather decrepid tool, a gtk-print tool (name?).  My config does not work yet.


Another potential downside relates to the use case for Sioyek, as a pdf annotation tool for research.  When a file is opened for the first time, sioyek indexes the file; this makes searches incredibly quick and useful.  Will this result in cluttering the system after a large number of pdfs are opened?  Probably another lightweight pdf viewer should be established as the default, to streamline the process of reviewing random files on the fly.  

Sioyek is uncluttered, as far as GUI.  There hardly is any GUI.  A fairly extensive set of utilities is tied to keystrokes.  Some of them are not exactly intuitive, at least to me.  


Some facilities:

  • Bookmarking
  • Search in google scholar (may not work right)
  • gg -> go to page 1
  • G -> go to last page
  • rt click -> mark line (not sure this is working with a track pad)
  • highlighting in various colors
  • search highlights
  • "Smart Jumps" to figures or references, with or without links: this didn't work out of the box, for me.
  • mark a location, give character as a name for the mark.

 

Yazi is turning out to be extremely useful.  I use it multiple times a day.  Some useful tweaks:

  • set a much larger preview size, then
  • install a preview to Toggle full view of the preview
  • somehow a special tweak was needed to view tiffs (this wasn't the case when I first tried yazi)
  • bookmarks.  yabm (Yet Another Book Mark) is great.  Another one was incompatible, AFAICT.
  • Zoom plugin
  • In a tips section of the web site, is recommended to install some code in the rc file of the shell, which allows to start with "y" and that leaves the terminal in the last used directory, when quit with "q".  Extremely useful.

A problem: not really a big one: yazi is undergoing rapid and intense development, so incomptibilities arise frequently.   Especially with plugins.

 

 


Some recent useful software additions

 A brief list for now:

  • dysk : concise "df" and graphic
  • sioyek: interesting  document reader.  Keyboard driven, mostly, maybe. Designed for academic papers and textbooks.  
  •  bpytop: A feature-ful process/memory/cpu monitor, text based; similar to top or htop.
  •  

Monday, May 19, 2025

Notes on Yazi

 Yazi is good.  Real good.  I used it over ssh to cull a directory on my second system; it worked as I had hoped, except, perhaps, for color theming.

 Knock on Wood.

 Viewing/reviewing

One of the needs that yazi has fulfilled for me is rapid viewing.  I can only imagine that should I ever be able to upgrade my current machines (RAM and CPU, as well as Storage), viewing would be even more friendly, especially for images.  The ability to flip through a directory of images, mark, and move, copy, or delete marked files, has made a huge difference to my workflow.  The only other tool that is useful in this regard is xnviewmp.  I use a fairly simple set of commands; probably if I mastered tabs and other complexities and plugins, yazi would be even more helpful.

 

By the way, the terminal "kitty" has wonderful facilities built in to work well for viewing images.   I prefer the black background and contrasty color scheme to any other.

 Ranger was good enough, until it wasn't.

Copy, Paste and move and Notes on plugins.

 [Cautionary note: I have lost files through carelessness here, and yazi has some gotchas to be navigated around.]

This is pretty much self-explanatory.  I will try to address this topic in detail at some future time.  For now, I'll mention a plugin that has helped speed these processes: 

  • yamb is a bookmarking plugin that works well.  In the absence of a much needed multi-step backup command, this does a good job when sorting through files and moving or copying them to another directory.  I have gotten around the mentioned limitation by defining  bookmarks for the current from directory, and the currently interesting goto directory,  I define a set of book marks to frequently used directories, as well.  This is an easy tool to master.  Like other plugins, issues have been experienced after updating yazi (it is under fairly intense development).  
  • toggle-pane: This helps when viewing both pdfs and images.  It is possible to configure the mazimum size of displayed images (I'm not certain whether this works for pdfs) by putting this code into ~?.config/yazi/yazi.toml:

                       [preview]

                      # Change them to your desired values
                      max_width  = 1800
                      max_height = 1800
  

       Plugins are found on a resources link on yazi's web pages.  Other sites exist with plubins.

 

Filtering

Filtering is a wonderful way to sort.  It has been even more useful to me, because of a filenaming scheme I have adopted.  I stole this idea from Protesilaos's Denote system.  I have not found denote very useful, but the concept of tags at the end of the filename, before the extension has become a useful tool allowing me to sort files more rapidly.  Here is an example.

    Jones--2060--JargonInBiology__taxonomy_nomenclature_zoology.pdf

I have started avoiding underscores in filenames ("_") in favor of using them in tags.  Interestingly, I saw a web page or thread somewhere about tagging styles, and this is one of several.  I used to use a 10 (or so) character at the beginning of lines in a bibliography, where each position had a meaning.  So anything related to crabs would have "c" in the third position (12ctx8urt-Anon-Crabs of the world) would be sorteable by sorting by the third position.  The Unix (GNU/Linux) sort command worked excellently.  

 With filtering  in yazi, one can narrow down a large folder/directory with pdfs about taxonomy: "_taxonom".  Prot has demonstrated the use of regular expressions to sort even more specifically in, I think emacs dired.  To avoid filtering for "_" in filenames, one would theoretically use a regular expression requiring a "__" (double underscore) somewhere before the "_:" single underscore.  This is probably easy.  Filtering is easier, I think.  It may be that emacs would be better, but so far, aside from a couple of learning experiences, Yazi is working well.

Moving around 

In either the getting started or tips section of the yazi main web pages, is a suggestion that makes yazi more useful.  

Inserting this code into .bashrc (and I think .zshrc as well) makes things easy.

 function y() {
    local tmp="$(mktemp -t "yazi-cwd.XXXXXX")" cwd
    yazi "$@" --cwd-file="$tmp"
    if cwd="$(command cat -- "$tmp")" && [ -n "$cwd" ] && [ "$cwd" != "$PWD" ]; then
        builtin cd -- "$cwd"
    fi
    rm -f -- "$tmp"
}

Then, when in a terminal, type "y" to start yazi.  After moving to another directly, quitting with "q" leaves the terminal in this new directory.

 

A Caution or Two 

 Plugins are often not mutually compatible.  I have tried several others, but some collided with these few i do use.  

Some plugins iexplicably use keybindings that are assigned to other functions by default, and it is not always apparently what key bindings may be used.  <F1> may display a help screen, which may be filtered.  Filtering rocks.  Bookmarks rock.

 

2025 May 19

 

 

             

Monday, February 10, 2025

Unix's and GNU/Linux's Unknown Superpowers: ptx, sort, and grep

The Unix operating system is wonderful.  GNU/Linux is wonderful.  Let me make this distinction, before discussing the aforementioned three tools: GNU is Not Unix. This is a story of how capitalism limits the human spirit, in order to extract financial gain.  

 Before writing about the remarkable tools provided by Unix---and now, the GNU computing system (one of the scions of which is GNU/Linux)---one may be forgiven for visiting certain concerns, including the generation of incomprehensible wealth by a few men who now seek to control our lives in ever more of their detail.  I suggest that this wealth was made primarily through marketing, on the backs of the ingenuity of other men, some of whom were out-maneuvered and left behind by certain well known business men.  

It came about, during the early days of computing, long ago: a gifted programmer by the name of Richard Stallman worked at MIT Artificial Intelligence Laboratory (MITAIL), along with other highly capable computer programmers.   Events during that period led to unforeseeable changes, as a behemoth technical-industrial complex grew beyond all imagination, based on deft slight-of-hand and shrewd business moves, as entrepreneurs snatched up computer code, patented and copyrighted it, and concealed it from hapless consumers.  Shrewd manipulations of programmers and code enabled a few men, in less than 50 years, to accumulate over half of the wealth of the financial world.  

This they were able to do by convincing consumers not to share, with the complicity of the legal "intellectual property" infrastructure; to violate the golden rule that many had cherished since childhood.  Make many copies of a useful computer program for a few pennies each, and set prices  arbitrarily high.  The consumer will be so mesmerized by the utility and flashiness of these programs that she will believe you are doing her a favor, selling it so cheap.  Convince governments to sign lucrative contracts.  Update products at speed, and charge for the updates.  Sue public schools when they did not update to newer licenses.  the profits would be almost unlimited.  And, now, change the business model and lease software, so leverage even greater control, and much greater profits. Never do unto others anything without transactional benefits to self.  Above all, do not share a computer program, even though they are useful.  This was a massive wealth grab. 

Tim Wu has written about the  entrepreneurs who built incomprehensible wealth by these same strategies: seize control of the media channels, prevent newcomers by various and nefarious means, and charge arbitrarily high rates for the use of this now valuable commodity:  information.    Radio. Telegraph, Telephone, and Television have been used to manipulate the public, by employing choke points---selective filters---to manipulate elections, gain power, and sell snake oil.  The political landscape has been altered in the interest of gaining more power, prestige, wealth, and control.  It's OK, these persons believe, to lie and deceive.  Today's politicians have gained success by following a model of radical "gas lighting," and in doing so have built empires  of corruption and greed.  

You may know this story. 

 Richard Stallman has been painted under the dark lights of deception as an extremist, even among other adherents to Free Software.  This is how the insidious tactics of industry and politics can influence our opinions to their own benefits.  The Petroleum Industry, in a similar manner, convinces us that global climate change is a hoax, that CO2 is fine.  Our politicians convince us, who are eager to consume their distorted and deceptive messaging, of these things, and many others.  We know they are lying, but the steam engines of their corruption are traveling down the tracks with such unrelenting force and momentum that they seemingly have taken over our own minds and hearts.  

 This is about one little part of the story. 

The MITAIL was at the forefront, in those beginning days of computing.  Richard Stallman was there; he noticed that the software interface of their very expensive printer was inefficient, klunky.  In the hopes of improving this interface, he kindly offered to the company that owned the patents,  to tweak the software and make it work better.  The manufacturer, however, was unwilling to open up the code behind that software, fearing loss of control and profit if it ware revealed.  

This is an important story, but this is not our story today.  Suffice it to say that Richard Stallman decided to do an end-around, in the finest traditions of human cooperation and sharing.  The Unix Operating System, owned by Bell Laboratories (AT&T), was an outstanding work of programming, a thoughtfully produced system of software, an interface of great utility, with numerous subprograms attached to a central kernel, making a whole that by far exceeded the sum of its parts.  But AT& T owned Unix. In the capitalistic world, this was a fantastic money maker.  But the cost of ownership was onerous, making it impossible for the average computer user to purchase.  And as Stallman had already learned, the impediments that proprietary software posed to even the best programmers presented various difficulties---even beyond simple ownership.  The experience with a printer had taught that lesson.  


And what about sharing these wonderful tools?  Stallman recalled an earlier time when the many talented programmers and users at the MITAIL had freely shared programs they had developed.  As a consequence of this open access to the universe of such software programs, even greater programs were spawned from the synergism...  But now many of those programmers had been enticed to leave this world by unimaginably lucrative contracts.  But  these contracts had been concealed within a trojan horse: the non-disclosure agreement.  The code, they promised, they would not share.  And even the end users, who purchased such programs today as Photoshop, Lightroom, games, typing tutors, graphics printing programs....  ad infinitum, are coerced into not sharing the software they purchase.  The industry is serious about this limitation.

So Richard Stallman reached out on the internet to propose a collaboration of programmers to build a new computer system, a free system, that could be "shared without dishonor."  Remember how Unix comprises a set of software units that are hook in to a kernel?  Those units include various commands: copy and move files, for example.  Even typing on a keyboard is handled by tools hooked into the kernel.  The construct, from a plethora of these units, is an Operating System (OS).  Such operating systems as the proprietary MS-DOS, Windows, and MacOS are example of proprietary OSes.  It is forbidden by the proprietors of these OSes to share the programs.  At great expense, these companies have constructed elaborate schemes to make it difficult or (they hope) impossible to share these programs.  

 Do you see where we are going with this?  This massive construct, including now Microsoft, Facebook, Apple, Google, Oracle, Adobe, and others, has used a strategy of concealment, of forcing others to not share, to build huge empires, and horde wealth.  Intimidating the consumer, for example by suing Portland, Oregon's public schools to force purchase of contracts for every copy of windows that had not been updated, as "required" by the manufacturer.  

But GNU is Not Unix.  Richard Stallman recognized Unix as a wonderful model for an OS.  If thousands of programmers all over the world, connected by the Internet,  were to each write one or a small number of   utility programs, mirroring those that perform the functions  of  the Unix programs, while avoiding even any appearance of copying the Unix code, a new operating system could be built up through community action, a citizen-built computing system.  This system was successfully built.  Even a protracted legal challenge by the owners of Unix, SCO, many years later, could not find any copied code in the GNU/Linux operating system.

When I was isolated in Chuuk Lagoon, I had been conducting research that required access to computers.  My family donated a laptop computer, but the Windows OS was clumsy, and, to cut to the chase, could not help me with my project.  An editor that had been sent to me by a linguist at the University of Hawaii was capable, but it was "cripple ware," in a sense, and without the manual---which sold separately for hundreds of dollars--I could not proceed.

 

 The Free Software Foundation (FSF) responded to my request for an editor, sending a version of the Emacs editor ported to Windows, along with 14 disks of Unix tools, also that would run on Windows.  This was just what I needed.  Among these tools were a number of Unix text tools, opening a window into a new world of possibilities.  Very important to me were sort, grep, and ptx.  These, as far as I am concerned, are core Unix (hence Linux) superpowers. 

sort

Windows had a sort utility, but it was extremely limited.  In fact, the more I learned of the tools granted to me by the FSF, the better I understood the fraud that had been perpetrated on the public, corporate world, and governments by Microsoft.  I can liken the difference between the facilities provided by Unix (and later by Linux and FreeBSD) is comparable to the difference between a set of cheap carpenter's tools, and professional quality tools.  GNU sort is the Cadillac of sort programs, the Rolls.  

Later, on Guam, I was able to download a copy of the new Linux OS on Guam.  It was then that I discovered liberation.  From that day, in the early 1990s, until today, in 2025, I never never felt comfortable on a Windows computer.  At that time, one of the biggest differences was that Linux was capable of doing true multitasking, while Windows only could do one operation at a time.  With GNU/Linux I could type a letter while printing a different document, and running another process in the background.  Another difference was the Linux did not crash.  Let me repeat, it never crashed or locked up.  Windows regularly did so.  

grep
 
 I will not explain grep.  Today, in the 2020s, one can run grep on Windows machines.  Suffice to say that grep is a tool that enables searching through a set of files for a search term, and printing to the screen a list of matches. With pipes (signified by the character "|", one can direct the output of grep to another program, like sort.  Then, from sort, one may direct its output to less, a better file browsing tool than Unix's more, a tool that pages through a file.  To do this one would type:
 
    $ grep <filename.txt> | sort | less

 

ptx

 My need was specialized, as I was dealing with lexical text, lists of animal names.  To weed through these, I found ptx was incredible.  Ptx means "permutated index."  What follows is a small segment of output from a search of a text file about flowers and seasons of fishing. 


  /Luma, altitude 100-125 meters,   flower, August 4, 1921, Garber no./
      /\section{Burrows: a different   flower, and palolo.\hfill{}\textsc/
     specifically, he does mention a   flower.
     with certain things. There is a    flower called bust ginger;/    /time
      \textit{Erythrina\/} sp. as an      indicator species.            /about
        /constellation as a seasonal     indicator, he refers to this/
   a resident of Wonipw,/       This    indicator would only be useful to
   the/    /solar alignment as a key   indicator of the   emergence of
      /\hfill{}\textsc{whale:season:    indicator}} \label{sec:org5e260c7}/
     /tree fruits are ripe, it is an        indicator that whales are running/
Samoa/    /was used as a seasonal  indicator on some islands (e.g.,
  /2015, Trees of Yap:  sometimes   indicator of seasons} \label{sec:/
   . \textbf{Sometimes used as an   indicator of season.}       /flowers
   a resident of Wonipw,/       This   indicator would only be useful to
    /variegata\/} flowers in the dry   season, and has been mentioned (/
      and that it flowers during dry   season.       /of year of flowering,
      /these two periods the iéboua   season; the \%\%    season of the/
       /the iéboua season; the \%\%   season of the sun or heat iéboua-/
    /or heat iéboua-délat, and the   season of cold iéboua-tsiam.


tldr

 

Tldr (too long didn't read) is a modern innovation that summarizes the man page for a command.  In Unix, it was specified that who ever wrote a program must provide a "man": page, a manual to its use and options.  Here is output from tldr:
 
$ tldr ptx

  Generate a permuted index of words from text files.
  More information: <https://www.gnu.org/software/coreutils/manual/html_node/ptx-invocation.html>.

  Generate a permuted index where the first field of each line is an index reference:

      ptx --references path/to/file

  Generate a permuted index with automatically generated index references:

      ptx --auto-reference path/to/file

  Generate a permuted index with a fixed width:

      ptx --width=width_in_columns path/to/file

  Generate a permuted index with a list of filtered words:

      ptx --only-file=path/to/filter path/to/file

  Generate a permuted index with SYSV-style behaviors:

      ptx --traditional path/to/file 

How I got into Emacs (and Free Software)

This is a long tale, not confined to Emacs alone. This is, indeed, a story about why I am a confirmed Free Software advocate. This has been...