Welcome to Hexatomic!

You are reading the user documentation for Hexatomic.

What is Hexatomic?

Hexatomic is an extensible OS-independent platform for deep multi-layer linguistic corpus annotation.

Research projects want to answer very specific research questions. When they use corpora, they may have to use specific software that can

• handle the format their corpus data is available in, or
• provide annotation functionality for the annotation types they want to use.

Projects may also need additional software to search their data. If they have corpora in more than one format, they may have to duplicate the number of software tools they have to use in order to answer their research question. And if this software doesn't exist yet, they have to implement a new tool from scratch.

Hexatomic aims to alleviate this situation, and reduce the number of tools a project will have to use (and install, and maintain, and learn) to 1.

How does it do that?

1. It works with a generic graph-based data model, that can handle many different types of annotations.
2. It includes a converter framework, which allows the import of a multitude of different corpus formats, provides them in the generic graph-based model for manipulation (corpus-building, annotation, cleaning, etc.), and can export the corpus data to yet another multitude of corpus and other data formats.
3. It includes powerful corpus search functionality, that offers the usual free text- and regex-based search, but can also search across linguistic structures and build complex queries across layers.

What if Hexatomic cannot handle my specific annotation type/use case?

Hexatomic is built to be extensible through plugins. If it doesn't offer what you are looking for, you don't have to implement a new tool from scratch, but can instead build a plugin that does what you need. On top, the existing functionality, the data model, import and export functionality, and search come for free with that.

What kind of software is Hexatomic?

Hexatomic is software for the desktop. You download it to your computer. It does not need an internet connection to run, and therefore you can also use it in the field, or on the train en route to a conference.

Specifically, it is an Eclipse e4 application implemented in Java.

Hexatomic is free and open source under the Apache License, Version 2.0.

How can Hexatomic be used?

You can use Hexatomic to do, for example, any or all of the following:

Build a corpus from scratch

Merge different corpora into a new one

Annotate an existing corpus

Error correct a corpus

Search a corpus

Hexatomic installation & start

Download

1. Go to the download site for the latest release of Hexatomic:
   https://github.com/hexatomic/hexatomic/releases/latest.
2. Download the .zip file for your operating system:
   • Linux: Download hexatomic-<version>-linux.gtk.x86_64.zip
   • Mac OS: Download hexatomic-<version>-macosx.cocoa.x86_64.zip
   • Windows: Download hexatomic-<version>-win32.win32.x86_64.zip
3. Extract the downloaded zip file to a directory of your choice.¹

That's it. You can now run Hexatomic.

Run Hexatomic

1. Go to the folder into which you have unzipped the Hexatomic download.
2. Run Hexatomic by
   • double-clicking on the launcher file in a file manager, or
   • starting the launcher file from the command line.

• In Linux, the launcher file is simply called hexatomic and can be started with ./hexatomic.
• In Windows, the launcher file is simply called hexatomic.exe and can be started with hexatomic.
• For Mac OS X, we provide an .app file called hexatomic.app.

Usage

This section describes how to use the user interface of Hexatomic. See one of the following sub-sections for more details.

Working with projects

Hexatomic works on a single Salt project at any one time. A project consists of a directory containing a project file (saltProject.salt) and a number of sub-directories containing the Salt document files.

To open a project in Hexatomic, click on File in the main menu and select the option Open Salt Project. A new file dialog window will come up. Choose the folder containing the project file you want to open, and open it.

After opening a Salt project, you can select Save Salt Project, which will save the project as Salt XML files in the structure explained at the beginning of this section into the same folder that they have been loaded from. Alternatively, you can select Save Salt Project As... to select a different location to save the files to. You can see the currently loaded Salt project path in the Window title. If there are unsaved changes in any document or in the corpus structure, a * will be appended to the title.

Saving the project will always save all documents, not only the openend ones. If you close an editor, the changes are not lost as long as you save the project afterwards.

To start a new and empty project in the same application window, choose Start New Salt Project.

If an action would discard unsaved changes, you are always given the choice to cancel it.

Editing the corpus structure

Each corpus project can consist of multiple documents which are organized into

• corpus graphs,
• corpora, and
• sub-corpora.

Even in simple projects with only one document, this corpus structure exists and can be used to extend or re-organize existing corpora. The minimal structure for a corpus with one document is therefore: One corpus graph containing one corpus containing one document.

In Hexatomic, the corpus structure is always visible in the special “Corpus Structure” editor.

Corpus graphs

Corpora, sub-corpora and documents are organized in a hierarchical structure, the so-called corpus graph. A project in Hexatomic can have more than one corpus graph, but for most projects a single corpus graph is sufficient. In the special case where you import different corpora from different annotation formats into the same Hexatomic project for merging them, you will need more than one corpus graph.

In an empty project, just click on the “Add“ button to add a new corpus structure.

The default “Add“ button is context-sensitive, and will add elements "intelligently", depending on which type of element is currently selected in the corpus structure. To explicitly choose the element to add, click on the small arrow on the right side of the button and a drop-down menu with the different options will appear.

If you delete a corpus graph, all of its documents and corpora will also be deleted. Before you delete a (sub-) corpus, delete all of its child elements first.

Corpora and sub-corpora

Inside a corpus graph, the different corpora and sub-corpora are organized as a hierarchy. A corpus graph should only contain one top-level corpus, whose name is often used as corpus name when exporting the corpus to a different format. To add a sub-corpus, select the parent corpus, click on the arrow on the right side of the “Add” button and choose “(Sub-) Corpus”. You can edit the name of a corpus by double-clicking on its entry and pressing enter when finished.

Documents

When a corpus is selected, the default action for the "Add" button is to add a new document. When a document is selected, the "Add" button will create a new sibling document in the same parent corpus. Documents must have a corpus as a parent and contain the base text and linguistic annotations. You can move a document from one (sub-) corpus to another by dragging and dropping it.

It is possible to apply a filter, to only show documents whose names contain a certain string.

Opening an editor

To open an editor for a document of the corpus, first select the corpus in the "Corpus Structure" editor on the left. Then, use the right mouse button to open the context menu for the selected document and click "Open with Text Viewer". This will open a new view where the text of the document is displayed.

Graph Editor

The graph editor is for visualizing and annotating annotation graphs. It provides a general visualization that displays all possible types of annotation in a graph of annotatable elements.

Filters

On the right-hand side of the interface, you can select which segment of the current document to show in the graph view. For large graphs, it can take some time until its layout is calculated. The checkbox next to the segment indicates if this calculation is finished. You can select more than one segment to display, by holding the Ctrl key while clicking on additional segments. You can also show a whole range of segments by holding the Shift key and clicking on the last segment of the range you want to select.

You can also choose to display spans and their annotations in the graph, by checking the checkbox Include spans. Spans are special nodes to collect a number of tokens and to annotate them all at once. If you want to learn more about spans, please read the Salt documentation.

Similarly, you can show or hide pointing relations between nodes in the graph by using the checkbox Include pointing relations.

And you can filter the segments that include annotations of a specific name by using the filter text field above the list of segments.

Graph view

The left-hand side of the graph editor is taken up by the graph view, which displays the nodes and relations in the data model of the current document.

You can navigate the graph view as follows:

• Zoom in and out by using the mouse wheel.

  • You will zoom in to where your mouse cursor is.

• Zoom in and out by using using the keyboard.

  • If you press and hold the Ctrl key, you can zoom in with the + key and zoom out with the - key.

• Move the area of the graph that is displayed by using the keyboard:

  • The Arrow keys move the area in the respective direction, and PgUp and PgDown move it up and down.
  • If you press and hold the Shift key and then use the arrow or paging keys, you move more quickly.

• Move the area of the graph that is displayed by using the mouse wheel and function keys:

  • If you hold down Shift key while moving the mouse wheel, you can scroll up and down.
  • If you hold down Ctrl key while moving the mouse wheel, you can scroll left and right.

• You can center the view around a specific point in the graph by double-clicking that point.

If you don't like the layout of the graph, you can change it by simply dragging nodes with your mouse.

Console

On the bottom of the graph editor, there is the console which you can use to actually edit the graph. How to do this is explained in detail in the next section, Editing the graph.

Editing the graph

The graph editor contains a console, which you can use to manipulate the annotation graph. You first enter a command by entering it as text behind the so-called prompt > and pressing Enter.

Commands typically start with its name and a list of arguments. The arguments are specific to each command but can share similar syntax. Hexatomics command line syntax is similar to the one of GraphAnno.

Currently, the following commands are supported.

• t: Tokenize
• tb/ta: Tokenize before /after a token
• n: New node
• e: New edge
• a: Annotate
• d: Delete elements

Tokenize: t

Tokenize the given argument string and add the tokens to the annotation graph. String values can be enclosed in quotes, e.g., for punctuation and for tokens that include whitespace.

Examples

t This is an example "."

This command will result in 5 tokens: [This] [is] [an] [example] [.].

If you call t again, the new token will be appended to the end. E.g. calling t Not . will result in 7 tokens in total: [This] [is] [an] [example] [.] [Not] [.]. Note that the dot is not escaped with " quotation marks in this example, and that t Not. would also work.

Other than that, escaping punctuation with quotation marks is required for all non-alphabetical characters, to ensure correct tokenization, such as in t I "'" m ....

Tokenize before (tb) and after (ta) a given token

Tokenize the given argument string and add the tokens to the annotation graph before or after a given reference token.

Examples

Starting with an initial text with the two tokens [This] [text] (first one is called "t1" and the second one "t2"), executing

tb #t2 very simple

will append the two new tokens before the second token: [This] [very] [simple] [text]. Given the new tokens, calling the following command will insert the two new tokens after the first token ([This] [is] [a] [very] [simple] [text]).

ta #t1 is a

See also

• Identifying elements

New node: n

The command n will create a new node, and dominance relations between the new node and existing nodes.

Additionally, it can be used to annotate the new node in the same command.

Arguments starting with # refer to the node names to which dominance edges are added (e.g. #someNodeName).

When the creation was successful, the console will print a message giving the name of the new node and its annotations.

Examples

Starting with the tokens [This] [is] [an] [example] [.], the following command will group "an example" to a node with the label "cat=NP".

n cat:NP #t3 #t4

The following command creates a new node using the namespace "tiger" for the annotation.

n tiger:cat:NP #t1

You can mix nodes and tokens in the n command. Also, the number of dominated nodes is not restricted.

n cat:VP #t2 #n1
n cat:S #n2 #n3

See also

• Identifying elements
• Defining annotations

New edge: e

You can add two types of edges to the graph: dominance relations (e.g., for syntax trees) and pointing relations (directed edges without a specific semantic). Dominance edges are created with the syntax e #source > #target where #source is a node reference to the source node and #target a node reference to the target node. For pointing relations, use -> instead of >. This syntax is used to reference edges in general, e.g., when annotating or deleting them. As with new nodes, initial annotations can be added as arguments: e #source > #target name:value.

Examples

e #t2 -> #t1 func:nsubj

This adds a pointing relation between #t2 and #t1 with an annotation named "func" and the value "nsubj."

e #n4 > #t5

This example adds a dominance relation between the existing nodes.

See also

• Identifying elements
• Defining annotations

Annotate: a

Adds, updates, or deletes annotations on existing nodes or edges. Takes as arguments the nodes or edges which should be annotated, and the annotation to add, change, or delete. You can delete existing annotations by leaving the value in the annotation attribute empty.

Examples

a pos:DT #t1 #t3

Sets the annotation "pos" to the value "DT" for both nodes "t1" and "t3".

a pos: #t1

Deletes the "pos" annotation for the "t1" node.

a func_alt:nominal_subject #t2 -> #t1

Adds the "func_alt" annotation to the existing pointing relation between "t2" and "t1".

See also

• Identifying elements
• Defining annotations

Delete elements: d

Deletes any node or edge of the graph. Give the elements to delete as an argument.

Examples

d #t1 #t2

Deletes nodes "t1" and "t2".

d #t4 -> #t3

Deletes the pointing relation between "t4" and "t3".

See also

• Identifying elements

Identifying elements

Elements in the graph are identified by the identifier on the node/edge in the graph. In the example below, the tokens have the identifiers sTok1, sTok2, sTok3, and t28.

Note that identifiers for the same type of element may look different within one and the same document (as in sTok1 and t28).

Defining annotations

Annotation arguments have the form name:value or namespace:name:value.

This is true both for defining new annotations (namespace is optional), and for addressing existing annotations (namespace is required).

Grid Editor

The grid editor is for annotating tokens and spans.

Tokens are the smallest countable units in a data source. Spans are units that span tokens. To learn more about how Hexatomic's data model - Salt - defines these units, please refer to the Salt documentation.

Data source selection

A corpus document may contain more than one data source (text, audio, video), and their corresponding annotations. The grid editor displays one of these data sources at a time.

On the upper hand side of the grid editor interface, there is a dropdown menu to select the data source which should be displayed. If a document contains only a single data source, it is automatically selected.

Rows, columns, and cells

Rows in the grid contain a single token, its annotations, and the annotations on spans that overlap this token. A token in this case is the section of the data source the token covers. Depending on the type of data source and the resolution of the tokenization, this may be a word, a morpheme, a phoneme, a section of an audio or video source, etc.

Columns in the grid contain all values of a single qualified annotation in this document, that is, a unique combination of namespace and name of an annotation. Additionally, the tokens in a document - or rather, the segment of the document text they represent - are displayed in separate columns preceding any annotation columns.

Note that overlapping spans which are annotated with the same qualified annotation cannot be visualized within the same column. Instead, the annotation values are spread over more than one adjacent columns, whose headers are then suffixed with the count of existing columns for that qualified annotation.

Example
One span (S1) covers the first token in a data source, another one (S2) covers the first and second token. Both spans are annotated with values for an annotation five::span_1. Obviously, the overlap of both spans (both include the first token) cannot be visualized in a single column. Instead, S1's annotation value val_span_1 will be displayed in one column with the header five::span_1, S2's annotation value val_span_3 in another column with the header five::span_1 (2).

Merged cells represent annotations on spans that cover more than one token. You can see this in the last grid column in the screenshot above.

To distinguish between the different types of content diplayed in the grid cells, they are styled differently:

• Token text is displayed in italics.
• Token annotations are displayed in a black font.
• Span annotations are displayed in a green font.

Navigation and selection

The grid can be scrolled with the mouse using the scrollbars. If you have a mouse wheel, it will scroll horizontally. If you press Shift while using the mouse wheel, the grid will scroll vertically.

Alternatively, you can navigate the grid by selecting a cell and using the arrow keys to navigate. Pressing Home will jump to the first column, End will jump to the last column. PageUp and PageDown will jump a page up or down, a page being approximately the number of rows that fit the screen.

You can select whole rows or columns by clicking on the respective header. When you select a cell, press and hold Shift and click on another cell, the region between these two cells will be selected. This also works for headers, where all rows or columns between the two selected header cells will be selected.

Selecting non-adjacent cells, rows, or columns works by pressing and holding Ctrl while clicking the items to select.

Text search

You can search the grid using free text or regular expressions, with a number of options, e.g., search in columns first.

To bring up the search window, press Ctrl + F.

Display options

Freezing grid sections

You can freeze a section of the grid, so that it it remains visible while you scroll. This comes in handy for example when you have a large grid and want to keep the token column visible at all times.

To freeze a specific section, select the last cell, row or column that should remain visible, and press Shift + Alt + F. To unfreeze the grid, press Shift + Alt + F again.

Column and row freezing is also available from a popup menu in row and column headers, where you can (re-) set the row or column freeze, or toggle the general freeze state. The popup menu is opened by right-clicking the row or column header.

Hiding columns

You can hide columns by selecting one or more columns, and clicking Hide column(s) in the popup menu, available on right-click in any of the selected column headers. To show columns again, click the option Show all columns in the same popup menu.

Auto-resizing rows and columns

You can auto-resize rows columns, so that the row is high/column is wide enough to show all cell contents and the header content completely. To do so, simply select one or more rows or columns and click Auto-resize row(s) or Auto-resize column(s) in the popup menu, available on right-click in any of the selected headers.

Moving columns

You can move columns to another position in the grid by clicking on and holding the header of a column, and dragging it to its new position.

Export to Excel

If, for whatever reason, you want to export the grid to a file in the Excel .xls format, you can do so. Press Ctrl + E. This brings up a dialog for saving the file.

Note that exporting to an Excel spreadsheet will lose the actual data model. Only the string values of annotations, the headers, and the token texts will be exported. Additionally, merged cells will be separated.

Editing values in the grid

Editing cells

Editing a single annotation

To edit a single annotation value, you have to activate the single cell editor. The left-hand side of the screenshot above shows an activated single cell editor.

There are several ways to activate a single cell editor:

A. Double-click on the cell you want to edit.
B. Press the Space key.
C. Just start typing the new annotation value.

Editing multiple annotations at once

You can edit multiple values at once. To do so, select more than one cell (see Navigation and selection), and press Space. This will bring up a multi-cell editor window where you can edit the value of all selected annotations. The right-hand side of the screenshot above shows three selected cells, and the multi-cell editor window to change their values.

Adding or changing values

You can add annotation values to empty cells, or change existing ones.

Adding or changing a single value

In the single cell editor, type in the new annotation value and press Enter to commit the new value.

You can cancel the edit by pressing Esc. The cell value will remain the same as before you started editing it.

Adding or changing multiple values at once

In the multi-cell editor window, enter the new value for all selected cells, and commit it by clicking OK, or pressing Enter.

As with the single cell editor, you can cancel the edit by pressing Esc, or by clicking Cancel.

You can also do both adding and changing at once. If you have selected a mixture of empty cells and ones with existing annotation values, the new value you commit in the multi-cell editr window will be set to all cells alike.

Deleting annotations

You can delete annotations in two different ways:

1. Set an empty annotation value in one or more cells. This works regardless of whether you edit a single cell, or multiple cells at once.
2. Select one or more annotation cells, and then
   A. either press the Del key, or
   B. right-click with the mouse and select Delete cell(s) from the context menu. This menu item will only be available when deleting the selected cells is possible.

After you have deleted one or more annotations, one or more spans could be left without any annotations. In this case, these spans will also be deleted.

Troubleshooting Hexatomic & getting help

Please go through the sections on this page to see if one is what you are looking for. If you don't find what you are looking for here, please report the issue in Hexatomic's issue tracker. The section "Reporting issues" explains how this is done (it's quick and easy).

Sections

How do I do X? Why doesn’t Y work? Where can I go to get help?
The documentation didn't help me!
I've found a bug!
I'm looking for Hexatomic's source code!
I need to contact the Hexatomic team!

How do I do X? Why doesn’t Y work? Where can I go to get help?

This user documentation is the one-stop source of information on using Hexatomic, and can hopefully help you.

If you have read through it and haven't found an answer to your question, you can join the Hexatomic User Mailing List (to be announced) and ask your question there.

If you have an idea how we could make the documentation better so that the next person with your question can get it answered in the documentation, please let us know! "The documentation didn't help me!" below shows you how to do this.

The documentation didn't help me!

This user documentation is the one-stop source of information on using Hexatomic.

If you find that something is missing from the documentation, or that it could be made better in any way, please let us know! The way to do this is through reporting this issue on Hexatomic's GitHub page (cf. below for how to do this).

I've found a bug!

If you think you have found a bug, or Hexatomic does not work the way you have expected after reading the documentation, or Hexatomic doesn't work at all on your machine, please let us know! The way to do this is through reporting this issue on Hexatomic's GitHub page (cf. below for how to do this).

I'm looking for Hexatomic's source code!

It's open source and can be found online at https://github.com/hexatomic/hexatomic.

I need to contact the Hexatomic team!

We'd love to hear from you. Please write us an email at hexatomic [at] corpus-tools.org.

Reporting issues

We use GitHub, a web-based platform for collaboration on software, to develop Hexatomic. The Hexatomic GitHub page is at https://github.com/hexatomic/hexatomic.

This is the place where you can report an issue (a bug, missing documentation, etc.) and suggest new features.

To do so, you need a GitHub user account. If you don't have one yet, you can register for one at https://github.com/join. It's free!

First of all, please read the Contributing guidelines. It's a quick read, and the guidelines contain important details.

Then, when you are logged in, go to https://github.com/hexatomic/hexatomic/issues/new to create a new issue.