NOTICE: This page is frequently updated as new features and training materials are created. Questions may be sent to info (at) insitulabs.org
THE ISL DATA SYSTEM OVERVIEW
PURPOSE
Historically, biological sample collection at remote field locations and laboratory analyses have been carried out independently of one another, by separate institutions and/or groups of stakeholders. Most data systems that support these distinct parts of the research process have been developed in isolation, and remain 1) unintegrated or 2) unadaptable to diverse environments with power and data service constraints. As we endeavor to set-up more fully-functional molecular laboratories in the field (in-situ), our goal is to seamlessly integrate biological sample collection with laboratory sample analysis and data sharing applications in an intuitive, user-friendly, and secure way.
ACCESS
We strive to build off of other prior openware initiatives, and remain committed to making all our data system tools low cost, and where possible, completely free for scientific research, public health, and conservation applications. The system is fully supported for all official members of the ISL initiative, external parties should send email inquires to info (at) insitulabs.org.
DESCRIPTION
The In Situ Laboratory Data System is a combination of tools to capture, organize, clean, and synchornize data for a variety of downstream applications. The following describes the flow of data:
- Data enters the system via direct entry, spreadsheet upload, or 3rd-party data form collection tools. We recommend the use of digital forms, such as those provided by the Open Data Kit (there are many good options), as they ensure users collect and submit information in standardized ways.
- Data is organized into a MongoDB database platform. MongoDB is similar to other database platforms, but we like how it accepts data in many formats and structures (incl. mixed data types) without complaint.
- Attachments and media are automatically moved into an Amazon S3 server, text and numerical data are stored directly in the cloud database, which is backed-up.
- Users control access and interact with the database via the ISL app, a software designed by field and laboratory research personnel, then implemented and maintained by the ISL technology team.
STRATEGIC OVERVIEW
The ISL Data App allows researchers detailed control of a database without requiring programing experience. Basic computer literacy is required.
Key Functionalities – The WHAT and the WHY
| WHAT | WHY |
| Operates on common internet browsers | The system is not tied to Windows, Apple OS, or Linux. |
| Language customization | Each deployment of the system allows any number of language translations. Facilitating international collaborations. |
| Simple graphics and organization | Not built to be flashy, rather it must work when internet bandwidth is limited. Should feel familiar, like browsing a personal computer file system. |
| Customizable to different use-cases | Structure of data tables, custom views, and relationships are controlled by end-user |
| Simple and bulk data table edit options | Many researchers default to spreadsheets because they cannot change records efficiently |
| In-built auditing and operation reversal | All data operations are tracked and easily reversed by permitted users |
| Specific user access for data viewing and editing | Open collaboration requires giving specific table-by-table and view-by-view access |
| Cross-table relationships | Easily add one or more links between tables |
| Batch import and update | When it is easier to review data and make corrections outside of the system (i.e. in Excel), efficiently batch update any modified fields. |
| Data filtering, sorting, summary | Self-explanatory |
| Custom VIEW creation | Transformations between wide/long data tables, and merge multiple tables together by linked or common variables |
| Data movement and tracking | Move whole or partial records between tables, as in a laboratory or analysis workflow, establishing live links between sources and destinations |
Basic Features
An empty database, there are no tables. Select “create source” to get started.
Give the source a name, description, and start manually adding variables. When finished, “Save” the new table
The SOURCE page now has tables
Instead of adding variables manually, give a new source table a name and select “save”, then import variables and data from any XLSX or CSV spreadsheet.
Importing a new source has two steps - (1) load the source from machine, (2) check the data and then select “Import records”
Enter a table by selecting it’s name from the SOURCE page, then select “Actions” then “Creat” to manually enter a new record.
“Double-click” any field and select “Attachment” to insert a document. Images and PDFs can be previewed directly in the system.
Bulk update records by selecting their check-box on the left-most side, then double click in any cell and enter a new value. All selected records will now have that new value.
Notice that a variable can have mixed data types, as shown in Var3, this can be disallowed if desired.
In the upper-right corner, 'clicking' user name reveals a drop down menu with options to control 'Theme' mode, 'Settings', 'Users', and quick access to other databases available this account.
In 'Settings' create categories for SOURCE tables and VIEWS. These categories are used organize the contents of the database like a folders on a computer.
Two SOURCE table categories have been made "Project A" and "Project B", they can be expanded and collapsed, like folders on a computer.
In the 'Users' console, give staff and partners access to specific sets of data. Allow them to EDIT or only VIEW each item.
From the 'User' console, access the 'Audit' panel. In the 'Audit' panel all activity of the data system and undo operations, if necessary.
Advanced Feature: VIEWS
The VIEWS console will be empty until a view is created. Here there is one view, Table1_Table2.
Views allow source tables to be combined row by row or merged by a common column. This view has five variables. Source data may be mapped to these fields, or they may be left unmapped for the addition of information that is specific just to the view.
This type of view, combines multiple tables by row, 'source name' indicates where any mapped data originates. Cells are color-coded to indicate if values are mapped or unique to the view. When a cell is manually edited, the cell color is temporarily highlighted.
All source and view tables have a "analysis" feature, rows can be grouped and summarized for quick data insights.
Advanced Feature: RELATIONSHIPS & DATA MOVEMENT
Rows of data can be moved from a view or source table to any other table. This ACTION is called "Copy To".
After executing the 'Copy To' action, a page will indicate the new records that have been created in the destination table, 'click' the link to see them in the destination table
In the "Copy To" panel map selected records from one source or view to another destination source table. Notice the pre-established relationship between the two tables that is indicated at the bottom. This will ensure that data which is copied from one table to another remains linked. When ready, scroll to the bottom and select "Copy"
By 'clicking' a look-up field, a pop-up window shows the linked record(s). Here we see the destination source table, and we also see that it is linked back to the origin source table.
If 'source look-up' fieles were correctly established before executing the 'copy to' action, the origin records are linked to the destination records. The updated fields are highlighted temporarily, until the the page is refreshed.
Instead of selecting a traditional data type, 'Source Lookup' fields are used to refer to related tables, similar to how primary and foreign keys function in a traditional relational database.