iDigBio Media Ingestion Application

iDigBio Image Ingestion Utility (ver. 2.5.2)

The image ingestion tool can be used to upload web images (JPGs) from one’s local system into the iDigBio image server. Once the images are web accessible from the image server, they can then be mapped to specimen records residing within any Symbiota portal. It’s recommended that the JPGs have minor jpg compression applied since it can greatly reduce the file size and improve efficiency. A typical image of an herbarium specimen (ca 4000 x 6000px) should have a file size in the range of 1-6M. If the image is too large, it will slow down data entry, OCR processing, and may inhibit uploads depending on web server configurations. Click here to download a Word document version of these instructions. 

  1. Download and install the iDigBio Image Ingestion Utility from their GitHub repository (https://github.com/iDigBio/idigbio-media-appliance/wiki/Installing). This only has to be done once, if the software is already installed, continue to step 2.
    • Windows users: After install, if typing “idigbio_media_appliance” in the start menu fails to start the application, restart your computer and trying again. Also try typing “idigbio_media_appliance.bat”. If that still fails, search your file system for the “bat” file located in the “Scripts” directory within the root media appliance directory. Depending on the operating system and installation options, the root directory may be one of the following: C:\idigbioMediaAppliance\, C:\Users\<YOUR_USER_NAME>\idigbioMediaAppliance\, C:\Program Files\idigbioMediaAppliance\. Once you locate the .bat file, I recommend creating a shortcut on your desktop (right click on file and select Create Shortcut).
    • Enter Security keys: When you run the Image Ingestion Utility for the first time, you will be prompted for a UUID and an associated API key. If you do not have a UUID and API key pair, you should contact your portal, project administrator, or the iDigBio contact listed on the installation page. Account Alias field is optional and can be left blank.
    • If you have previous versions of the appliance installed, you may want to clear your browser cache to make sure that previous versions and code are not loaded from the cache.
  2. Upload Images to iDigBio: In order to upload the files, the Image Ingestion Utility needs to create a CSV file that lists the local file locations for the images you wish to load. Click on the “Generate CSV” tab to have the Utility create this file for you. Within this tab, enter the following. Also see screenshot below.
    • Upload Path: Enter your full pathway to where your images are stored on your local system (e.g. C:\Images\ or /Images/). BUG with PC using Windows OS: Clicking on “Choose Directory” button will open a file browser, but it may open it behind the main browser window. If nothing seems to happen when you click the button, minimize the main window and you will likely find the file browser panel hiding behind. If you clicked the button multiple times, you will have to close multiple file browser panels. If the appliance, it may be due to multiple active windows open behind the appliance. Once you locate the file browser, use it to select the file directory containing the images you wish to load. Or you can type in the path manually.
    • GUID Syntax: Leave as is (GUID: randomly generated UUID), or select your preference
    • Clicking on the blue “Generate CSV & Upload Files” button will generate a CSV file and start the image uploading process. The Upload History tab should automatically open and a progress bar will be displayed. If you have a large number of images, the upload process might take a while so you might want to grab a cup of tea or coffee.
      • If some of the upload fails halfway through due to a network connection issue, or such. You can restart the process by going to the “Upload History” tab and clicking on the “Restart Upload Task” button.

 

mia_2

 

  1. Link Image URLs to Specimen Records: Once the upload process has completed, all the images will be web accessible through any web browser. Follow the steps below to link the image URLs to specimen records managed within a Symbiota portal instance.
    • Generate Upload Report: Create a report of the upload process by going to the “Upload History” tab and clicking on the “Download Media as CSV” button (see screenshot below). This should download a CSV report to your local system. Depending on the configuration of your local system, the report might be placed within your download folder, desktop, or you may be asked where you want to save the file. You can use the pulldown to the right of the page to limit the report to only the most recent records. Note that there are new buttons to purge missing and failed image loadings.

 

mia_2

 

  • Log into your Symbiota portal, navigate to your management menu, select Processing Toolbox, and then the Image Loading. If you don’t see an “iDigBio CSV Upload” profile setup like the one in the screenshot below, you will need to ask your portal manager to set one up using the following directions. Note that this only needs to be done once.
    • Select “iDigBio CSV Upload” from the Upload Type drop-down menu,
    • Enter a regular expression that will capture the catalog number from the file name. Remember to enclose the portion of the expression representing the catalog number in parenthesis. Setting up a regular expressions can be difficult, thus don’t feel shy about asking for help. Also see: http://www.regexr.com
  • Link Image URL to your Collection: From within the Image loading tab, click the “Choose File button” and select the Upload Report that you generated in the previous step. Click the “Process Output File” button. This will start the linking process which inspects each record and does the following: 1) extracts the catalog number from the file name, 2) looks to see if a record with that that catalog number already exist, 3a) if it does, it links the image to the existing record, 3b) if not, it links the image to a new empty records primed only with the specimen catalog number
  • Note that if a large number of images are loaded on to the iDigBio system by several contributors at once, the processing scripts can fall behind and create a delay in generating the image derivatives used in the portal display. Therefore, if you see an image not available message after mapping to your data portal, be patient and the images should eventually appear.  If they are not available within a 24 hours period, email your portal manager or iDigBio support staff to check on the status. 

 

mia_4

* If you have a CSV skeletal records file that needs to be uploaded, this can be done using the “Skeletal File Upload” option in your collection management menu.