IRODS storage user documentation

From BILS Wiki
Jump to: navigation, search

Introduction

iRODS is a data grid/data management tool. It allows you to store data in a unified namespace using multiple storage resources, to replicate data so that copies exist on multiple systems, and to store checksums and arbitrary metadata with a file. The BILS iRODS configuration supports accessing iRODS through either the native iRODS tools such as the UNIX “i-Commands” or via the HTTPS protocols, using your browser.

The purpose of the BILS iRODS service is to provide an easy-to-use, high-availability system for long-term storage to the Swedish life science community.

Accounts

The BILS irods storage service is free of charge for storage of up to 2TB. For larger storage requirements, there is a service charge of YY SEK/TB/year. For write privileges to BILS IRODS, you will be sent an authentication device. For read-only access, username and password is sufficient authentication.

In order to obtain a user account for BILS irods service, go to the BILS service account registry.

General iRODS concepts

In principal, iRODS closely resembles most file storage solutions. One of the key differentiators is the ability to add generic metadata to objects, and to add rules that manages the data automatically in different ways. iRODS also supports various access controls.

Access

Obtaining the command line tools

If you want to use the iRODS command line tools from your own workstation, you need to install the iRODS icommands package. Source code as well as binaries for several different GNU/Linux distributions can be obtained from the iRODS download page.

iRODS setup: Command-line usage

The iRODS command-line utilities, collectively referred to as “i-commands”, use an environment file to store iRODS initialization information under a user’s home directory. A template version of that file is shown below in Example 1 below. To set up your computing environment, copy and paste this text into a file called ~/.irods/irods_environment.json, then edit the file, replacing the value after “irods_user_name” with your LDAP username (usually your e-mail address). Once you have created and saved this file, you can issue the “iinit” command to start your iRODS session, after which you can store and retrieve data normally using the i-commands as documented below. If you will be accessing iRODS only through your browser, you do not need to create this configuration file.

Example 1. iRODS configuration template

{
  "irods_host": "irods.storage.bils.se",
  "irods_authentication_scheme": "PAM",
  "irods_zone_name": "bilsZone",
  "irods_port": 1248,
  "irods_user_name": "user.email-address@xu.se"
}

If your system has an older version (<4.1) of the icommands installed, please use the old setup instructions

See below for more information on icommands usage.

iRODS setup: GUI usage

See User Documentation - iDrop web

Basic i-Commands

Once you have configured the ~/.irods/.irodsEnvfile, you can use i-commands to access and manipulate data in the system. The i-commands nomenclature mimcs that of UNIX but with an “i” prepended to the command name e.g. ils, imkdir, icd. Generally, i-commands are functionally equivalent to their UNIX counterparts. Complete i-commands documentation can be found on the iRODS site. The following table summarizes some of the most common i-commands. Caveat: If you’re used to unix-scripting, be aware that because of how the i-commands handle state, they may not always behave as expected.

Table 1.0 Common i-commands

i-command Syntax Description
iinit iinit yourpassword initialize and start an iRODS session
ils ils <file or directory> like UNIX ls, list files or directories
ilsresc ilsresc list all iRODS resources
iput iput -v filename /bilsZone/home/user/target_subdirectory store file/s into the system
iget iget filename /bilsZone/home/user/target_subdirectory retrieve file/s from the system
imkdir imkdir <new_directory> like UNIX mkdir, create a new directory (collection)
icp icp <source> <destination> copy a file or directory, many options available

A Few Other Useful i-Commands

irsync

Use the irsync command to synchronize a local directory with iRODS, similar to the Unix rsync command. It can be used to make an exact copy of a directory hierarchy on a local disk within iRODS, or retrieve an exact copy of a directory hierarchy already stored in iRODS. It may also be used to create an exact copy of a file or directory within iRODS. iRODS paths are identified with an i: prefix in the irsync command.

For example, if you have created a directory within iRODS called /home/joeuser/myproject, and you wish to retrieve an exact copy of that directory on your local computer, run the command:

irsync -r i:/home/joeuser/myproject /path/to/joeusers/work/directory

After editing the files on your local computer, you can then synchronize the data back into iRODS , using the command:

irsync -r /path/to/joeusers/work/directory i:/home/joeuser/myproject
irm

Use the irm command to delete data entirely or just from a single resource. irm without options deletes all copies of a file. irm with the "-n" switch deletes a specific replica. For example, if you have stored data initially in the "cache" resource and then replicated it to Ranch, replica 0 will be the copy stored on the cache, and the command:

irm -n 0 <path_and_filename>

will remove the file from the iRODS cache resource, while leaving the archived copy intact. Use "ils -l" before deleting a replica to ensure that you have a copy in more that one resource, and that you are deleting the correct replica.

irm options include:

  • -f force data removal
  • -v for verbose
  • -r for recursive
  • -h for help
ichmod

All users can see all other users’ collections and files but cannot access, , where they do not have permissions. The ichmod command, like the UNIX chmod command, allows a user to give file access permission to other users or groups.

Read Permission:

ichmod read testuser testfile.txt 
ils -A testfile.txt

The "ils -A" command shows the Access Control List for the file “testfile.txt”. Here, testuser has been given read permission on testfile.

Ownership Permission: Giving another user ownership permission will enable them to change ACL for the file or folder. For example, to give testuser ownership permissions means testuser can them extend read/write/owner permissions to other users.

ichmod own testuser testfile.txt

Removing Permissions: You can assign “null” to remove permissions from the ACL for a file or folder:

ichmod null testuser testfile.txt

ichmod command options include:

  • -v for verbose
  • -r for recursive
  • -R for Resource
imeta

The imeta command is one way to add user-defined metadata (data about data). ’imeta add -d’ means to add to the data-object . These are Attribute , Value, and optional Units items , a.k.a. AVUs.

imeta add -d coll2/fileb1 platform ILLUMINA

imeta can also be used as a powerful tool to query on metadata to find data-objects of interest:

imeta qu -d platform = ILLUMINA
iquest

The iquest command can be used for querying for various information from the iRODS SQL database (ICAT). It is a little bit too low-level for most daily tasks, but it can be useful e.g. for calculating the resource usage of files and folders in iRODS.

Calculating resource usage is done in the following way (modify the file path according to the actual folder):

iquest "select sum(DATA_SIZE) where COLL_NAME like '/tempZone/home/rods%'"

Use iquest -h to see more examples, or man iquest to read an overview help text.

External links

Some links where you might find more documentation: