Deep Lens Survey: fiat tools
Fiat tools for manipulating DLS data

If you've ever downloaded and looked at the DLS catalogs, you'll see that they have a funny header that starts with "# fiat 1.0". All comment lines start with "#" so that the files are reasonably easy to manipulate with awk or similar tools, but we have a set of tools designed specifically for typical operations on catalog data.


All these tools print help if -h is used as an option, either because -h is the help option or because it is a nonexistent option. For each tool, I provide a brief description and then present the help. Output is to stdout unless otherwise specified.


Calculate any function of columns and put the values in a new or existing column.

Usage: fiatarith [-dhsv] expr fiatfile
  Perform perl expr on fiatfile, e.g. 'r=sqrt(x*x+y*y)'
  In this example, x and y must be valid columns; r is 
  overwritten or created as necessary.
    -d: debug.  Print output header and interpreted expr, then exit.
    -h: print this help
    -s: don't output lines where errors such as divide by 0 would occur.
        (Default behavior is to exit with an error message.)
    -v: verbose

  Note: currently the only real way to catch errors such as divide by zero
        is with a little program like this: 
        However, the -s option should be adequate in most cases.
The way to do things like zeropoint a catalog. Example:
fiatarith mag=mag+25.2 >
Or calculate ellipticities:
fiatarith 'ell=sqrt((ixx-iyy)*(ixx-iyy)+4*ixy*ixy)/(ixx+iyy)' cat1 > cat2
In this case, an ell column will be created if it doesn't already exist. You can specify more than one expression with semicolons. For example:
If you have a column named, say, "sin", fiatarith will get confused with the sin function. Refer to the column as SIN since fiat column names are case-insensitive, whereas the function can only be referred to as "sin". If you want to see how your expression is finally passed to a perl eval, use the -d option.


Usage: fiatbin [-msv] column1 column2 range nbins fiatfile
 Bin column1 of fiatfile into nbins over range.  Print bin center,
 number in bin,
 and mean, stddev, min, and max of column2 in that bin.
  -m: treat second column as a magnitude; sum the luminosities and
      the total magnitude
  -s: suppress printing of bins with no objects.  
  -v: verbose

  --range is either a range such as 0-20, or "a" for auto
  --if nbins is of the form "5l", 5 log-spaced bins are used
  --if nbins is of the form "0.5w", 0.5 is used as a bin width
fiatbin is now strictly 1-d, and is much better in 1-d than the old version which did both 1-d and 2-d binning (now called fiatimage). For example, to get the total magnitude in 8 log-spaced annuli from 50 to 5000 pixels radius,
fiatbin -m r mag 50-5000 8l catalog
To get statistics on the ixx as a function of x, in 10 equal-size bins over the range of the catalog,
fiatbin x ixx a 10 catalog

Filter a catalog and ouput selected objects to stdout.

Usage: fiatfilter [-hinv] boolean fiatfile
  Filters objects in fiatfile which meet the boolean condition,
  e.g. 'x<1000 && y>2000'.
  Outputs new fiat catalog to stdout unless -i is used.
    -d: debug.  Print expression as passed to perl and exit.
    -h: print this help
    -i: inplace--overwrite input file with output
    -n: print only the line numbers (0-indexed) of objects which pass filter
        Do not use with -i!
    -r: reverse sense of filter.  Print objects for which expression is false.
    -v: verbose

  Special constructions:
    exclude(filename) reads in a Ian-format region file and excludes 
                      objects in those regions.
    exclude(filename,xname,yname) does the same for coordinates "xname" and
                      "yname" instead of the default "x" and "y"
    Warning: constructions like !exclude will not have the desired effect.
    $nin refers to line number in the file, 0-indexed.  Hence "$nin<100"
          will pass the first 100 objects.
The filter condition can be any valid perl boolean expression, except with column names instead of perl variables. If something in the expression is neither a column name nor a common perl function, a warning is printed. In most cases, the filter still works (e.g. the thing which is neither a column name nor a common perl function is simply a constant), but the user should double check.

Warning: You can make lots of errors which can't be caught. For instance, '=>' is a valid perl operator, but it is NOT the same as '>=', and you almost certainly want the latter! Similarly with '=' which is assignment and '==' which is comparison. This is the first thing to check if you get unexpected behavior.


 fiatfilter 'x<1000 && y>2000 && ampid=1' file1 >file2


Generate a fiat file according to a prescription.

Usage: fiatgenerate nobjects columnname=recipe ...
  where recipe is a perl expression.  Examples:
    id=$i : object number (zero-indexed)
    x=rand(2047)+1 : x is uniformly distributed between 1 and 2048
    z=0.5*gauss()+1 : z is normally distributed about 1 with an rms of 0.5

  Be sure to quote arguments which need protection from the shell!
  Refer to previous columns as $col[1], $col[2], etc. Example:

  fiatgenerate 1000 'x=rand(2048)' 'y=rand(2048)' 'r=sqrt($col[1]*$col[1]+$col[2]*$col[2])'

  If the command line gets tediously long, use a filename in place of 
  columnname=recipe form.  The file should have one columnname=recipe 
  prescription per line.  This also eliminates the need to protect arguments 
  from the shell.  filenames and recipes may be interspersed on the command line.

Note: gauss() is not a builtin perl function. It's a useful shortcut provided by fiatgenerate. I hope to provide more of these upon request.


Print basic information, such as number of columns and entries. Not terribly useful in real life, but provides the basic perl code which anyone can recycle.

Usage: fiatinfo [-chln] [-k keyword] [-p colname,colname,...] fiatfiles
  -c: count entries
  -h: print this help
  -n: name columns
  -k: get value of keyword
  -l: list all keyword-value pairs
  -p: print named columns
  if no options specified, -c and -n are switched on

An example from a converted sextractor file:

% fiatinfo
146 entries
col 1: number
col 2: x
col 3: y
col 4: mag
col 5: magerr
col 6: background
col 7: x2
col 8: y2
col 9: xy
col 10: a
col 11: b
col 12: theta
col 13: ellipticity
col 14: flags


List functions of columns to stdout. Unlike most of the other tools, this does not produce another fiat file by default--it was intended to produce streams of numbers for input to other programs, for example for calculating statistics. But it will produce another fiat file if -n is used.

Usage: fiatlist [-dhv] [-n names] quantity fiatfile
  Outputs an ascii list of quantity to stdout.  quantity is
  any valid perl function of column names, e.g. 'x' or 'sqrt(x*x+y*y)'
  or 'x, ,y, ,mag' for multiple output columns.

    -d: debug: print 'quantity' as passed to the perl interpreter and exit
    -h: print this help
    -n: print a fiat header with names, e.g. "e1 e2".  By default there
        is no header, just an ascii list.  If -n- is used, the output column
        names are just as given in the quantity to be listed.  Examples:
        fiatlist -n'e1 e2' '(ixx-iyy)/(ixx+iyy), ,2*ixy/(ixx+iyy)'
        produces a new fiat file with e1 and e2 columns.
        fiatlist -n- 'e1, ,e2' does the same if e1 and e2 already exist.
    -v: verbose
The 'quantity' argument is doctored so that, say, "x" is replaced with the correct column number, and then passed to the perl print function. That doctoring sometimes has confusing results. For instance, 'x y' doesn't print x, then a space, then y; to do that, use
fiatlist 'x, ,y' fiatfile
If you have a column named, say, "sin", fiatlist will get confused with the sin function. Refer to the column as SIN since fiat column names are case-insensitive, whereas the function can only be referred to as "sin". If you want to see how your expression is finally passed to a perl eval, use the -d option.


fiatlist 'sqrt(x*x+y*y), ,atan2(y,x)' fiatfile
produces a list of radii and angles, and
fiatlist mag fiatfile | binstrm c 10 30 20
produces a luminosity function.


Paste two fiat files "side-by-side". Actually the data are side-by-side but of course the headers are manipulated appropriately.

Usage: fiatpaste [-v] [-reject|-pass columns] [-suffix suffix]
fiatfile ...
   Paste fiatfiles line-by-line to stdout, writing appropriate header.
   Number of output lines will be equal to shortest input file.
   -h: print this help
   -v: verbose
   -reject: reject named columns from next fiat file.
   -pass: pass named columns only from next fiat file.  Incompatible 
   with -reject when used on the same catalog.  However, you may mix 
   and match with an arbitrary number of catalogs, e.g. 
   fiatpaste -pass "x y" file1 -reject "x y" file2
   -suffix: add suffix to column names of next fiat file.
Warning: Make sure the catalogs are appropriate to paste together. If one is longer than the other, fiatpaste will silently truncate. See fiatmatch for another way to do similar things.


Delete or rearrange columns.

Usage: fiatrecolumn columns fiatfile
     Example: fiatrecolumn 1-3,5 file1>file2
fiatrecolumn is the only tool that deals with column numbers rather than names. That's because it's so much easier to specify 1-n than to type out the first n column names on the command line. Column numbers begin with one, and the specified columns are copied to stdout in the order given.

Example: rearrange columns

fiatrecolumn 4,3,2,1 file1 > file2


Interactively examine an image and its catalog.

Usage: fiatreview [-dnv] [-c cols] fitsfile fiatfile
  -c: use columns instead of "x y ixx iyy ixy"
  -n: no overlays except for specifically selected objects
  -d: debug
  -v: verbose

  Use fiatfile name of "-" to look at the image without a catalog.

You don't need an ximtool or saoimage. fiatreview will do its own display. Once the image and catalog are read in, you can do any of the following:
Image appearance:
        left mouse button: change brightness/contrast
        middle mouse button: pan
        + or =: zoom in by 2x around cursor position
        -: zoom out by 2x around cursor position
        n: load the same view of a new image
        r: rescale image intensities

Object selection: (red overlay=unmarked, green=marked,yellow=current):
        f: find object nearest cursor, and make it current
        m: toggle marked state of current object
        w: draw a rectangular window by moving mouse
           when done, hit m to mark objects inside, M to unmark

Catalog operations: 
        F: run fiatfilter (give expression at prompt) and mark
           all objects that survive filter
        s: save marked objects
        S: save unmarked objects

        right mouse button: print x, y, pixel value
        h: help
        q: quit

Usage: fiatshuffle [-h] columns fiatfile
    -h: print this help
    -v: verbose

  Example: fiatshuffle "ixx iyy ixy" fiatfile
           shuffles the moments while leaving all else intact

Shuffle quantities randomly over the catalog entries useful for making sure "signals" disappear in a random catalog with the same statistical properties as the data, and for estimating statistical errors.


Sort a fiat catalog.

Usage: fiatsort [-hlr] column file 
  Sorts file by given column and prints to stdout.
  -h: print this help
  -l: sort in lexical order (default order is numerical)
  -r: reverse (descending) order

A Word on the Format Itself

The philosophy behind fiat is not to create another rigid format, but to be so flexible as to satisfy the needs of both users and programmers. Users like to have ascii tables they can look and edit without needing any special programs. Programmers need to read the right information into their programs without too much hassle. So here's a sample fiat file with the coordinates and magnitudes of two objects:

# fiat 1.0
# ttype1 = x
# ttype2 = y
# ttype3 = mag
12.01 1034.67 19.81
1002.59 25.82 21.55

We have only two major rules: the usual unix convention that comments are marked by '#' at the start of the line, and columns are named in the fits table style with '# ttypen = name', where n is the number of the column (beginning with one). Hence users can easily browse and edit, and programmers can easily and robustly read in the right information.

Conversions To/From Other Formats

I have written four converters to fiat format, focas2fiat for focas catalogs, phot2fiat for ProFit .phot files, sex2fiat for sextractor ASCII_HEAD type catalogs, and gallist2fiat for for the lists produced by gallist in the IRAF artcat package. Apart from focas2fiat, they all take every file on the command line and stdin and write to stdout. A warning is printed if the input doesn't look like the required format. Note that sex2fiat reflects some of our tastes--we rename X_IMAGE to x, X2 to ixx, and so on. Also note that focas2fiat is just a script; you must have the focas program catlist in your path for it to work properly.

It's tough to convert from fiat to other formats, because none of them can hold the variety of information that fiat can. If we knew of such a format, we wouldn't come up with another one! However, sometimes conversion is desirable to deal with legacy software. Hence there is a fiat2focas program which deals only with x, y, mag, ixx, iyy, and ixy (this program is not made by default, since it requires focas libraries).

Many people use unheadered ascii lists in their everyday work. A simple way to convert to this format is

grep -v '#' fiatfile > newfile

To convert the other way, simply edit the file to put in a header explaining what the columns are, as demonstrated in the previous section.

Downloads and Installation Instructions
David Wittman
wittman at physics dot bell-labs dot com (anti-spam format)
last modified April 11, 2003