Random netcdf4-python notes

Some notes on using netcdf4-python

• Use the shared netcdf4 libraries to compile. (ROMS needs static netcdf4 libraries -- so you need to build both).
  
• The new version returns a np.ma.MaskedArray when there are missing values.
  
• Using ncks (from the netCDF operators) to compress -- need to be careful about specifying netcdf4_classic format. Standard netcdf4 format does not play nice with MFDataset.
  
• The new netCDF4 is DAP enabled, a feature that is passed through to netcdf4-python. All you need to do is specify a DAP-URL, and you can see your data over the internet!
  

Learning the shell.

This seems a good introduction to the unix shell.

LinuxCommand.org: Learning the shell.

non-linear colormaps in matplotlib

Here is a way to do non-linear colormaps in matplotlib. You can select a base colormap, and a series of levels, and this class will remap the (linear) colormap such that your new selected levels define the linear positions along the map. This can be used for plotting bathymetry, specifically, where you may want to have more contours in the shallows than deep. Grab the code here: nlcmap.py.

WaveMetrics - scientific graphing, data analysis, curve fitting & image processing software

From the scipy mailing list:

It's hard to describe what this amazing program does because it does
so much. I have a few personal dislikes about it but by comparison,
Matlab "isn't even wrong," to misuse a quote from Wolfgang Pauli.

WaveMetrics - scientific graphing, data analysis, curve fitting & image processing software

pydap and pupynere

pydap is a python module that will allow you to read (and serve?) netcdf files over the internet. This is very useful if you don't want to have an entire model run on your laptop. John Evans and I are working on prslice that uses this package.

Even more cool, however, is pupynere (PUre PYthon NEtcdf REader) that can read (only, for now) NetCDF files with only builtin python packages and nympy.

Compiling netcdf with ifort 9.1 on an Intel Mac

First, if you haven't already done so, you need to do this if you are runing Xcode 2.3, like I am. Otherwise, ifort will not make it through configure.

cd /usr/lib
sudo ln -s gcc/darwin/3.3/crt2.o .

Then, you can set up your environmental variables just like you would using ifort in Linux. For example:

export FC=ifort
export F90=ifort
export FFLAGS="-O -mp"
export CPPFLAGS="-DNDEBUG -DpgiFortran"

Then, like always, run

./configure --prefix=/usr/local
make

Shed Skin -- python to C++ compiler

Shed Skin - Shed Skin is a compiler that takes pure python code, and coverts it to annotated C++ code. Speedup is (suposedly) much higher than other similar methods.

stack size limits using ifort

I use ifort on both Linux and Intel Macs. I have come across some strange behaviors that make me think that the stack is not really unlimited. This post on automatic array stack size limit bug - Intel Fortran Compiler for Linux and Mac OS - Intel Software Network Forumsto the ifort message boards seems to suggest this is a bug in ifort.

This seems to be less of an issue on my Linux machines, where I can set ulimit to unlimited (particularly the stack). On my Mac, I can't set ulimit to unlimited (and if you want to increase it at all, you need to add a flag that lets you use the 'soft' limit). As far as I can tell, partitioning through MPI does not seem to help, but I will look into it more.

Real-time stream flow data in Google Earth.

The USGS has released two kml files for Google Earth that show real time river guage data relative to climatologies. You can click on the colored points and get information about the station. A sample screen grab from New England today shows the rain New England has been getting recently.

37mm blog

37mm's blog is a work of art, with tones of useful information to boot.

PyDSTool

PyDSTool is a python module for dealing with dynamical systems. For when the SciPy ode solver is just not good enough..

PyNGL and PyNIO

There is an effort at UCAR to translate NGL (the NCAR Graphics Language) to python: PyNGL. Also, as part of this project, there is PyNIO, and input/output module that reads and writes in many different formats supported by NGL.

TVTK

Although matplotlib supports 2D graphics (and quite well, I might add), there is no support for 3D graphics. VTK is clearly the way to go, but there has been no 'pythonic' interface... until TVTK.

Also of interest might be NCVTK, which provides another interface for visualization of NetCDF files using VTK.

Python: module netCDF4

YESSSSS!

Python: module netCDF4

Mesh/Grid Generation Software

Here is an alarminly long (perhaps complete) list of Mesh/Grid Generation Software . Aside from those known to the ROMS community, Pavel Sakov has a powerful set of grid generation utilities.

Also of interest, NCARs ngmath, a set of 2- and 3D interpolation utilities, and TSFIT by Oleg Davydov

NetCDF CF conventions File Checker

The NetCDF(CF) File Checker uses CDMS (Climate Data Management System) from CDAT (Climate Data Analysis Toolkit).

Calling Java classes within Python

Here are some interesting candidates for calling Java classes from within Python. As for Jython, I think there is a reasonable case against it (for the sorts of things I want to do) on the JPype front page.

• JPype - Java to Python integration
  
• Python/Java Wrapper Generator
  

Disk Inventory X

Daring Fireball posted a link to Disk Inventory X. This is an excelent example of functional design. Now I know that those history files are what is using all my disk space.. my music collection has nothing to do with it..

NetworkX

NetworkX is a python package for "reation, manipulation, and study of the structure, dynamics, and functions of complex networks." It looks quite cool.

Big screens really are better

During his lecture, Edward Tufte discussed the merits of a large screen. His basic argument is that a greater percentage of your screen real estate is used for applications (i.e., useful information), and less for operating system bloat. This argument seems compelling in its simplicity. (This is another reason I prefer non-MS machines; Windows seems to waste quite a bit screen space, leaving only a sliver for useful information.) Following this advice, I recently purchased a 30" Apple Cinema display. Perhaps not surprisingly, I absolutely love it. However, it is more than just having a giant screen -- it helps me get my work done. Now here's the proof

On the bigger screen, people completed the tasks at least 10 percent more quickly - and some as much as 44 percent more quickly. They were also more likely to remember the seven-digit number, which showed that the multitasking was clearly less taxing on their brains.
(from Meet the Life Hackers)