Large data files¶
When setting up simulation models or analyzing data it is sometimes the case that large data files (say larger than 10 MB) are necessary, XPSEC table models and Chandra event lists to be fed into ixpeobssim being typical examples.
Warning
Please refrain from pushing large data files to the ixpeobssim repository—a code repository is for code, not for data! Even setting aside the fact that data files are essentially impossible to diff (i.e., if and when you push a new version of a file you are effectively pushing the entire thing over and over again, not just the difference with respect to the previous version), if you push 1 GB of FITS files to the repository, you are effectively forcing everybody else to download 1 GB of data at the next pull, and you are increasing the dimensions of the bare repository in a non-reversible way.
As a rule of thumb, try and avoid pushing files larger than a few MB, unless there is a compelling reason for doing so. (Really, once you have pushed a file there’s essentially no way to get rid of it in git.)
If you have large auxiliary files that are potentially useful for other collaborators there is a separate repository that we (ab)use specifically for this purpose, so that only the subset of users that care about a specific set of files needs to worry about. You will notice that we don’t actually push files on that repository—we attach files in the associated download page, instead. You should refer to the README file on the ixpeobssim_auxfile repository for minimal instructions about the usage of that facility.
On the ixpeobssim
side, as of version 14.0.0, we have a mechanism in place to
direct the user to this repository when a functionality that needs one or more
auxiliary file(s) is used for the first time. Typically all you have to do is
download the necessary file(s) into your $IXPEOBSSIM_AUXFILES
directory
(defaulting to ~/ixpeobssimauxfiles
), and there are facilities to provide
instructions to the user to do so.
To this end, the ixpeobssim.srcmodel.magnetar.xMagnetarModelsT2020
interface to
the magnetar table models described in
Taverna et al. 2020
is the original reason for setting up the large-file infrastructure, and a
good source of inspiration for the associated usage.
In a nutshell, ixpeobssim
provides three convenience functions to handle
auxiliary (large) files, and you can protect the sensitive part of your
configuration or analysis code by a simple call along the lines of
from ixpeobsim.utils.logging_ import abort
from ixpeobssim import auxfile_missing
if auxfiles_missing(*file_list):
abort()
This should interrupt the execution of the program and print on the terminal enough information for the user to be able to download the right file(s) in the right location.