serving files from farm via https

[ctb]

So! good news for those of us trying to distribute files! per e-mail (copied below) ~/public_html/ is exposed and supports symlinks, too!

so if you want to expose files, do

mkdir ~/public_html/
ln -s /path/to/directory_with_files ~/public_html/new_dir_name

and voila, it will be exposed as https://farm.cse.ucdavis.edu/~USERNAME/new_dir_name/.

files that you want made available must be readable by everyone (or, at least the apache process), so a+r, and all of the directories above the file must be a+x.

For example, see https://farm.cse.ucdavis.edu/~ctbrown/sourmash-db/, which is served via ~ctbrown/public_html/sourmash-db/, which in turn is symlinked to /group/ctbrowngrp/sourmash-db/.

• you can browse and find filenames by default! 🎉
• this can be a bit of a security risk in terms of accidentally exposing files you don't want exposed, so if you want to turn off such file browsing via the Web, you can touch index.html in directories you don't want exposed, that blank page will be loaded in place of the directory index. (although if you know the filename(s) you can still grab them). this is a nice option b/c it leaves the directories on farm itself listable via the command line.

It is unclear what Apache access etc options are supported for per-directory configuration (see this guide for background) but I will pursue as I find interest :)

Note that this is NOT a good way to distribute archival files that should be persistent, you should explore osf.io, figshare.com, and/or zenodo.org for those purposes - they will give you archival URLs suitable for publication.

Appendix - e-mail from farm admin

Hi Titus,

This one might actually be quite simple. Farm supports serving web files from your home directory. You can just make a ~/public_html/ directory, make sure it, and your home directory are 775 (or at least readable and executable by Apache), and start populating it with files.

My quick testing shows that symlinks work as well, so you can just populate ~/public_html/ with symlinks to where the data resides. You will need to ensure that Apache at least has read and executable (for directories) on all paths leading up to the files you need to share.

Let me know if you have further questions.
Omen Wild
HPC Administrator
UCD HPC Core Facility

[ctb]

re userdir configuration, see /etc/apache2/mods-enabled/userdir.conf on farm -

        UserDir public_html
        UserDir disabled root

        <Directory /home/*/public_html>
                AllowOverride FileInfo AuthConfig Limit Indexes
                Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
                Require method GET POST OPTIONS
        </Directory>

looks like we can create .htaccess files (per this guide) to do many things, but presumably not execute scripts/do CGI. Yay!

[t-shoemaker]

@ctb do you know if this feature will persist after the Farm upgrades? I'm noticing that it's down for the moment

[ctb]

inquiring now!

[t-shoemaker]

Let me know if you need someone to bang around on things. Thanks!

[ctb]

it sounds like it's working again! give it a try @t-shoemaker !

[t-shoemaker]

I can confirm that it is indeed working. Thanks, Titus!