~gtf/culture-vulture

A site for listing cultural events

#40 Add National Theatre

~gtf filed ticket on culture-vulture todo

1 year, 6 months ago

#39 Migrate to DaisyUI

~gtf filed ticket on culture-vulture todo

1 year, 6 months ago
View project feed  RSS 

#Culture Vulture

Culture Vulture is an application to present a curated list of cultural events in London.

#Background

I have maintained, on-and-off, a curated listing of cultural events in which I'm interested, based on collating things from the various emails and websites of venues like the National Theatre, Barbican, and Sadlers Wells (among many others). At first this was an email list, and then it became a Notion database.

The problem is that trawling these pages is quite time-consuming, and it's easy to forget one. I wanted a website which would do all the trawling and scraping of "What's On" pages and emails for me, and then present a list of events I could review, and approve for display, with all the details pre-filled.

#System Design

Culture Vulture consists of the following components:

  • a series of scrapers and parsers for web-pages and emails web-based
  • a UI for the listing (including filters and administrative functions)

#Uniquely identifying events

Since scrapers will be run multiple times, there is no way of ensuring that the same events do not turn up twice. Therefore we produce a unique "origination" URI using the following format:

uri:event-origination:<source-key>:<event-title>:<event-start-date>

The source-key is a fixed string relating to the scraper-parser which sources the event.

#Installation and Deployment

There are two parts to the application: frontend assets (stylesheets and scripts) and the server-side application (which renders HTML or JSON to the client).

Build artefacts are created in the dist folder, and there are three rules for doing this:

  • make scripts: this uses tsc to compile scripts in src/frontend/ts
  • make styles: this copies over stylesheets from src/frontend/css
  • make binaries: this uses stack to build the server and CLI binaries

#Installation (without docker)

If you just wish to run culture-vulture, you will require:

  • GHC 9.2.8
  • stack
  • tsc (typescript compiler)

You can install GHC and stack using ghcup which is what we would recommend.

You can install the dependencies using make build-deps, or just build the entire application using make binaries.

#Running locally for development

There is a basic docker-compose file for setting up a postgresql instance for local testing and development. You can run this using:

docker-compose -f docker/compose-local.yaml -p culture-vulture up -d

You will then need to source the local environment variables needed to connect to it, which you can do by sourcing local-env.sh.

Once you have the local environment setup, you can use make dev-watch which will watch for local file changes and rebuild accordingly, while also running a development server, or make dev-server to run a simple development server.

#Deploying to a docker host

We use docker-compose to run the application in production, so you will need a host provisioned accordingly. You will need to create a folder at /usr/local/src/culture-vulture to which the user on your host has read, write, and execute permissions.

Please see docker/SECRETS.md for information on the secrets files you will need to create in the docker directory before deploying.

Once this is done, you can simply run SERVER_ADDRESS=<your server> make deploy, and the application will be deployed complete with letsencrypt SSL certificates for your chosen domain and an nginx reverse-proxy.

#For first time deployments

When deploying for the first time, a slightly annoying hack is needed to work around the fact you have no SSL certificates yet. You will first need to comment out the second (HTTPS) server block in docker/nginx/servers/culturevulture.conf.template, run make deploy, and then uncomment it and run make deploy again once the certificates have been provisioned.

#Testing

Unit and integration tests can be found in the test folder, organised by module. You can run tests (once you have a local database and appropriate environment variables) by running make test.

#Publishing a release

When publishing a release, first ensure that the project builds, especially make docker-image. Then you need to update the CHANGELOG.md, and the version numbers in the following files:

  • package.json
  • package.yaml

Once those changes have been committed and you have verified it all works, use git tag to tag this commit:

git tag v1.4.0
git push
git push --tags

Then make a docker image:

make docker-image

And once it has built, tag the docker image accordingly (note the difference between the git tag v1.4.0 and the docker tag 1.4.0) and push the image:

docker tag <hash> gtf21/culture-vulture:latest
docker tag <hash> gtf21/culture-vulture:1.4.0
docker push -a gtf21/culture-vulture