Load into Database

Out of the box, we support loading the data into a PostgreSQL database.

Note: You can always load it into another database, in which case, you need to modify the code to support your database. You can checkout:

Prerequisites

  • Have NodeJS, pnpm, and Git installed.
  • Clone the FNTU repository. Run git clone git@github.com:Acrylic125/fntu.git then cd scraper and pnpm i.
  • Have a database ready. Refer to the PostgreSQL Setup section below.

PostgreSQL Setup

Below we have some recommended ways to setup a PostgreSQL database.

MethodSteps

Local PostgreSQL

Refer to this Guide

Docker

For a local setup,

  • Run docker run -d --name fntu-postgres -e POSTGRES_PASSWORD=mysecretpassword -e POSTGRES_USER=myuser -e POSTGRES_DB=fntu-db -p 5432:5432 postgres:16
  • Try it out with docker exec -it fntu-postgres psql -U myuser -d fntu-db

Your DATABASE_URL should be postgresql://myuser:mysecretpassword@localhost:5432/fntu-db

Optional, but we recommend setting up Docker Compose, with a mounted volume.

version: '3'
services:
  fntu-postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: mysecretpassword
      POSTGRES_USER: myuser
      POSTGRES_DB: fntu-db
    ports:
      - 5432:5432
    volumes:
      - ./data:/var/lib/postgresql/data

Run docker compose up -d to start the database.

Supabase

We will use Supabase to host the database without using their SDKs.

  • Go to here and create a new project.
  • Name the project as whatever you want, and give it a password. Save the password somewhere safe.

Once you are in the project, at the top of the nav bar, click Connect.

  • Click on the ORMs tab, and select Drizzle. Copy the DATABASE_URL.
  • Replace the [YOUR-PASSWORD] with the password you saved earlier.

Once cloned and you have a database ready, Copy .env.example and name it .env. Configure it based on the instructions in .env.

Example:

DATABASE_URL=postgresql://myuser:mysecretpassword@localhost:5432/fntu-db

Loading the Data into the Database

First, push the schema to the database. You will need to do this if the table schemas ever change.

pnpm drizzle-kit push

Go Download the Data, unzip it, and put the unzipped download folder somewhere.

Then, run the insertion script.

pnpm run start insert /path/to/download