huwebshop

Git for the HU Webshop project.

Introduction

This project was created to assist the students of TAAI-V1GP-19 (Group Project) with using the provided database of real-life products, sessions and anonymized profiles. Last year (2018-2019), students were given this same data and asked to create a recommendation service. However, getting the database to a visibly workable state proved to be such a difficult task, that much of the course's runtime was dedicated to that alone. We hope that by removing that roadblock in advance, students can dedicate their time to the actually interesting part of the course.

The project contains the following:

1. An example webshop, which can connect to a preconfigured database, either local or remote, and display basic pages such as product listing pages, product detail pages and a rudimentary shopping cart;
2. A dummy recommendation service, which uses the same database to provide recommendations through a REST interface, although at the moment, it simply returns randomly chosen products;

In the following sections, we will go into more detail on the requirements for setting up this project on your own device, how to run it, and what it should look like.

The author of this project is Nick Roumimper ([email protected]). If there are any questions about this project that this Readme can't answer (which may very well happen), please feel free to contact me.

Requirements

To run this code, you need to have the following programs and libraries installed:

• Python 3 (website: https://www.python.org/). This code was developed using Python 3.7. Some of the libraries used here use methods that are set to become deprecated in Python 3.8; when using any version beyond 3.7, be sure to use the most recent versions possible.
• MongoDB Community Edition (webpage: https://docs.mongodb.com/manual/administration/install-community/). This allows you to run a MongoDB database locally; almost all students will want to do this. It is strongly advised to also include MongoDB Compass (an option that comes up during the installation wizard).
• Flask (command line: pip install Flask). This code was developed using Flask 1.0.3, and automatically included Jinja2 v. 2.10.1 and Werkzeug v. 0.15.4, amongst other things.
• Pymongo (command line: pip install pymongo). This code was developed using Pymongo 3.8.0.
• Flask-RESTful (command line: pip install flask-restful). This code was developed using Flask Restful 0.3.7. This library allows you to run the dummy recommendation service locally; almost all students will want to do this, at least to start out with.
• Python-Dotenv (command line: pip install python-dotenv). This code was developed using Python-Dotenv 0.10.3.
• Requests (command line: pip install requests). This code was developed using Requests 2.22.0.

Sources Included in this Repository

• jQuery v. 3.4.1, from https://jquery.com
• Roboto font files, Latin and Latin-Extended, downloaded from Google Fonts (https://fonts.google.com)
• CSS snippets for responsive buttons, provided by Federico Dossena (https://fdossena.com/?p=html5cool/buttons/i.frag)

First Run Instructions

In order to get this project to run, the following steps need to be completed:

1. Set up the database, either locally or remotely;
2. Configure the environment variables for the local webshop and recommendation service;
3. Start up the recommendation service;
4. Start up the webshop.

We will expand upon the process for each of these steps in the upcoming subsections.

Database Setup

The course's teacher is expected to have provided you with the files that make up the database at this point. Note that these are not included in this project, due to their size. Alternatively, you have received credentials for a remote MongoDB cluster to connect to. If neither apply, please contact your course's teacher.

Please follow the instructions in the subsection applicable to your situation.

Setting Up a Local Database

You are expected to have three files in your possession:

1. products.json, containing the full set of products available in the webshop;
2. sessions.json (sometimes called sessions4.json), containing all applicable sessions for our subset of the data;
3. profiles.json (sometimes called visitors.json), containing the profiles associated with the sessions.

With MongoDB Community Edition installed on your device, ideally including MongoDB Compass, take the following steps:

• Create a database called huwebshop (alongside the default ones: admin/config/local). In MongoDB Compass, you see this option after connecting to your local database with default settings; if asked for the name of a collection, call it "products". When using the Mongo Daemon, a command such as "use huwebshop" may be required.

• Make sure you can access the mongoimport tool from the folder containing your data. On most computers, that means adding the path to the folder containing the executables (typically something like C:\Program Files\MongoDB\Server\4.2\bin) to your PATH environment variable. If that doesn't work, or you're working off of misbehaving HU employee laptops like us, you can also navigate to this same folder, copy the executable and paste it alongside the three files (yuck).

• Import products.json to a collection called products. A mongoimport command is structured like this:

  mongoimport --db database name --collection collection name --file file name --legacy

  The legacy flag is required when using mongoimport version 4.2 or beyond, because the files are in the old Extended JSON format (v1). Anyone who has recently installed mongoimport will have a version of 4.2 or beyond; if you have an older version (command line: mongoimport --version), consider leaving off the legacy flag.

  So in this particular case, you would have to execute the following from the command prompt:

  mongoimport --db huwebshop --collection products --file products.json --legacy

• Import sessions.json/sessions4.json to a collection called sessions. In other words, execute the mongoimport command:

  mongoimport --db huwebshop --collection sessions --file sessions.json --legacy

  with a different filename, if necessary.

• Import profiles.json/visitors.json to a collection called profiles. In other words, execute the mongoimport command:

  mongoimport --db huwebshop --collection profiles --file profiles.json --legacy

  with a different filename, if necessary.

And that should do it! You now have a database running on your local system with all three collections necessary to feed both your webshop instance and your recommendation engine.

Connecting to a Remote Database

You are expected to have received credentials for a remote MongoDB database cluster, which can be explored in your browser by signing in at https://www.mongodb.com. If you have these credentials, we'll assume you got them from your teacher, and you don't have to set up the collections anymore. This section is primarily dedicated to ensuring that eventually connecting goes smoothly.

Be sure to check the following when you try connecting for the first time:

• Are you on the network whitelist? Unless explicitly configured to allow everyone, MongoDB clusters are set to only accept connections from preset IP addresses. Check under Security > Network Access in the web interface to see which applies to you.

• Do you have both read and write rights? Although this webshop code will not change the preexisting data, it may attempt to add a collection called categoryindex if it does not yet exist. Check exactly which rights you have under Security > Database Access in the web interface.

• Do you know the cluster's server name? You presumably have a username and a password, but do you also have the last missing piece, the cluster's server name? This slightly odd identifier is also needed to connect to this cluster. It can be found by navigating to Clusters > Command Line Tools > Connect Instructions > Choose a Connection Method > Connect Your Application, and examining the field under "Connection String Only". This should be in the format of:

  mongodb+srv://{username}:{password}@{server name}/test?retryWrites=true&w=majority

  As you may have guessed, the server name we're referring to is the section between the at sign and the forward slash. If you don't have this, write this down and keep it handy!

Configure the Environment Variables

In order to make running the code easier after it has been set up the first time, we have moved certain settings to a .env file. You will need such a file in the top-level directory of this repository if either of the following two applies:

1. You wish to connect to a remote MongoDB cluster;
2. You wish to connect to a recommendation service that is not at the default location (which here is http://127.0.0.1:5001).

If you are trying to connect to your local MongoDB, using the default dummy recommendation service, you don't need to perform this step.

To set this up easily from a Unix shell (which includes the Git Bash included with most distributions of Git), you can run huw_remote_setup.sh (command line: sh huw_remote_setup.sh). However, at its core, the .env file is just a text file containing four variables:

MONGODBSERVER=*server name for the remote MongoDB cluster*
MONGODBUSER=*user name for the remote MongoDB cluster*
MONGODBPASSWORD=*password for the remote MongoDB cluster*
RECOMADDRESS=*address for the recommendation service*

Any variables you're not using, you can leave blank (e.g. MONGODBSERVER=). Note that the file must be called exactly .env. So technically, the file doesn't have a name, it only has an extension.

Start the Recommendation Service

To run the recommendation service, you will need to open a terminal window and navigate to the top-level directory of this repository. If you have a Unix shell handy, you can run huw_recommend.sh (command line: sh huw_recommend.sh). If not, you need to perform the following commands (written for Windows, Command Prompt specifically):

set FLASK_APP=huw_recommend.py
python -m flask run --port 5001

Depending on your system and type of terminal, these commands may need to be slightly different. Refer to the Flask Quickstart for more information on these commands.

You will know if/when the code is running when its terminal shows the following message:

* Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)

If this code runs, you can't see it do anything yet. Just be sure to leave this terminal window open!

Start the Webshop

Now, let's run the webshop! You will need to open another, separate terminal window and navigate to the top-level directory of this repository. If you have a Unix shell handy, you can run huw.sh (command line: sh huw.sh). If not, you need to perform the following commands (written for Windows, Command Prompt specifically):

set FLASK_APP=huw.py
python -m flask run

When you run this program for the first time, it may take a while to start up. It should be faster on future iterations. You will know if/when the code is running when its terminal shows the following message:

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

While both terminal windows are running, go to your browser of choice and navigate to http://127.0.0.1:5000 (the default location and port for Flask projects). If you see a front page like the one shown below, congratulations! This environment is your test webshop for the Group Project.

Stopping Either Process

When you're done with this particular session, or you want to stop and restart either service because you've made changes, you can terminate the process by going into the appropriate terminal window and pressing, as suggested, CTRL+C.

The Project in Action

When running, the front page of the webshop should look something like this:

The text may be different in future versions, but the overall layout should remain more or less the same. The bar at the top allows you to change the profile ID you're viewing the site with; it defaults upon restart to the first one it can find. In the database, you will find these IDs in the profiles table, as the _id property, e.g. ObjectId("5a3a1169a825610001bc0f3a") . This allows you to test your recommendation engine with different profiles.

Through the main menu, you can reach the product overview and detail pages. You can tell the recommendation service is working when refreshing the page leads to entirely different random suggestions:

---

---

...and then, after refreshing...

---

---

Feel free to play around and test this webshop (or rather, a shell of a webshop) out! If you run into any errors, or have ideas for improvements that aren't yet listed in this Readme, feel free to contact the author at [email protected].

Design Philosophy

If you're going to change parts of this code, it may be helpful to understand the design philosophy behind it. Much of this section goes into the webshop part, seeing as that is the most complex element. You can also review this section if you're interested as to how the project is structured.

• The recommendation service is as simple as possible. It is a REST service with one GET method, that takes the profile id and number of recommendations as parameters and returns that number of random products from the database. In fact, it doesn't even use the profile id - this is only to demonstrate how parameters are passed to the service. This is what you'll want to expand on and get creative with.
• The project uses default directories for its content. The Flask default setup stores any static resources (CSS, Javascript, images) in the static folder, and any HTML templates in the templates folder.
• The default session mechanism in Flask is used to store user data. Flask maintains a session mechanism that encrypts its cookies given a special secret key; here, the secret key is randomly generated whenever the webshop is started. In the session object, we store the profile id, shopping cart, number of items per page and so on.
• The Python code for the webshop does not render HTML. Since Jinja, the templating engine behind Flask, can process all kinds of complex objects, the Python code's role is essentially to retrieve the necessary data and pass it along. This also means that the .html files in the templates folder contain a lot of interesting logic.
• The Python code uses the add_url_rule method instead of decorators. If you're used to working with Flask, you may expect to see the @app.route() decorator around. However, we've attempted to keep as much logic as possible inside the single class of this project. You can find the equivalent add_url_rule function calls in the __init__ of huw.py.
• Dynamic Javascript requests are handled through POST calls. Several events stemming from the webshop occur "in-page", i.e. without navigating to a different URL. To distinguish between these requests and regular, viewable pages, we have restricted the associated functions to POST calls. You can find these methods at the bottom of the class.
• The Jinja templates extend the base template (base.html). For a webshop this simple, we can use one general template for almost all pages, only changing the center div's content for each page - so that is what we did.

Thank you for working with this project! We hope it helps to have this base setup at your disposal.

Todo (Reference for the Developers)

• Design improvements and tweaks (closer to the real thing)
  • Product page layout improvements
  • Stylize the dynamic shopping cart element
• Optional improvements
  • Extend documentation to non-Windows systems
  • Create shell scripts for all systems
  • Packaging?
  • Sorted category index
  • Correct pagination redirects
  • Error pages
  • Image generator
  • Search function
  • Offset and page relocation pagination
  • Responsive design
  • Price calculation and discount rule implementation