Home Technology Upload Manager: Open Source Library

Upload Manager: Open Source Library

by dharmeshasodiya

upload-manager is an easy open source python library which manages all uploading, downloading, state management functionality and multiple scenarios of our uploads & processing of your project.

Features

  • Uploading file to AWS S3
  • Reading uploaded file content from AWS S3
  • Upload Processing state management
  • Upload Manager Dashboard
  • Upload tracking

Use Cases

Upload manager library can be used in different use cases. Given below are the some processing scenarios:

Sync processing – In Sync processing we are uploading and processing the file on the same server. In this case, User don’t have to plug the functionality of downloading a file before the process, User has to just perform 3 operations i.e 1. Upload the input file 2. Upload status change 3. Uploading response file to s3.

Async processing – Async processing is the widely implemented processing, where we take the request on one server and process it on another server using a queuing mechanism. User has to plug functionality for both uploading as well as downloading the file before processing as processing will be done on another server.

Simple processing – Simple processing is when you already have the file uploaded on S3 and you just want to process the file and keep track of processing. Here, user can exclude the functionality of uploading file to S3, he just have to call create upload functionality(it creates a DB track for upload) and further processing can be done, similar to Async processing.

Examples

Example 1: A Product has a daily report that contains data which is obtained after heavy computation. And multiple users want to look at it or download it. Instead of doing computation again and again for generating the report we can use Upload Manager. Let’s see How? User has to generate this report on scheduled time by some scheduled jobs and upload it to S3 using upload manager and upload name should be such a way that it can be uniquely identified among the other day reports.

Example 2: Service A has a module which do some calculation on data and generate a report. And same calculated data, service B wants for some processing, Here, we can avoid making an API call to service A or a DB Query by using Upload Manager. Let’s see How? User can use Upload manager in service A and upload this report to S3. Now, the Upload ID or the S3 file link can be shared to service B. Service B can retrieve this data and use for processing.

Note – In above example, Upload Manager should have a separate database which can be accessed by multiple services.

Architecture Design

Installation

pip install https://github.com/Bigbasket/bbuploadmanager # BB internal

OR

pip install upload-manager

Prerequisite

  • Python 3 and above
  • Django 2.1 and above

Environment

AWS_STORAGE_BUCKET_NAME

  • variable need to be set in environment to specify aws bucket.

AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY

  • variable need to be set in environment, Only if I Am Role is not enabled on the machine.

Sample Code

# Library Plugin code to be added in settings.py
INSTALLED_APPS = ["upload_manager.apps.UploadManagerConfig"]

# Below code to be implementation in your application.py\
# imports
from upload_manager.service import UploadManagerService

# file upload example
upload_service = UploadManagerService(<upload_name>, <username>, <S3_bucket_path>, files=[<file_obj>,])
upload_obj = upload_service.start_upload()

# reading uploaded file data in async thread
file_data = UploadManagerService.get_upload_content_for_id(upload_obj.id)

# writing response file example
upload_service.file_content = [("OrderId", "Status", "Message"),(11020304, "Success", "")]
upload_service.update_response_for_upload(upload_obj_id)

# updating upload status example
UploadManagerService.change_upload_status_to_inprocess(upload_obj.id)
UploadManagerService.change_upload_status_to_complete(upload_obj.id)
UploadManagerService.change_upload_status_to_failed(upload_obj.id)`

Dashboard View

Above image shows dashboard view, how the uploads can be viewed and tracked in django Admin.

  • Request File – Column contains a hyperlink to the uploaded file which was uploaded as part of start upload
  • Response File – Column contains a hyperlink to the uploaded response file which was uploaded after the processing.

You may also like

1 comment

Nikhil Kesari March 24, 2021 - 7:06 pm

Very nice.

Reply

Leave a Reply to Nikhil Kesari Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: